Proxies for Web Scraping: Types, Privacy, and Detection

Web scraping almost always runs into the same wall: a site notices that one IP address is firing a suspicious number of requests, and starts blocking you, throwing CAPTCHAs, or serving empty pages. Proxies solve this problem — they swap your real IP for someone else's and spread the load across many addresses. But not all proxies are equal. There are dozens of types, and the choice decides whether your scraper slips through unnoticed or burns out on the first hundred requests.

Below is a breakdown of every major type of web scraping proxy — how they work, their privacy levels, how they're billed, and code examples in several languages. By the end you'll know which proxies for web scraping fit which job, and why a good proxy is necessary but never sufficient on its own.

What a proxy is and why scraping needs one

A proxy server is a middleman between your scraper and the target site. The request doesn't go directly — it travels through the proxy, so the site sees the proxy's IP instead of yours. For scraping, that buys you three things:

1. Hiding your real IP — you're harder to identify and ban personally.
2. Spreading the load — a pool of hundreds of IPs lets you send many requests without exceeding per-address rate limits.
3. Bypassing geo-restrictions — you can collect data as if you were located in a specific country. Prices, search results, and content are often region-dependent.

Classification by protocol: HTTP, HTTPS, SOCKS

The first thing to understand is the level a proxy operates on.

HTTP proxy

Works only with HTTP traffic. It "understands" the structure of an HTTP request, can read and modify headers, and can cache responses. For scraping ordinary sites, that's often enough.

HTTPS proxy (HTTP CONNECT)

The same thing, but able to tunnel encrypted TLS traffic through the CONNECT method. The proxy doesn't see the content (it's encrypted) — it just forwards the bytes. Today almost the entire web runs on HTTPS, so CONNECT support is mandatory.

SOCKS4 and SOCKS5

SOCKS operates at a lower level — it's a universal tunnel for any TCP traffic (and SOCKS5 also handles UDP). The proxy doesn't inspect the content: it doesn't care whether it's HTTP, FTP, WebSocket, or anything else.

How SOCKS and HTTP proxies differ:

SOCKS4 vs SOCKS5:

• SOCKS4 is old — no password authentication, no IPv6, no UDP, and it can't resolve DNS on the proxy side (the domain is turned into an IP on your side, which is a leak: your DNS lookups reveal what you're scraping).
• SOCKS5 supports authentication, IPv6, UDP, and remote DNS resolution (the socks5h URL scheme). It's the better choice for scraping because DNS doesn't leak.

Rule of thumb: if you're choosing between them, take SOCKS5 — and use the socks5h:// scheme in code so DNS resolves through the proxy.

Classification by IP source

Far more important than the protocol is where the IP address came from. That's what decides how "human" you look to anti-bot systems.

Datacenter proxies

IPs owned by hosting providers and data centers (AWS, Google Cloud, OVH, Hetzner, and so on).

• Cheap, very fast, available by the thousand.
• Easy to detect: anti-bot systems keep databases of datacenter subnets, and many sites block these IPs pre-emptively.
• Good for scraping "friendly" sites without serious protection.

Residential proxies

IPs belonging to real home users, assigned by internet service providers (ISPs). Traffic flows through live people's devices — often via P2P networks or SDKs embedded in apps.

• Look like ordinary users; hard to tell apart from live traffic.
• More expensive, slower, sometimes unstable.
• Good for sites with serious defenses — marketplaces like Amazon and Walmart, social networks, search engines. Residential proxies for scraping are the workhorse for tough targets.

Mobile proxies

IPs from mobile carriers (3G/4G/5G). The twist: carriers use CGNAT, so hundreds of real subscribers sit behind a single IP.

• The highest trust level: banning such an IP means banning a crowd of real people, so sites tread carefully.
• The most expensive, with lower speed and stability.
• Used where everything else gets banned instantly.

ISP proxies (static residential)

A hybrid: the IPs formally belong to an ISP (like residential) but physically live in a data center (like datacenter proxies).

• Datacenter speed plus ISP reputation. They're static (they don't change), which is handy for long sessions.
• More expensive than datacenter.

Quick comparison by stealth and price:

Datacenter  →  ISP  →  Residential  →  Mobile
cheaper/                            pricier/
easier to spot                      harder to spot

IPv4 vs IPv6

The IPv4 address space has run out — addresses are scarce and expensive. IPv6 is practically infinite, so IPv6 proxies are noticeably cheaper.

But there's a catch: IPv6 proxies only work if the target site supports IPv6. Many large sites are still IPv4-only — an IPv6 proxy simply can't connect to them.

Bottom line: IPv6 is a bargain for scraping sites that definitely support it (Google and many global services). For everything else, and for maximum compatibility, use IPv4.

Proxy privacy (anonymity) levels

A proxy can "out" you in different ways — by adding service HTTP headers to your request. Based on exactly what gets passed along, there are three levels.

1. Transparent

Passes your real IP in the headers and openly announces that it's a proxy.

Via: 1.1 proxy
X-Forwarded-For: YOUR_REAL_IP

Useless for scraping — the site sees both the fact of the proxy and you. These usually sit in corporate or public networks for caching.

2. Anonymous

Hides your real IP but does not hide the fact that a proxy is in use.

Via: 1.1 proxy
X-Forwarded-For: PROXY_IP   (or absent, but Via gives the proxy away)

Your real IP is protected, but the anti-bot system sees the "this is a proxy" flag and may treat it with suspicion.

3. Elite / high anonymous

Passes neither your IP nor any sign of a proxy. The request looks exactly like a direct visit.

(no Via, X-Forwarded-For, X-Real-IP, Proxy-Connection)

Why do some proxies pass this data and others don't? It's down to the proxy server's configuration. Transparent proxies deliberately add X-Forwarded-For so the target server knows the original client (by design, for caching/corporate proxies). Elite proxies are set up to strip all of it. The headers that give proxies away: Via, X-Forwarded-For, X-Real-IP, Forwarded, Proxy-Connection, X-Proxy-ID.

For scraping you want elite proxies only. Anonymous is at your own risk; transparent, never.

Rotation: static and rotating proxies

• Static (sticky) — the same IP holds for a long time. Needed wherever a session matters (login, cart) so the site doesn't "lose" you when the IP changes.
• Rotating — the IP changes automatically: on every request, or every N minutes. Ideal for bulk collection where the session doesn't matter. Often the provider gives you a single gateway endpoint and handles rotation on their side from a pool of thousands of addresses. See rotating proxies for how to build this into a scraper.

Tor as a free proxy

Tor (The Onion Router) is an anonymity network where traffic passes through a chain of three nodes: entry (guard), middle, and exit. Each layer of encryption is peeled off at its own node — hence "onion" routing. Locally, Tor exposes a SOCKS5 proxy (port 9050 by default) that a scraper can connect to.

Yes, Tor also lets you choose geography and nodes. Through the torrc config you can control which countries you exit through:

# Exit only through nodes in these countries
ExitNodes {us},{de},{nl}
StrictNodes 1

# You can hard-set entry/exit nodes by country
EntryNodes {de}
ExcludeNodes {ru},{cn}
ExcludeExitNodes {ru}

Country codes go in curly braces by ISO ({us}, {de}, {fr}). StrictNodes 1 forbids stepping outside the list. To get a new chain (a new exit IP), send the NEWNYM signal to the control port.

Downsides of Tor for scraping:

• Slow — three hops plus a congested network.
• Exit nodes are blocklisted — the list of exit nodes is public, and many sites block all of Tor pre-emptively.
• Tiny IP pool — exit nodes number in the thousands, not the millions that residential providers offer. Rotation is limited.
• Fine for one-off tasks and experiments, but not for industrial scraping.

Free proxy lists

The internet is full of sites offering free proxy lists — thousands of IP:port pairs you can download right now. It sounds tempting, but the quality is reliably bad:

• Most are dead by the time you download them — you're lucky if 5–20% are alive.
• Slow and unstable — they respond, then they don't.
• Often transparent — they leak your IP.
• Already blocklisted — thousands of people used them before you, and serious sites banned them long ago.
• A security risk — you don't know who runs the proxy; it can intercept or tamper with traffic (especially dangerous for unencrypted HTTP and when passing logins).

Bottom line: free proxies are okay for learning and throwaway experiments, but any serious or commercial scraping needs paid ones. That's exactly why any public proxy must be checked before use (see below).

Checking (validating) proxies

You can't just grab a proxy and use it — especially from public lists. They need checking, because they constantly die, slow down, and change behavior. What to test:

1. Is it alive — does it respond at all, and with what status code.
2. Speed / latency — response time (ping, time to first byte). Reject the slow ones.
3. Anonymity level — does it leak your real IP. Do a request to an echo service (an endpoint that returns your IP and headers) and check for X-Forwarded-For and Via.
4. Real geo — an IP may claim to be "German" but actually sit in another country. Verify against a GeoIP database.
5. Type / reputation — is it blocklisted, is it flagged as datacenter/proxy (via fraud-score services).
6. HTTPS support — does CONNECT pass, does it break TLS.

The simplest Python checker — try to reach an echo service through the proxy and see what comes back:

import requests

def check_proxy(proxy: str, timeout: int = 8):
    proxies = {"http": proxy, "https": proxy}
    try:
        r = requests.get("https://httpbin.org/get",
                         proxies=proxies, timeout=timeout)
        data = r.json()
        origin_ip = data.get("origin")
        headers = data.get("headers", {})
        leaked = any(h in headers for h in
                     ("X-Forwarded-For", "Via", "X-Real-Ip"))
        return {
            "ok": True,
            "ip": origin_ip,
            "anonymous": not leaked,   # True = real IP did not leak
            "latency": r.elapsed.total_seconds(),
        }
    except Exception as e:
        return {"ok": False, "error": str(e)}

print(check_proxy("http://user:pass@1.2.3.4:8080"))

How proxies affect detection during scraping

A proxy is not an invisibility cloak. Anti-bot systems (Cloudflare, DataDome, HUMAN/PerimeterX, Akamai) look at many signals, and the IP is only one of them.

What gives a scraper away on the IP front:

• IP type. Datacenter subnets are known and flagged as "not human." Residential and mobile inspire more trust. This is the main factor your proxy choice influences.
• IP reputation (fraud score). The address's history: spam, botnets, past bans. A "dirty" IP gets caught even if it's residential.
• Request rate from one IP (rate limiting). Too many requests per second means a ban. A rotating proxy pool spreads the load and lowers the rate per address.
• Geo distribution. If "one user" hops from the US to Brazil and back within a minute, that's an obvious bot. Within a single session, keep the IP stable.
• ASN. The autonomous system number reveals who owns the IP (a host, or a real ISP).

What proxies do NOT cover (and why a proxy alone isn't enough):

• TLS/JA3 fingerprint — the signature of your TLS handshake. requests and curl don't look like a browser, and that shows through any proxy.
• HTTP headers and their order — a broken or non-standard User-Agent, missing the usual browser headers.
• Browser fingerprint — Canvas, WebGL, fonts, JS environment (relevant for headless browsers).
• Behavior — clicks and navigation that are too fast, too even, too inhuman.

Takeaway: a quality proxy (residential/mobile, elite, with a good reputation) sharply cuts IP-based detection, but it has to be combined with proper headers, a realistic fingerprint, and a sane request rate. A proxy is necessary, but not sufficient. If you're fighting Cloudflare or DataDome, pair proxies with browser fingerprint evasion and, where needed, CAPTCHA solving.

Billing: what you actually pay for

Providers charge in different ways, and the model decides what's cheapest for your task.

• Per GB (traffic). You pay for gigabytes transferred, with a huge, "free" IP pool. Typical for residential/mobile. Dangerous when scraping heavy pages with images and video — the bill climbs fast. Save money by downloading only HTML and blocking media.
• Per IP. You pay for specific addresses (say, 100 static datacenter/ISP proxies), with unlimited traffic. Good for large data volumes and stable IPs.
• Per port / per thread. You pay for the number of concurrent connections, not the volume. Common with mobile proxies.
• Per request or subscription. Typical for turnkey scraping APIs, where the provider rotates proxies for you and returns ready HTML.

How to choose: many small requests to light pages → per-GB or per-request is cheaper; moving large volumes → per-IP with unlimited traffic wins.

Code examples: connecting to a proxy in different languages

The proxy string format is nearly universal: scheme://login:password@host:port, e.g. http://user:pass@1.2.3.4:8080.

Python — requests (synchronous)

import requests

proxies = {
    "http":  "http://user:pass@1.2.3.4:8080",
    "https": "http://user:pass@1.2.3.4:8080",
}

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(r.json())

# SOCKS5 with remote DNS (needs: pip install requests[socks])
socks = {"http": "socks5h://user:pass@1.2.3.4:1080",
         "https": "socks5h://user:pass@1.2.3.4:1080"}
print(requests.get("https://httpbin.org/ip", proxies=socks).json())

Python — httpx (async, with rotation)

import asyncio, httpx, random

POOL = [
    "http://user:pass@1.2.3.4:8080",
    "http://user:pass@5.6.7.8:8080",
]

async def fetch(url):
    proxy = random.choice(POOL)          # random IP from the pool per request
    async with httpx.AsyncClient(proxy=proxy, timeout=10) as client:
        resp = await client.get(url)
        return resp.status_code, resp.text[:80]

async def main():
    tasks = [fetch("https://httpbin.org/ip") for _ in range(5)]
    for code, body in await asyncio.gather(*tasks):
        print(code, body)

asyncio.run(main())

Python — Tor (SOCKS5 + new chain)

import requests
from stem import Signal
from stem.control import Controller

proxies = {"http":  "socks5h://127.0.0.1:9050",
           "https": "socks5h://127.0.0.1:9050"}

print(requests.get("https://httpbin.org/ip", proxies=proxies).json())

# Request a new exit IP (a new chain)
with Controller.from_port(port=9051) as c:
    c.authenticate(password="your_password")
    c.signal(Signal.NEWNYM)

Node.js — axios + HttpsProxyAgent

const axios = require("axios");
const { HttpsProxyAgent } = require("https-proxy-agent");

const agent = new HttpsProxyAgent("http://user:pass@1.2.3.4:8080");

axios.get("https://httpbin.org/ip", { httpsAgent: agent, httpAgent: agent })
  .then(res => console.log(res.data))
  .catch(err => console.error(err.message));

Node.js — SOCKS5

const axios = require("axios");
const { SocksProxyAgent } = require("socks-proxy-agent");

// socks5h:// — DNS resolves on the proxy side
const agent = new SocksProxyAgent("socks5h://user:pass@1.2.3.4:1080");

axios.get("https://httpbin.org/ip", { httpAgent: agent, httpsAgent: agent })
  .then(res => console.log(res.data));

Node.js — Playwright (headless browser through a proxy)

const { chromium } = require("playwright");

(async () => {
  const browser = await chromium.launch({
    proxy: {
      server: "http://1.2.3.4:8080",
      username: "user",
      password: "pass",
    },
  });
  const page = await browser.newPage();
  await page.goto("https://httpbin.org/ip");
  console.log(await page.innerText("body"));
  await browser.close();
})();

Go — net/http

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
)

func main() {
    proxyURL, _ := url.Parse("http://user:pass@1.2.3.4:8080")
    client := &http.Client{
        Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
    }
    resp, err := client.Get("https://httpbin.org/ip")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    body, _ := io.ReadAll(resp.Body)
    fmt.Println(string(body))
}

PHP — cURL

<?php
$ch = curl_init("https://httpbin.org/ip");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => "1.2.3.4:8080",
    CURLOPT_PROXYUSERPWD   => "user:pass",
    CURLOPT_PROXYTYPE      => CURLPROXY_HTTP, // or CURLPROXY_SOCKS5_HOSTNAME
    CURLOPT_TIMEOUT        => 10,
]);
echo curl_exec($ch);
curl_close($ch);

cURL from the command line

# HTTP proxy
curl -x http://user:pass@1.2.3.4:8080 https://httpbin.org/ip

# SOCKS5 with remote DNS
curl --proxy socks5h://user:pass@1.2.3.4:1080 https://httpbin.org/ip

Proxy selection checklist for scraping

The key rules:

1. Use elite anonymity — no X-Forwarded-For/Via leaks.
2. Always check any public/free proxy before going live.
3. For protected sites, use residential or mobile, not datacenter.
4. Proxies close IP-based detection, but not fingerprinting — combine them with proper headers and a sane request rate.
5. Match the billing model to your task: many light requests → per-GB/request; large volumes → per-IP.

When you'd rather not manage proxies at all

Building and maintaining a proxy stack — sourcing clean IPs, rotating them, checking health, tuning fingerprints, and handling CAPTCHAs — is a project in itself. If you just need the data, scraping.pro runs this end to end as a done-for-you data extraction service: the right proxy tier for each target, managed rotation, and delivery as clean structured output, or as an ongoing data-as-a-service feed. You get the results without operating the infrastructure.