Residential Proxy Scraper: Bright Data in Action

Some sites give up their data after a single requests.get(). The interesting ones don't. Business directories, marketplaces, travel aggregators and social platforms sit behind layers of IP reputation checks, rate limits and fingerprinting that flag datacenter traffic on sight. The reliable way through is a residential proxy scraper: a crawler that routes its requests through real home IP addresses so the target sees ordinary users instead of a server farm.

This article walks through building one with Bright Data — the provider formerly known as Luminati, which rebranded in 2021. We'll cover why residential IPs work where datacenter proxies fail, how to set up rotating residential proxies, the exact request patterns that survive tough anti-bot systems, and a worked case study of pulling every listing from a protected directory-style aggregator.

Why residential proxies beat protected targets

A proxy is a middleman: your request goes through it, and the site sees the proxy's IP, not yours. The question is where that IP comes from.

• Datacenter proxies are owned by hosting providers (AWS, OVH, Hetzner). They're cheap and fast, but anti-bot vendors keep databases of datacenter subnets and block or cloak them pre-emptively. Even a "clean" datacenter IP can be served fake or modified data once it's flagged as non-human.
• Residential IPs are assigned by Internet Service Providers to real households. Traffic from them looks like a normal person on a home connection. Providers like Bright Data source these through opt-in SDKs and consented peer networks where a user's device relays traffic while idle and connected — the provider is interested in the IP, not the user's data.

The practical upshot: for a well-defended target, a residential proxy scraper is often the difference between 7,000 clean records and a wall of CAPTCHAs. Pair it with rotating proxies and you spread requests across thousands of addresses so no single IP trips a per-address rate limit.

Rule of thumb: use cheap datacenter proxies for friendly sites, and reach for residential (or mobile) IPs the moment a target starts blocking, cloaking, or CAPTCHA-gating you.

The modern Bright Data stack (ex-Luminati)

If you last used this provider as Luminati, the concepts carry over but the names changed. Today the toolbox looks like this:

• Proxy zones — named configurations for a proxy type and its settings. You typically run separate zones for residential, datacenter, ISP/static residential, and mobile IPs, each with its own endpoint and credentials.
• Proxy Manager (LPM) — a free, open-source local service that sits in front of the zones and automates rotation, retries, per-domain rules, header overrides and stats. It's still on GitHub and still useful when you want fine-grained control on your own machine.
• Web Unlocker — a hands-off endpoint that handles rotation, header/fingerprint management and CAPTCHA solving automatically, returning the unblocked HTML. This is the modern replacement for hand-tuning LPM rules on the hardest targets.
• Scraping Browser — a cloud, anti-detect headless browser for JavaScript-heavy sites, controllable via Playwright or Puppeteer.

For most scraping jobs you interact with a zone through a single super proxy endpoint and encode your options (country, session, city) in the username.

Connecting to a rotating residential zone

The rotating residential endpoint is brd.superproxy.io:33335. Credentials follow the pattern brd-customer-<account>-zone-<zone>, and you append targeting parameters to the username.

import requests

USER = "brd-customer-<account>-zone-residential"
PASSWORD = "<zone-password>"
ENDPOINT = "brd.superproxy.io:33335"

proxy = f"http://{USER}:{PASSWORD}@{ENDPOINT}"
proxies = {"http": proxy, "https": proxy}

# Verify the exit IP the site will actually see
resp = requests.get("https://geo.brdtest.com/mygeo.json",
                    proxies=proxies, timeout=30)
print(resp.json())   # {"country": "US", "asn": {...}, ...}

On a rotating residential zone, every request gets a fresh IP by default. When you need the same IP to persist across a multi-step flow (login, pagination that depends on a session cookie), request a sticky session by adding a session ID and a country to the username:

def sticky_proxy(session_id: str, country: str = "us") -> dict:
    user = f"{USER}-country-{country}-session-{session_id}"
    url = f"http://{user}:{PASSWORD}@{ENDPOINT}"
    return {"http": url, "https": url}

Change session_id to jump to a new IP; keep it constant to hold one.

The blocking techniques you're actually fighting

Before the case study, it's worth naming what a protected aggregator throws at you — because residential IPs only solve part of it. Common defenses:

• IP-reputation blocks — entire datacenter subnets and known-bad ranges denied outright; some sites block whole geolocations.
• Rate limiting — a cap on requests per IP per second/minute. The fix is a large pool with continuous rotation so each IP stays under the limit.
• Fingerprinting — the site inspects your User-Agent, header order, TLS/JA3 signature and JavaScript environment to tell a crawler from a browser. Rotating a residential IP while sending a robotic, unchanging header set still gets you caught.
• Behavioral checks and CAPTCHAs — triggered when a session looks non-human; often leads to CAPTCHA challenges or silently altered data.

A residential proxy scraper handles the IP layer. You still have to look human at the request layer: realistic and varied headers, sane pacing, and a real browser for JS-gated pages.

Rotation strategy: round-robin and the waterfall method

Two patterns cover most jobs.

Round-robin rotation cycles through a pool of IPs, using each for a small number of requests before switching. In Proxy Manager you set the pool size and cap max requests per IP at 1 to rotate on every call. In plain code, you rotate by changing the session token (or simply relying on the zone's default per-request rotation).

The waterfall method is a retry ladder: try the request with a cheap, broadly-targeted residential IP first; on a failure signal — a 4xx/5xx, a redirect to a block page, a CAPTCHA marker, or an abnormally small response body — automatically retry with a more expensive, more specific IP (city-targeted, then mobile, then Web Unlocker). You escalate only when you have to, which keeps bandwidth cost down.

Here's the waterfall logic in application code:

import time, requests

def looks_blocked(r: requests.Response) -> bool:
    if r.status_code in (403, 429, 503):
        return True
    body = r.text.lower()
    if "captcha" in body or "unusual traffic" in body:
        return True
    if len(r.text) < 2000:          # suspiciously small = block page
        return True
    return False

def fetch(url: str, headers: dict, tiers: list[dict]) -> requests.Response | None:
    for proxies in tiers:            # cheapest tier first, then escalate
        try:
            r = requests.get(url, headers=headers, proxies=proxies, timeout=30)
            if not looks_blocked(r):
                return r
        except requests.RequestException:
            pass
        time.sleep(1.5)
    return None                      # all tiers exhausted

Case study: extracting a protected directory aggregator

The target is a national business directory — think a hotels-by-state listing behind pagination and rate limits. The goal: collect every listing URL across all US states without the aggregator detecting the crawl or spoofing us with duplicate results.

Setup. One rotating residential zone, US-targeted, with the waterfall fetch above. We iterate states, page through results, and move to the next state once a page stops yielding new listings.

import re, time, requests
from urllib.parse import quote

USER = "brd-customer-<account>-zone-residential"
PASSWORD = "<zone-password>"
ENDPOINT = "brd.superproxy.io:33335"

def proxy_us() -> dict:
    url = f"http://{USER}-country-us:{PASSWORD}@{ENDPOINT}"
    return {"http": url, "https": url}

HEADERS = {
    "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/124.0 Safari/537.36"),
    "Accept-Language": "en-US,en;q=0.9",
}

STATES = ["AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI","ID","IL",
          "IN","IA","KS","KY","LA","ME","MD","MA","MI","MN","MS","MO","MT",
          "NE","NV","NH","NJ","NM","NY","NC","ND","OH","OK","OR","PA","RI",
          "SC","SD","TN","TX","UT","VT","VA","WA","WV","WI","WY","DC"]

URL = "https://example-directory.com/search?terms=hotel&geo={state}&page={page}"
LINK_RE = re.compile(r'class="business-name"\s+href="([^"?]+)')

all_links: set[str] = set()
for state in STATES:
    page, seen_before = 1, -1
    while len(all_links) != seen_before:          # stop when a page adds nothing
        seen_before = len(all_links)
        url = URL.format(state=state, page=page)
        r = requests.get(url, headers=HEADERS, proxies=proxy_us(), timeout=30)
        for href in LINK_RE.findall(r.text):
            all_links.add(href)
        page += 1
        time.sleep(1.0)                            # be polite; avoid rate limits

with open("listings.txt", "w") as f:
    f.write("\n".join(sorted(all_links)))
print(f"Collected {len(all_links)} unique listings")

Two details that matter in practice:

1. Deduplication as a defense check. We store listings in a set and stop a state once a page yields no new URLs. If an aggregator tried to spoof a bot by repeating the same listings across pages, you'd see the set stop growing — a useful signal that you're being fed decoy data rather than fresh records.
2. Pacing. A short sleep plus per-request IP rotation keeps each address comfortably under the site's rate limit. On this kind of target, a clean residential pool routinely returns the full listing set — in our archived run, roughly 7,000 unique records across a few hundred requests — where a datacenter pool would have stalled on CAPTCHAs early.

For JavaScript-rendered listings, swap requests for the Scraping Browser (or a local Playwright driver pointed at a residential proxy) so the page's scripts run before you parse. See scraping dynamic, JavaScript-heavy content for that pattern.

The toughest targets: exclusive (dedicated) residential IPs

Some domains are so aggressive that even shared residential IPs get burned quickly, because other users of the same pool may already have abused that IP against the same site. For these, Bright Data offers exclusive / dedicated residential IPs — addresses allocated to you alone for a specific target domain. (In the old Luminati world these were the "gIP" zones.)

The classic use case is a geo-restricted or heavily-defended site where shared IPs simply fail: you configure a dedicated zone targeted at that one domain, and because nobody else is hammering the target from those IPs, the success rate is far higher than with a general pool. It's more expensive per IP, so reserve it for the handful of domains that genuinely need it and keep the shared rotating pool for everything else.

Hands-off alternative: Web Unlocker

Building and tuning all of the above is worthwhile when you want control and low cost at scale. When you'd rather not babysit rotation, headers and CAPTCHAs, Bright Data's Web Unlocker wraps residential IPs, fingerprint management and CAPTCHA solving into a single request — you send a URL, it returns unblocked HTML. It costs more per successful request than raw proxies, but it collapses a lot of anti-bot engineering into one call. Many teams run raw residential proxies for bulk, forgiving targets and route only the stubborn URLs through Web Unlocker.

Best practices for a residential proxy scraper

• Match the header story to the IP. A US residential IP paired with a fresh, realistic browser User-Agent and Accept-Language: en-US is coherent; a robotic default header set is not.
• Rotate at the right granularity. New IP per request for stateless listing pages; sticky session for multi-step flows that rely on cookies.
• Escalate, don't over-spend. Waterfall from cheap to expensive IP types; most requests should succeed on the first, cheapest tier.
• Watch for silent cloaking. Tiny response bodies, missing fields, or repeated/decoy data mean you're detected even without an HTTP error. Validate response size and content, not just status codes.
• Respect the target. Throttle, honor rate limits, and only collect public data. Good manners also happen to reduce blocks.

FAQ

Is "Luminati proxy" the same as Bright Data? Yes. Luminati rebranded to Bright Data in 2021. Old zone concepts (residential, datacenter, exclusive IPs, Proxy Manager) still exist under the new name and endpoints.

Rotating vs. sticky residential — which do I want? Rotating (a new IP per request) for large, stateless crawls like listing pages. Sticky (one IP held for minutes) when a session cookie or login has to persist across requests.

Are residential proxies legal to use for scraping? Using residential proxies is legal; what matters is what and how you scrape. Stick to publicly accessible data, respect rate limits and terms where they apply, and avoid personal data you have no basis to collect.

How many IPs do I actually need? It depends on the target's rate limit and your throughput. Rotation through a large pool means you rarely reuse an address inside its cooldown window — that, not the raw pool size, is what keeps you unblocked.

Doing it for you

A tuned residential proxy scraper — with rotation, waterfall retries, fingerprinting and CAPTCHA handling — is a lot of moving parts to maintain as targets change. If you'd rather receive clean, structured records than run the pipeline, scraping.pro offers this as a managed web scraping service, including the hardest anti-bot targets. And when you need continuous feeds rather than one-off pulls, our data as a service delivers them on a schedule.