Scraping Business Directories: Tips and Tricks

Business directories — Yellow Pages, Yelp, Yell, Google Maps, industry association listings, chamber-of-commerce rosters — are among the richest sources of B2B data on the web: company names, addresses, phone numbers, websites, categories, opening hours, and reviews, all in one place. That's exactly why they're a magnet for scraping business directories for lead generation, market research, and local SEO. It's also why they're some of the hardest sites to scrape well. This guide collects the practical tips and tricks that separate a directory scraper that quietly builds a clean dataset from one that gets blocked on page two and drowns in duplicates.

Why directories are their own kind of hard

Directories (a.k.a. data aggregators) behave differently from ordinary websites, and understanding their traits up front saves you a lot of pain:

• The data is query-driven, not page-driven. There's no tidy list of URLs to crawl. You reach records by searching — usually a category plus a location (?what=plumbers&where=Chicago). Your first job is to figure out the search interface and enumerate the queries that cover the data you want.
• The dataset is huge and hard to size. You often can't tell how many listings exist without probing. Expect to write a small reconnaissance script just to estimate volume before committing to a full crawl.
• The data changes constantly. Businesses open, close, move, and rebrand. A one-shot scrape goes stale fast, so real value comes from re-scraping on a schedule and detecting what changed.
• They fight scraping aggressively. Directories treat their data as the product, so they deploy rate limiting, IP bans, browser fingerprinting, captchas, and honeypots. Plan for defenses from the first request, not as an afterthought.

Keep those four characteristics in mind and every tip below follows naturally.

Tip 1: Map the site before you write a scraper

Spend an hour understanding the structure before coding. Look for the cheapest path to the data:

• Category + location URLs. Most directories expose predictable listing pages like /plumbers/chicago-il. If so, you can enumerate categories × cities instead of driving the search box.
• Sitemaps. Check /sitemap.xml and any sitemap index. Directories frequently publish sitemaps that link every listing — a goldmine that skips the search entirely. See our guide to crawling a sitemap.
• A hidden JSON/API endpoint. Open your browser's Network tab and watch what loads as you scroll or paginate. Many directories render listings from a JSON or GraphQL response you can call directly — cleaner and faster than parsing HTML. This is the API-style scraping route, and it's the single biggest time-saver when available.

Only fall back to automating the search form when none of these exist.

Tip 2: Handle pagination deliberately

Pagination is where naive scrapers quietly lose half the data. Directories use several schemes, and you have to detect which one you're facing:

• Numbered pages — ?page=2, &start=20. The simplest: increment until you get an empty result or a repeat of the previous page.
• Offset/limit — ?offset=40&limit=20. Same idea, step by the page size.
• "Load more" buttons and infinite scroll — the page fetches more via XHR as you scroll or click. Either replay that XHR call directly (best), or drive a headless browser and scroll/click until no new items appear.
• Capped result sets — many directories cap a single search at, say, 1,000 results no matter how many exist. The trick is to narrow each query (by neighborhood, ZIP code, or sub-category) so no single query exceeds the cap, then union the results.

A robust loop stops on three conditions: an empty page, a page identical to the last, or a page count that hits the site's cap.

import httpx
from selectolax.parser import HTMLParser

def scrape_category(base_url, headers):
    seen_signatures = set()
    page = 1
    while True:
        url = f"{base_url}&page={page}"
        html = httpx.get(url, headers=headers, timeout=30).text
        cards = HTMLParser(html).css('.listing-card')
        if not cards:
            break                      # empty page -> done
        rows = [parse_card(c) for c in cards]
        sig = tuple(r["name"] for r in rows)
        if sig in seen_signatures:     # repeated page -> done
            break
        seen_signatures.add(sig)
        yield from rows
        page += 1

Tip 3: Deduplicate aggressively

Directories duplicate listings internally, and your query-based crawl will fetch the same business under multiple categories or overlapping locations. Without deduplication your "50,000 leads" might be 30,000 real businesses. Build a normalized dedup key:

• Phone number is the most reliable key — strip formatting to digits only ((312) 555-0170 → 3125550170). One business, one main line, most of the time.
• Name + address, both normalized: lowercase, trim, collapse whitespace, standardize abbreviations (St. → street, Ave → avenue), drop punctuation.
• Website domain as a secondary key.

For near-duplicates (typos, "Joe's Plumbing" vs "Joes Plumbing LLC"), use fuzzy matching on the normalized name within the same postal code. The general discipline here is covered in our guide to data normalization.

import re

def dedup_key(rec):
    phone = re.sub(r"\D", "", rec.get("phone") or "")
    if phone:
        return ("phone", phone)
    name = re.sub(r"[^a-z0-9]", "", (rec.get("name") or "").lower())
    zip5 = (rec.get("zip") or "")[:5]
    return ("name_zip", name, zip5)

def deduplicate(records):
    seen, out = set(), []
    for r in records:
        k = dedup_key(r)
        if k not in seen:
            seen.add(k)
            out.append(r)
    return out

Tip 4: Get past the anti-bot defenses

Directories are vigilant, so blend in. The core anti-blocking toolkit:

• Rotating residential proxies. Directories block datacenter IP ranges quickly. Route requests through rotating residential proxies so traffic looks like many ordinary visitors, and geo-target the country you're scraping.
• Human-like pacing. Randomize delays between requests, cap concurrency per IP, and avoid a machine-gun cadence. Politeness is also the best way to avoid a lawsuit and a permanent ban.
• Realistic headers and browser behavior. Send a genuine User-Agent, Accept-Language, and referer; when a site fingerprints hard, use a real headless browser (Playwright) so the JavaScript environment looks authentic.
• Captcha handling. When challenges appear, back off and retry from a fresh IP, or pipe them through a captcha-solving service.
• Watch for honeypots. Some directories plant hidden links or fake listings to catch bots. Don't blindly follow every link; stick to the listing selectors you validated.

The realistic split on a directory project is roughly 30% extraction logic and 70% staying unblocked — budget your time accordingly. Our overview of anti-scraping techniques goes deeper.

Tip 5: Structure the data into a clean schema

Raw scraped listings are messy. Decide your target schema first and normalize into it as you go, so the output is analysis- and CRM-ready. A solid business-listing schema:

Store it in a real database (Postgres, or even SQLite for smaller jobs), not a pile of CSVs. Because directory data is huge and changes over time, a database makes deduplication, incremental updates, and change detection tractable. Export to CSV or a spreadsheet only at the reporting stage.

Tip 6: Track freshness, don't just snapshot

Because listings go stale, the valuable pattern is incremental re-scraping: keep a scraped_at timestamp per record, re-crawl on a schedule, and diff against what you have. New businesses get inserted, closed ones flagged, moved ones updated. That's the difference between a decaying one-off list and a living dataset your sales team can trust.

A note on legality

Scraping public business data sits in a generally defensible zone — company names, addresses, and phone numbers are factual and publicly displayed — but stay careful:

• Personal data (individual names, personal mobile numbers, sole-trader details) can fall under GDPR/CCPA even in a "business" directory. Have a lawful basis and avoid collecting more personal data than you need.
• Terms of Service on most directories prohibit automated access. That's a contract issue, heightened if you have an account. Respect robots.txt and reasonable rate limits.
• Don't republish wholesale. Facts aren't copyrightable, but a directory's curated descriptions, photos, and its database as a compilation may be protected. Use the data for leads and analysis, not to clone the directory.

This isn't legal advice, and rules differ by jurisdiction — get proper counsel before a commercial rollout.

Special case: map-based directories

A huge share of local-business data now lives on map platforms rather than classic Yellow Pages sites. If your targets are on Google Maps, the extraction mechanics (place cards, infinite-scroll pagination, review data) are specialized enough that we cover them separately — and offer Google Maps scraping as a done-for-you service when you'd rather receive the leads than build the crawler.

When to buy instead of build

A one-city, one-category pull is a weekend project. Scraping national directories across thousands of category-location combinations — with proxies, captchas, deduplication, geocoding, and scheduled refreshes — is an ongoing engineering commitment. If the goal is the dataset, not the pipeline, scraping.pro delivers structured, deduplicated business-listing data on demand as a managed web scraping service, refreshed on your schedule. You define the categories and geographies; clean leads land in your CRM.

The bottom line

Scraping business directories rewards preparation over brute force: map the site to find sitemaps or hidden JSON before you touch the search box, handle pagination deliberately (and split queries to beat result caps), deduplicate on a normalized phone or name+ZIP key, blend in with rotating residential proxies and human pacing, and normalize everything into a clean, timestamped schema in a real database. Do that, and the by-product is a fresh, deduplicated lead list instead of a blocked scraper and a spreadsheet full of duplicates.