Scraping Business Directories: Tips and Tricks

In the first quarter of 2025, 284,000 private establishments in the United States closed for good, taking 809,000 jobs with them. In the fourth quarter of the same year, 338,000 opened. Those are the US Bureau of Labor Statistics Business Employment Dynamics figures, and four quarters at that rate means more than a million dead records a year sitting in directories that have not caught up. Whatever you extract today is a snapshot of a population turning over underneath you, and the directory hands you the closed restaurant with the same confidence as the open one.

Getting HTML out of Yellow Pages or Yelp is a solved problem. Knowing which of the fifty thousand rows you now own are real, which are the same company counted three times, and which were true last March is the part that takes the work. Every price, cap, version and license below was read from the vendor's own pricing page, documentation or repository on 10 August 2026. Where a widely repeated number turned out to be wrong, including in the earlier version of this article, it is corrected in place.

Why directories are their own kind of hard

Directories behave differently from ordinary websites, and three traits drive almost everything else.

• The data is query-driven, not page-driven. There is no tidy list of URLs to crawl. You reach records by searching, usually a category plus a location, and the site decides how many it will show you. Your first job is not writing a parser. It is working out an enumeration of queries whose union covers the population you want, which is the part most projects underestimate. A national pull is not one crawl. It is a few hundred thousand small ones, each of which can fail quietly.
• The data is the product, so they defend it. Rate limits, IP reputation, TLS fingerprinting, challenge pages and named blocks in robots.txt are normal here. Plan for them in the first hour, not after the first ban.
• The dataset is hard to size from outside. Expect a reconnaissance script whose only output is an estimate.

Before you scrape, check whether someone already published it

Most directory-scraping guides skip this, and skipping it is expensive. A large slice of the world's business listings is already a bulk download: deduplicated across providers, normalized, free.

Overture Maps Foundation publishes a places theme monthly. The July 2026 release carries roughly 75 million places from nine providers, led by Meta at 60.6 million, Microsoft at 6.3 million and Foursquare at 4.7 million. Records carry names, addresses, categories, websites, phones, socials, emails and brand where the contributing source had them, plus a confidence score that the schema defines as "a score between 0 and 1 indicating how confident we are that the place exists". A zero is not missing data. It means permanently closed. The theme is published under CDLA Permissive 2.0 for most contributors and Apache 2.0 for Foursquare, and it contains no OpenStreetMap data.

Reading it takes no data engineering team. The overturemaps CLI, version 1.0.1 as of 30 June 2026, pulls a bounding box straight out of the cloud-hosted GeoParquet. DuckDB reads the same files over S3:

INSTALL spatial; INSTALL httpfs;
LOAD spatial; LOAD httpfs;
SET s3_region = 'us-west-2';

SELECT id,
       names.primary AS name,
       confidence,
       phones[1]     AS phone,
       websites[1]   AS website,
       addresses[1]  AS address
FROM read_parquet(
       's3://overturemaps-us-west-2/release/2026-07-22.0/theme=places/type=place/*',
       hive_partitioning = 1)
WHERE bbox.xmin BETWEEN -87.94 AND -87.52
  AND bbox.ymin BETWEEN  41.64 AND  42.02
  AND confidence > 0.5;

Pin the release identifier the way that query does, or read it from the STAC catalog. The schema moves: the category representation was reworked into taxonomy and basic_category, so a query written against last year's column names fails against this year's files. Foursquare Open Source Places is worth pulling alongside it, under Apache 2.0, for its date_closed column and its unresolved_flags on records the pipeline distrusts.

Neither gives you complete coverage of small local trades, current opening hours, or review text. Both give you a baseline population to diff your scrape against. Finding out on day one that your crawl returns two thirds of what Overture lists for the same metro is a better use of a Tuesday than finding the same gap six months later.

And most of the rest is for sale, at a price you can look up

Two paid routes matter, and both are priced in a way that decides your architecture for you.

Yelp sells the Places API, formerly Fusion, on three tiers: Base at $229 a month, Enhanced at $299, Premium at $643. Each includes 30,000 calls a month, with overage per 1,000 calls at $5.91, $6.57 and $14.13, and a daily ceiling of 5,000 calls resetting at midnight UTC. The number that matters more than any of those sits in the FAQ: "You may cache Yelp Places API content for a maximum of 24 hours." Business IDs can be stored indefinitely; everything hanging off them cannot. This API is built to decorate a live product, not to fill a CRM, and no tier changes that. Current figures are on Yelp's pricing page.

Google Maps Platform prices Places per SKU, and the tiering is where budgets die. On the pricing page as of 7 August 2026, Text Search Pro and Nearby Search Pro are $32 per 1,000 calls with 5,000 free events a month. Place Details Pro is $17 per 1,000 with 5,000 free, Place Details Essentials $5 per 1,000 with 10,000 free. The IDs-only SKUs are free and unlimited, a real lever if your job is reconciliation rather than enrichment. Geocoding runs $5 per 1,000 with 10,000 free, Address Validation Pro $17 per 1,000 with 5,000 free.

Then read section 3.2.3(a) of the Maps Platform terms, where the customer agrees not to "pre-fetch, index, store, reshare, or rehost Google Maps Content outside the services" and, in the same list, not to "copy and save business names, addresses, or user reviews". Section 14.3 permits caching latitude and longitude for up to 30 consecutive calendar days, while place_id values may be cached indefinitely. The API with the best local data is licensed so that what you want to build with it is what it forbids.

Tip 1: Map the site before you write a scraper

Spend an hour on reconnaissance. You are looking for the cheapest path to the records, in this order.

• A sitemap. The protocol caps a single file at 50,000 URLs and 50MB uncompressed, and an index at 50,000 sitemaps, so a directory of any size publishes a tree of them. When one exists it skips the search interface entirely, and our guide to crawling a sitemap covers the mechanics. One correction to the earlier version of this article, which called sitemaps a goldmine and left it there: they are not reliably advertised. The robots.txt of yellowpages.com carries no Sitemap: line at all, so discovery there means guessing conventional paths.
• A JSON or GraphQL endpoint. Open the Network tab and paginate by hand. Listings that animate into place are usually arriving as JSON you can request directly, which is the API-style scraping route and still the biggest time-saver available.
• Embedded JSON-LD. Directories publish LocalBusiness structured data because Google's local business documentation requires name and address for rich results and recommends telephone, geo, openingHoursSpecification and aggregateRating on top. That block sits in a <script type="application/ld+json"> tag, it is already typed, and it survives CSS refactors that break every selector you wrote. Parse it first, fall back to the DOM.
• Predictable category and location URLs, where they exist. Verify them against robots.txt rather than assuming.

Then read robots.txt properly, because it is a standard now. RFC 9309, published in 2022, requires crawlers to parse at least 500 kibibytes of it, says the cached copy should not be used for more than 24 hours, and states that a 5xx response means the crawler "MUST assume complete disallow". Crawl-delay is not in the specification at all.

The trap that catches people on their first day. The yellowpages.com file contains a record reading User-agent: scrapy followed by Disallow: /. Scrapy's default USER_AGENT is Scrapy/VERSION (+https://scrapy.org), and a project created by scrapy startproject ships with ROBOTSTXT_OBEY = True in its settings. Point a default Scrapy project at that host and it collects nothing, correctly, quietly, with no error that looks like a block. The same file disallows /search* and /listings/ to everyone, the two paths a naive crawler reaches for first.

Tip 2: Handle pagination deliberately

Pagination is where naive scrapers lose half the data. Four schemes cover almost everything: numbered pages, offset and limit, load-more buttons and infinite scroll driven by XHR, and capped result sets. The first three are mechanical. Replay the XHR when you can, and drive a headless browser only when you cannot.

Caps are the interesting one, and here the earlier version of this article was too generous. It said directories cap a search at "say, 1,000 results". The published caps are an order of magnitude tighter. Yelp's business search takes limit up to 50 and offset up to 1000, but the documentation states the endpoint returns "up to 240 businesses" for a given search. Google's Text Search returns 20 per page and "a maximum of 60 results across all pages", and Nearby Search at most 20 per request, with no pagination and a radius ceiling of 50,000 metres. HTML front ends are looser than their APIs, but they cap too.

So the unit of work is not "plumbers in Chicago". It is a query narrow enough that its result count sits under the cap, and you find that number empirically. Split by ZIP code, neighborhood, sub-category, rating band, any facet the site exposes, until every leaf query returns fewer rows than the ceiling. Then union. A city that looked like one query becomes forty, and the forty are what make the crawl honest.

The loop below stops on three conditions and refuses to guess about a fourth:

import httpx
from selectolax.parser import HTMLParser

MAX_PAGES = 40          # nothing legitimate needs more; a paging bug does

def scrape_query(base_url, headers, client):
    seen_ids = set()
    for page in range(1, MAX_PAGES + 1):
        r = client.get(f"{base_url}&page={page}", headers=headers, timeout=30)
        if r.status_code in (403, 429):
            raise Blocked(base_url, page, r.status_code)
        r.raise_for_status()
        cards = HTMLParser(r.text).css('.listing-card')
        if not cards:
            return                                  # empty page: done
        rows = [parse_card(c) for c in cards]
        fresh = [row for row in rows if row["source_id"] not in seen_ids]
        if not fresh:                               # full repeat: the cap
            return
        seen_ids.update(row["source_id"] for row in fresh)
        yield from fresh
    raise CapReached(base_url, len(seen_ids))       # never fall out silently

Three deliberate changes from the version this article used to print. Comparing a tuple of names for the whole page misses partial overlap, which is exactly what a capped directory returns as you walk past its ceiling, so identity now lives per record. A 403 or a 429 previously parsed to zero cards and looked identical to a finished category, which is how you lose a city and never learn it happened. And a hard page ceiling converts a runaway loop into an exception with a query attached.

Code here was written against Python 3.13, httpx 0.28.1 and selectolax 0.4.11. selectolax earns the dependency: its own benchmark over 754 domains puts the Lexbor backend at 2.39 seconds against 61.02 for BeautifulSoup on html.parser. Across a million listing pages that ratio becomes a scheduling decision.

Tip 3: Deduplicate on the right key

Your query-based crawl will fetch the same business under several categories and overlapping locations, and the directory duplicates listings internally on top of that. Without deduplication, 50,000 leads might be 30,000 businesses.

The phone number is not the primary key, and the earlier version of this article was wrong to say so. It fails in both directions. Franchise networks, medical groups and anything with a central switchboard put one number on dozens of locations, so keying on it collapses a chain into a single row. Worse, directories increasingly carry call-tracking numbers that route to the business through the directory, so the same company shows a different number on every site you scrape and cross-source merging quietly fails. Google's own Business Profile guidance says the number "must be under the direct control of the business" and forbids numbers that "redirect or 'refer' users to landing pages or phone numbers other than those of the actual business". Rules that specific exist because the practice is common.

Use a ladder instead, most reliable first:

1. The source's own identifier. A Yelp business ID, a Google place_id, an Overture GERS ID. Stable, authoritative within its source, and what you re-crawl against. Store it on day one; it is unrecoverable later.
2. The registrable domain of the website, with aggregator and social hosts filtered out. Two listings pointing at the same company site are the same company far more often than two sharing a phone number.
3. Normalized name plus postcode plus street number. The street number does the work in a strip mall.
4. Phone as corroboration. A matching number raises confidence in a pair you already suspect. It does not create the pair.

import re
from urllib.parse import urlsplit

AGGREGATORS = {"yelp.com", "yellowpages.com", "facebook.com",
               "google.com", "linktr.ee", "wixsite.com"}

def domain(url):
    host = (urlsplit(url or "").hostname or "").lower().removeprefix("www.")
    return "" if not host or host in AGGREGATORS else host

def blocking_keys(rec):
    """Candidate keys. Records sharing any key go in one block for scoring."""
    if rec.get("source") and rec.get("source_id"):
        yield ("src", rec["source"], rec["source_id"])
    d = domain(rec.get("website"))
    if d:
        yield ("domain", d)
    name = re.sub(r"[^a-z0-9]+", " ", (rec.get("name") or "").lower()).strip()
    house = re.match(r"\d+", (rec.get("street") or "").strip())
    if name and rec.get("zip"):
        yield ("name_zip", name[:24], rec["zip"][:5], house.group(0) if house else "")

Blocking, then scoring inside the block, is the pattern that scales. Comparing every record against every other is quadratic, and at a million rows that is half a trillion comparisons. rapidfuzz 3.14.5, released 7 April 2026 under MIT, handles string similarity inside a block. When blocks get large or match rules get argumentative, Splink 4.0.16, maintained by the UK Ministry of Justice, does probabilistic record linkage over DuckDB and claims to link "a million records on a laptop in around a minute". It gives you match weights you can defend instead of a threshold you picked.

Normalize phone numbers to E.164 with phonenumbers, the Python port of Google's libphonenumber, at 9.0.36 as of 1 August 2026. For addresses, libpostal is still the best open parser, trained on "over 1 billion addresses" and reporting "99.45% correct full parses", but its last tagged release is v1.1, 9 May 2018. The repository is not archived and commits continue; the release stream stopped eight years ago. Nothing is broken and nothing is moving. Budget for building from source. The broader discipline is covered in our guide to data normalization.

Tip 4: Get past the anti-bot defenses

Blending in has three layers, and most write-ups only cover the first.

• Network. Directories drop datacenter ranges quickly, so rotating residential proxies are the baseline, geo-targeted to the market you are pulling. Prices are public: Bright Data lists residential traffic at $8 per GB pay-as-you-go, discounted to $4 on 10 August 2026, falling to $2.50 per GB on a $1,999 monthly commitment. Bandwidth is the cost line people forget.
• Transport. This layer has moved most since directory-scraping guides were first written. A plain requests or httpx call is identifiable from its TLS handshake and HTTP/2 settings before a byte of HTML is exchanged, no matter how good the User-Agent string is. curl_cffi, at 0.16.0 as of 1 August 2026, exists for this: it binds libcurl-impersonate and reproduces "browsers' TLS/JA3 and HTTP/2 fingerprints" for Chrome, Safari and Safari iOS. If a site returns a challenge to your client and clean HTML to your browser from the same IP, check the fingerprint before shopping for proxies.
• Behavior. Randomize intervals, cap concurrency per host, and pace against the site's capacity rather than your machine's. Where a real browser environment is required, Playwright is the current default, at roughly two orders of magnitude more CPU and memory per page than an HTTP client.

Challenges get their own budget. Back off, retry from a fresh exit, and only then reach for a captcha-solving service. A solver pointed at a systematic block is money spent to keep failing.

Cloudflare announced on 1 July 2025 that it would block AI crawlers by default for new domains and launched pay per crawl, reviving HTTP status 402 as the signal that access is available for a price. Enforcement runs after WAF and bot management rules, so a directory behind Cloudflare can now express "not free" as distinct from "not allowed". Our overview of anti-scraping techniques goes deeper on the defensive side.

On honeypots and the 70/30 rule, two things plainly. Hidden links planted to catch crawlers are real, and the defense is dull: follow only the selectors you validated, and never follow a link out of an element the page never renders. But the figure this article used to quote, that a directory project splits 30% extraction and 70% staying unblocked, has no published measurement behind it. It matches our experience and it is a fair planning heuristic. It is not a finding, and numbers like it circulate mostly because they sound right.

Tip 5: Structure the data into a clean schema

Decide the target schema before the first request and normalize into it as you go.

Store it in Postgres, or SQLite for a small job, and not in a pile of CSVs. Deduplication, incremental updates and change detection become tractable the moment there is an index and a unique constraint, and stay intractable without one. Export to CSV at the reporting stage, when someone needs a file rather than a dataset.

Tip 6: Track freshness, do not just snapshot

Keep first_seen_at and last_seen_at per record, re-crawl on a schedule, and diff. New businesses insert, changed ones update with the old value kept, and disappearances get flagged rather than deleted.

Absence is not closure. A record missing from one crawl is more often a blocked request, a changed selector, a truncated result set or a badly narrowed query than a business that shut. Delete on first absence and your dataset erodes every time the site redesigns, unnoticed until a sales team works a list that has quietly shrunk. Require several consecutive misses across different query paths before you write closed, and record which crawl each miss came from. A run that produced a miss alongside an unusual number of 403s is evidence about your crawler, not about the business.

External corroboration is cheap. Overture's confidence drops to zero for permanently closed places, and Foursquare's open dataset carries an explicit date_closed. Joining your flagged-missing set against either turns a guess into a check.

What breaks when the crawl gets big

A one-city, one-category pull is a weekend. The arithmetic changes shape around the third zero, and it is worth doing on paper first.

Query enumeration explodes first. Take 500 categories across 1,000 cities. That is 500,000 searches before a single detail page, and at the caps above a query returning the full 240 rows at 20 per page owes 12 requests. Six million listing-page fetches is a plausible national pull, with detail pages on top.

Bandwidth is the bill. Six million pages at roughly 1MB each is about 6TB. Through residential proxies at the $4 per GB rate above, that is around $24,000 in traffic alone, before compute, before storage, before a single record is cleaned. This is the number that makes people narrow their scope, and narrowing early is almost always right.

Enrichment is metered, and the free option is not a loophole. Geocoding a million records at $5 per 1,000 is $5,000; address validation at $17 per 1,000 is $17,000. The public Nominatim service is not the way around that: its policy caps you at one request per second from a single thread on one machine, bans "systematic queries", and restricts scripts that run longer than a day to four requests per minute. A million addresses at four a minute is about 174 days. Run your own instance or pay someone.

Everything above repeats on your refresh schedule. A monthly cycle multiplies the whole pipeline by twelve, which is why re-crawling only the fraction of records most likely to have changed is worth more engineering than the extraction code it supports.

Where each source stops working

Special case: map-based directories

A large share of local business data now lives on map platforms rather than classic directory sites, and the mechanics differ enough to deserve separate treatment: place cards instead of list pages, scroll-driven pagination, review payloads that dwarf the listing. We cover Google Maps scraping separately for that reason.

Two constraints shape every approach here. The 60-result and 20-result caps are why grid search over overlapping circles is the standard workaround, and why the grid gets expensive fast in dense metros. And Google's terms treat storing the output as the prohibited act rather than the request, so the compliance question is about your database, not your crawler.

A note on legality

Scraping public business data sits in a generally defensible zone. Company names, addresses and phone numbers are facts, publicly displayed. The exposure is elsewhere, and the case law is more specific than the summaries that circulate.

Two corrections to claims that circulate constantly. The first is that hiQ beat LinkedIn. It did not. The Ninth Circuit held in April 2022 that the Computer Fraud and Abuse Act does not reach scraping of publicly available pages, and that part stands. The case then ended in a stipulated judgment for LinkedIn in December 2022: hiQ liable for breach of contract, a $500,000 payment, and a permanent injunction against scraping LinkedIn. The second claim is that this settled the question against scrapers. In January 2024, in Meta Platforms v. Bright Data, the Northern District of California granted Bright Data summary judgment on Meta's breach of contract claim, finding that Meta's terms did not prohibit collection of publicly visible data while logged out. Read together, the two say the contract matters more than the statute.

• Personal data does not stop being personal because a business is involved. A sole trader's mobile number, a named owner, a partner's email. Under GDPR that is personal data needing a lawful basis. The UK ICO's PECR guidance is blunter: sole traders and some partnerships are treated as individuals for direct marketing, so you may email or text them only with consent. Corporate bodies can be contacted without it. A list scraped from a directory of tradespeople is mostly individual subscribers.
• The CCPA business-to-business exemption is gone. It expired on 1 January 2023, and plenty of lead-generation practice has not caught up.
• If you sell the data, you may be a data broker. California's Delete Act runs through the DROP platform, and from 1 August 2026 registered brokers must process consumer deletion requests through it at least once every 45 days. Published penalties are $200 per day for failure to register and $200 per day per deletion request not honored. That deadline is nine days old as this is written.
• Terms of service are a contract question, heightened if you created an account to reach the data. Respect robots.txt and reasonable rate limits, and keep the evidence that you did.
• Do not republish wholesale. Facts are not copyrightable, but curated descriptions, photographs and the compilation itself may be, and database rights in the EU and UK protect the collection independent of its contents. Use the data for leads and analysis, not to rebuild the directory.

This is not legal advice, and rules differ by jurisdiction. Get proper counsel before a commercial rollout, and get it before the crawl rather than after.

When to buy instead of build

The line is easier to draw once the numbers above are on the table. One city and one category is a weekend project and you should build it. A national pull across thousands of category and location pairs, with proxies, challenges, deduplication, geocoding and a refresh schedule, is a standing commitment measured in tens of thousands of dollars a year in infrastructure before anyone's salary.

Price the alternatives against that. Open data is free and covers the head of the distribution. A marketplace scraper such as the Apify Google Maps actor advertises from $1.50 per 1,000 places and moves the compliance question onto you. The gap between that and $32 per 1,000 from the official API is not a discount. It is the terms of service. A managed web scraping service delivers structured, deduplicated listing data on a defined schedule, and data as a service is the same trade with the refresh cadence written into the arrangement rather than into your cron table.

The question that decides it is not technical. It is whether you want to own a crawler.

Preparation beats brute force

The order of operations is the whole trick. Check the open datasets first, because 75 million places for the cost of a download changes what is left to scrape. Read robots.txt and the API terms next, because they decide what you may keep, which is a different question from what you can fetch. Find the sitemap, the JSON endpoint or the JSON-LD block before you touch the search box. Measure the result cap and split queries until every leaf fits under it. Key on source IDs and domains, treat phone numbers as evidence, block before you score. Keep two timestamps and never delete a record because one crawl missed it.

Do that and the by-product is a dataset worth more each month it runs. Skip it and the by-product is a spreadsheet of duplicates, a third of them businesses that closed last year.