Web Scraping Best Practices: Core Principles

On 10 August 2026 we asked pypi.org for ten project pages in a row, one after another, with plain curl from a datacenter address. Four came back as the page we asked for, between 147 KB and 757 KB of HTML. The other six came back as 3,038 bytes carrying <title>Client Challenge</title>. All ten responses were HTTP 200.

A scraper that calls raise_for_status() recorded ten successes. A scraper that counted rows wrote six records with no name, no version and no release date, filed neatly beside four good ones. Nothing crashed, nothing logged, and the bad rows look exactly like real rows with missing fields. That is the failure this article is about, and it is the one tutorials skip.

Anyone can write a scraper that works on ten pages. The hard part is one that survives ten thousand: not blocked, not quietly collecting garbage, not dying halfway through a two-day run, not creating a legal problem. The principles below hold whatever language you use, and everything here was rechecked against first-party documentation, package source and vendor pages in August 2026.

What a broken scraper actually costs

A scraper fails in three ways, and only two of them tell you. Blocked is loud: status codes change, the error rate spikes, someone notices. Crashed is loud in the same way. Wrong is silent, and it is the expensive one, because nothing downstream can tell a genuinely empty field from a field your selector stopped finding three weeks ago.

Put a price on it. Parsing 10,000 saved pages with lxml takes about 104 seconds on one core. Re-fetching the same 10,000 at one request per second takes 2 hours 47 minutes, spent in front of whatever block detection the site runs. That ratio, roughly a hundred to one, is the argument behind most of what follows: pay seconds now instead of hours later.

The environment has also stopped being neutral. The Thales/Imperva 2026 Bad Bot Report, published 30 April 2026, put automated clients at 53% of all traffic it observed during 2025, split 40% bad bots and 13% benign automation, with AI-driven bot activity up 12.5-fold across the year. Sites are tuned for that ratio. Your well-behaved crawler arrives into a system that assumes it is one of the 40% and has to be talked out of it.

Be a good citizen

Restraint is more than manners. It is the cheapest way to stay unblocked, and it is what your position rests on if anyone asks what you were doing.

Prefer an official API, then look for the unofficial one

Check for a documented API before writing a line of parsing code. It is more stable, less legally fraught, and it survives a designer renaming a CSS class.

When there is no public API, open the devtools network tab and filter to XHR before reaching for a parser. Modern front ends fetch their own data as JSON, so the numbers often already exist in a clean, typed, paginated form nobody bothered to obfuscate. That endpoint is undocumented and can change without notice. It still beats regexing a rendered table.

robots.txt is a standard now, and your parser may not implement it

The robots.txt file at a site root tells automated clients which paths they may fetch. Since September 2022 it has been an IETF Standards Track document, RFC 9309, and the standard is more specific than the folklore around it:

• * matches any run of characters and $ anchors the end of a path.
• The longest matching rule wins, and Allow beats Disallow on an equal-length tie.
• Parsers must handle at least 500 kibibytes. Google enforces exactly that ceiling and ignores the rest. A cached copy should not be reused beyond 24 hours.
• A 4xx response means no restrictions apply. A 5xx means complete disallow, which is the opposite of what most hand-rolled code does.

Here is where most tutorials, this one's earlier version included, hand you a loaded gun. The stock example is Python's urllib.robotparser, and until May 2026 it implemented the 1996 draft rather than the RFC. We ran one four-rule file through the standard library on Python 3.11.15 and through Protego 0.6.2, the parser Scrapy uses:

User-agent: *
Disallow: /*?sort=
Disallow: /internal/
Allow: /internal/public/
Disallow: /*.pdf$

Three verdicts of four are wrong, and wrong in both directions. Two disallowed paths get crawled because the wildcard is treated as a literal character. One explicitly allowed path gets skipped because the old parser stops at the first matching line instead of the longest. That module does not merely misbehave; it misbehaves while reporting that it is obeying the file.

The fix is recent. Issue gh-138907, opened 15 September 2025, landed in the 3.14 branch on 4 May 2026 and shipped in Python 3.14.5 on 10 May 2026, with a backport to 3.13. It did not go to 3.12, which is in security-fix-only mode and stays wrong until its end of life in October 2028 per the Python release schedule. Below 3.13, use Protego and read the stdlib documentation as a description of a different module.

Two more things. Crawl-delay is not in RFC 9309, and Google states plainly that it does not support the field, so it is a request rather than a rule. Read the value anyway: it tells you what the operator considers reasonable. And the file governs fetching, not use, which is why Cloudflare bolted a Content Signals Policy onto the format in September 2025, declaring search, ai-input and ai-train preferences across 3.8 million managed domains. The signals carry no enforcement. They do record, in writing, what was asked for.

Say who you are, and throttle against the server's clock

Send a user agent that names your crawler and carries a URL where an operator can reach you; Scrapy's default, Scrapy/2.17.0 (+https://scrapy.org), is the shape to copy. A blocked crawler with a contact address sometimes gets unblocked. An anonymous one never does.

The advice to send "a request every second or two" is a number with no denominator, and it is wrong at both ends. One per second flattens a shared-hosting site whose pages take 900 ms to render. It is also 100 times slower than necessary against a CDN-fronted API answering in 8 ms, turning a twenty-minute job into a day and a half of exposure.

Pace against measured latency instead. Scrapy's AutoThrottle is the clearest published formulation: target delay is response latency divided by AUTOTHROTTLE_TARGET_CONCURRENCY, the next delay is the average of the current and target values, clamped between DOWNLOAD_DELAY and a 60-second ceiling, and latencies from non-200 responses may never reduce the delay. A slow server gets a slower crawler. That last clause is the subtle one: when a site starts erroring, a naive controller sees fast responses and speeds up.

Check what your framework actually does before you assume it is polite. Reading the defaults straight out of Scrapy 2.17.0:

CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY                 = 0
ROBOTSTXT_OBEY                 = False
AUTOTHROTTLE_ENABLED           = False

Eight concurrent requests per domain, no delay, no robots.txt, no throttling. The scrapy startproject template overrides three of those in the generated settings.py: ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1, DOWNLOAD_DELAY = 1. A generated project is polite; the same library embedded in your own script is not, and the difference is invisible unless you look.

Personal data is its own category

Public and personal are not opposites. The EDPB said so in Opinion 28/2024, adopted 18 December 2024, which sets out what a legitimate-interest basis for scraping personal data has to satisfy under the GDPR. The Dutch DPA fined Clearview AI EUR 30.5 million on 3 September 2024 for a face database assembled from publicly reachable images. Collect the minimum, keep it for a defined period, and know your basis before the first request rather than after the first complaint.

One myth deserves burying, because it appears in nearly every article on this subject. hiQ Labs did not win against LinkedIn. The Ninth Circuit rulings everyone cites were preliminary-injunction decisions about likelihood of success, the 2019 opinion was vacated by the Supreme Court and remanded after Van Buren v. United States (2021), and on the merits hiQ lost: summary judgment for LinkedIn on breach of contract in November 2022, then a USD 500,000 stipulated judgment and a permanent injunction that December, with the scraped data and the code that collected it deleted. The lesson is narrower than the myth. The Computer Fraud and Abuse Act is a weak instrument against logged-out scraping, so platforms sue on contract instead. The rest is in is web scraping legal, which is not legal advice but tells you which questions to ask.

Build scrapers that survive the run

Connections drop, servers hiccup, page structures change without warning. A scraper that assumes otherwise works until the run is long enough to matter.

1. Fetch incrementally and save as you go

Break a large job into independent pieces and write each piece to disk the moment it arrives. Holding thousands of records in memory for one final save means a connection failure at page 4,000 of 10,000 costs you 3,999 pages you already paid for.

Resumability is cheap to design in: a queue of pending URLs, a record of completed ones, a stable key per record. Two details separate resuming cleanly from resuming into a mess. Write to a temporary file and rename it into place, since rename is atomic on POSIX filesystems and a half-written JSON file is worse than a missing one. And write the completion marker last, so a crash between fetch and save reprocesses one page instead of skipping it.

2. Store the raw page first, parse it later

Separate fetching from parsing. Save the response bytes exactly as they arrive, then run extraction as a second pass over local files.

The reason is the hundred-to-one ratio from earlier. Find a structural quirk at page 900 of 1,000 and, parsing on the fly, you re-scrape everything in front of the site's block detection. With raw pages on disk you fix the selector and re-run locally, at zero extra requests.

The usual objection is storage, and it does not survive the numbers. Common Crawl's July 2026 archive holds 2.14 billion pages in 364.01 TiB of uncompressed content, which puts the average page at about 187 KB. Ten thousand of those is 1.87 GB raw. We compressed 1,464,789 bytes of real HTML and got 119,685 bytes with gzip -9 (12.24x) and 98,192 with zstd -19 (14.92x). The 1.87 GB becomes roughly 153 MB, for a run whose fetching cost three hours of wall clock.

This measurement and the parser timings later on come from one machine, Python 3.11.15 with lxml 6.1.0 and beautifulsoup4 4.14.3, over four real HTML documents of 147 KB to 757 KB fetched that morning. Your numbers will differ with page shape; the shape of the result will not.

Store more than the body. Keep the final URL after redirects, the status code, the headers and the fetch timestamp; each becomes evidence when a number is disputed six months later. The format already exists: WARC, standardized as ISO 28500:2017 and readable with warcio 1.8.1, stores request and response together. That is exactly the provenance a JSON dump of extracted fields throws away.

3. Parse strictly and validate everything

Be pessimistic. Assume the page may not have the structure you expect, and check every value before it reaches storage.

Money is not a float. The previous version of this article suggested round(float(value), 2), and that is a bug we shipped. Binary floating point cannot represent most decimal fractions, so error accumulates: adding 0.07 a hundred thousand times gives 6999.9999999921065, not 7000.00, and round(2.675, 2) returns 2.67, because the stored value sits fractionally below the midpoint. On one row nobody notices. On a reconciliation against a client's own totals, the mismatch is the first thing they see.

from decimal import Decimal, InvalidOperation
import re

def clean_price(raw):
    if raw is None:
        raise ValueError("price cell missing entirely")
    # drops currency symbols and every kind of space, including
    # U+00A0 and U+2009, which real price cells are full of
    text = re.sub(r"[^\d.,\-]", "", raw)
    if text.count(",") == 1 and text.count(".") == 0:
        text = text.replace(",", ".")      # 1299,00 -> 1299.00
    else:
        text = text.replace(",", "")       # 1,299.00 -> 1299.00
    try:
        return Decimal(text)
    except InvalidOperation:
        raise ValueError(f"unexpected price format: {raw!r}")

Encoding is the second silent corruption. HTTP dropped the ISO-8859-1 default for text/* media types in RFC 7231 (June 2014), and RFC 9110 kept it dropped. The requests library did not: get_encoding_from_headers() in version 2.34.2 still returns "ISO-8859-1" for any text/* response with no charset parameter. Because Response.text prefers a set encoding over sniffed content, a UTF-8 page that declares its charset only in a <meta> tag comes back as mojibake, and the mojibake is stable, so it reads as the site's own data. Pass r.content to your parser, or set r.encoding = r.apparent_encoding deliberately.

Then check meaning as well as shape. A date should parse, a country should be in ISO 3166-1, a currency in ISO 4217, a price inside a range you would believe. And distinguish absent from zero. Zero is a value, None is a missing value, the empty string is neither. Collapsing all three is how "out of stock" becomes "free".

Strict parsing pays twice. A spike in validation failures is your early warning that the layout changed; lenient parsing hides that until the bad rows are in production. Follow extraction with a proper data normalization pass, then validate again, since normalization is code and code has bugs.

4. Retry the way the server asked

Transient failures are normal, and there is a documented protocol for them that most retry loops ignore. Retry-After, defined in RFC 9110 section 10.2.3, carries either a delay in seconds or an HTTP date. On a 429 or a 503 it tells you exactly when to come back. Guessing is both ruder and slower.

Three things about the tooling, checked against current releases:

• requests does not retry at all by default. DEFAULT_RETRIES = 0, and even when you raise it, urllib3 retries only failed DNS lookups, socket connections and connection timeouts. Status codes are never retried unless you build a Retry yourself.
• urllib3's Retry honors Retry-After on exactly three status codes, 413, 429 and 503, with backoff backoff_factor * 2 ** (attempt - 1) capped at 120 seconds. The parameter to notice is backoff_jitter, which defaults to 0.0. Without it, a hundred workers that hit the same rate limit at the same moment sleep the same interval and hit it again together. Set it.
• Scrapy retries 429 and does not read Retry-After. RETRY_HTTP_CODES includes 408, 429 and the 5xx family, RETRY_TIMES is 2, and there is no backoff setting at all: a retried request goes back to the scheduler with its priority lowered by one, spaced only by DOWNLOAD_DELAY.

tenacity 9.1.4 gives you the pieces without the framework: wait_exponential_jitter, stop_after_attempt and stop_after_delay compose into one decorator. Whatever you use, retry only what is safe to repeat, cap elapsed time as well as attempt count, and never retry a 404 or a 403. Those are answers, not failures.

5. Watch the output, not the run

At thousands of records you cannot eyeball anything. Measure the shape of the data every run and compare it with the last.

• Fill rates per field. A field empty in 999 of 1,000 records means a wrong selector, or data that lives on another page.
• Response size distribution. The metric that would have caught the opening example instantly: 3,038 bytes against a corpus averaging 187 KB is nowhere near anything real. A floor on body size costs one line and catches challenge pages, empty templates and truncated responses, all of which arrive as HTTP 200.
• Volume and distribution deltas. Half as many rows as yesterday means a section broke; a 98/2 split where you expected 60/40 is worth chasing.
• Error and block rates by status code, including CAPTCHA hits and redirect chains ending somewhere unexpected.

Set thresholds and alert on them; a metric nobody reads does not exist.

What breaks when the job gets big

A script that handles ten pages and a pipeline that handles ten thousand a night are different programs, and the differences are not about speed.

Partial failure becomes the normal state. At ten pages the run either worked or it did not. At ten thousand, some pages always fail, and "did the run succeed?" stops being answerable. Replace it with a threshold decided in advance: good if at least 97% of URLs returned parseable records and no field's fill rate dropped more than five points.

The frontier stops fitting in a variable. A set of ten thousand URLs is nothing. Ten million is gigabytes, and a restart loses all of it. Move dedupe into SQLite or Redis early, key on the normalized URL, and strip the tracking parameters that make one page look like forty.

Parser choice becomes a budget line. We measured three common Python paths over the same 1.46 MB of real HTML: lxml.html.fromstring at 10.4 ms per page, BeautifulSoup over the lxml backend at 189 ms, and BeautifulSoup over html.parser at 270 ms. Across 10,000 pages that is 1.7 minutes of CPU against 31.5 against 45. Beautiful Soup's ergonomics are worth paying for on a hundred pages. On a nightly million they are a second machine.

Concurrency has two limits and people set one. A global cap protects your machine, a per-host cap protects the site. Sixteen workers across eight hosts is polite. Sixteen on one host is an incident.

Stay unblocked without being abusive

Politeness and staying unblocked are the same list.

• Do not fetch what you already have. Send If-None-Match with the stored ETag, or If-Modified-Since with the stored Last-Modified, and a well-behaved server answers 304 with an empty body. Conditional requests live in RFC 9110 section 13, caching in RFC 9111, both Internet Standards since June 2022. It is the largest saving on a recurring crawl and the one most scrapers skip.
• Let the site tell you what changed. Sitemaps cap at 50,000 URLs or 50 MB uncompressed per file, and lastmod is the field to read. Google uses it when it is accurate and ignores priority and changefreq outright, a fair guide to what each is worth.
• Send realistic headers. A request with no Accept, Accept-Language or Referer is a bare fingerprint. Mirror a real browser and keep the set coherent. A German residential address paired with Accept-Language: en-US is a worse signal than sending nothing.
• Rotate identities only at volume. Vary the user agent and route through rotating proxies when one address making thousands of requests would look like what it is; setup details are in proxies for scraping. Rotation is no substitute for pacing. It changes who is being rude, not whether anyone is.
• Render only when you must. A headless browser fetches every asset, runs every script and holds a process open per tab, against one connection and 10 ms of parsing for a plain request. Reserve it for genuinely dynamic content, after checking for the JSON endpoint underneath.
• Treat challenge rates as a thermometer. A CAPTCHA solving service clears one block. A rising challenge rate says the pacing is wrong, and paying per solve to push through buys your way past your own warning light.

Guard quality end to end

Deduplicate on a stable key. The same item appears on a category page, a search result and a canonical URL. Key on a site-assigned identifier where one exists, a normalized URL where it does not.

Normalize once, consistently. Dates to ISO 8601, currencies to ISO 4217 with the amount kept as an exact decimal, units converted with the original preserved, whitespace collapsed, UTF-8 everywhere.

Version your schema and your selectors. When a site changes and you adjust an XPath, record the date and the change alongside the data. Six months later someone asks why a series has a step in it, and the answer has to be findable.

Keep golden fixtures. Save a handful of real pages, one per layout variant, and run the parser against them in CI. It is the only test that catches a refactor breaking extraction before production does, and the fixtures are free if you followed the raw-storage rule.

Where these principles stop

Every list like this has an edge, and pretending otherwise is how people walk off it.

Behind a login the calculus changes. Account gating is the clearest legal line in the United States after Van Buren, and terms bind a logged-in user in a way they do not bind an anonymous visitor. Politeness does not make credentialed scraping safe.

Free text does not validate. Prices, dates and country codes have checkable shapes. A description does not, so field-level validation gives no coverage exactly where truncation and wrong-element picks hide. Sample and read them by hand, on a schedule.

Some data has no honest path. If robots.txt disallows the section, there is no API and the terms forbid extraction, nothing in this article turns that into a yes. It makes the no cheaper to find.

A deadline is a constraint like any other. If the polite pace cannot finish before the data goes stale, the real options are fewer pages, older data, or someone else's infrastructure: a managed extraction service or a delivered dataset is a schedule you buy rather than one you hope for. Adding concurrency until the site notices is not on the list.

What changed since these rules were written

The engineering advice has been stable for a decade. The context around it has not.

The web is majority automated, so defenses are calibrated for a mix in which you are the default suspect. Declaring yourself and pacing sensibly matter more than they did, not less.

Provenance became a legal question rather than a hygiene one. Bartz v. Anthropic split the halves cleanly: training on lawfully obtained books was held to be fair use, acquiring pirated copies was not, and the case settled for $1.5 billion with final approval in July 2026. In the EU, Article 53 obligations for general-purpose AI providers have applied since 2 August 2025, with the enforcement grace period for Code of Practice signatories ending 2 August 2026. The record of where each row came from, under what stated permissions, is now worth keeping. One more argument for storing raw responses with their headers and timestamps.

The permission layer around robots.txt also grew a vocabulary for use as well as access, and Content Signals is the deployed example. None of it binds anyone. Read it anyway: a crawler that respects a preference it was free to ignore is in a different position from one that never looked.

A checklist worth running

Before a scraper goes to scale, confirm:

• Checked for an official API, and for the JSON endpoint the front end calls
• robots.txt parsed by something that implements RFC 9309, not the 1996 draft
• User agent names the crawler and carries a contact URL
• Pacing derived from measured latency, with a per-host concurrency cap
• Retries honor Retry-After, use jitter, cap elapsed time, and skip 404 and 403
• Raw responses saved with status, final URL, headers and timestamp before parsing
• Money in Decimal, encoding set deliberately, absent distinguished from zero
• Per-field validation that fails loudly, plus a floor on response body size
• Resumable, with a durable record of done and pending URLs
• Fill rates, volumes, body sizes and status codes logged every run and compared with the last
• Conditional requests and sitemap lastmod used on anything recurring
• Personal data collected only where you have a basis, and only what you need

The habits that keep scrapers alive

Three ideas carry the list. Be polite, because it is cheaper than being blocked: read the file, name yourself, pace against the server's clock rather than a number you picked. Be resilient, because long runs fail in the middle: save as you go, keep the raw bytes, retry the way the protocol tells you to. Watch the output, because the expensive failure is the one that never raises an exception.

The scrapers that last are not the clever ones. They are the ones whose owners can answer, any given morning, how many pages came back, how many fields were filled, and which of yesterday's numbers moved.

Ten requests. Six challenge pages. Ten HTTP 200s.