Almost everything a web scraper does is, underneath, a conversation in HTTP. Your crawler sends a request, a server sends a response, and every decision in between — which URL, which method, which headers, which cookies to replay — determines whether you get the page you wanted, a 403, a CAPTCHA, or a rate-limit wall. You can write a scraper without understanding HTTP, but you can't debug one without it. When a page loads fine in your browser yet your script gets an empty body or a block page, the answer is almost always in the difference between the two HTTP conversations.
This guide explains HTTP from a scraping practitioner's point of view: the request/response cycle, methods, headers, cookies, caching, status codes, and the parts of the protocol (chunked transfer, keep-alive, HTTP/2 and HTTP/3, HTTPS) that quietly affect how your crawler behaves. It's meant to be the reference you keep open while building or fixing a scraper. If you're brand new to the field, start with what is web scraping and come back here for the mechanics.
What HTTP actually is
HTTP — the HyperText Transfer Protocol — is a text-based, request/response protocol that a client (your browser, your scraper, a mobile app) uses to ask a server for a resource and receive a reply. It's the application-layer protocol of the web, riding on top of TCP (for HTTP/1.1 and HTTP/2) or QUIC/UDP (for HTTP/3).
Two properties matter enormously for scraping:
- It's stateless. Each request is, by design, independent of every other. The server does not inherently remember that you fetched the login page a moment ago. State — "you are logged in," "you have items in your cart" — is bolted on afterward, almost always with cookies. This is why session handling is such a recurring theme in scraping.
- It's plain and inspectable. Because HTTP is fundamentally a structured text exchange, you can read it, replay it, and forge it. Everything a browser sends, a scraper can send too — the whole craft of looking like a real client comes down to reproducing the browser's HTTP conversation faithfully.
Older material often describes HTTP as the 1999 spec (HTTP/1.1, originally RFC 2616). That's badly out of date. HTTP/1.1 was re-specified in 2014 and again in 2022 (RFC 9110–9112), HTTP/2 (binary, multiplexed; RFC 9113) has been standard since 2015, and HTTP/3 (running over QUIC instead of TCP; RFC 9114) is now widely deployed. More on why versions matter to scrapers below.
The request/response cycle
A single HTTP exchange looks like this:
- Your client opens a connection to the server (a TCP handshake, plus a TLS handshake for HTTPS).
- The client sends a request: a start line, headers, and an optional body.
- The server sends a response: a status line, headers, and usually a body (the HTML, JSON, image, etc.).
- The connection is either reused for more requests (keep-alive, the default in HTTP/1.1) or closed.
A raw request the way your scraper sends it:
GET /products?page=2 HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Referer: https://example.com/products?page=1
Cookie: session=abc123; consent=1
Connection: keep-alive
And the response that comes back:
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Set-Cookie: session=abc123; Path=/; Secure; HttpOnly; SameSite=Lax
Cache-Control: max-age=300
Server: nginx
<!doctype html> ...
Everything else in this article is a detail of those two blocks.
HTTP methods
The start line of every request is <METHOD> <path> HTTP/<version>. The method tells the server what you want to do:
| Method | Purpose | Relevance to scraping |
|---|---|---|
GET |
Retrieve a resource; parameters go in the URL query string | The workhorse — most page and listing fetches |
POST |
Submit data in the request body | Login forms, search forms, and many hidden JSON/AJAX endpoints |
HEAD |
Like GET but returns headers only, no body |
Cheap link-liveness and content-type checks without downloading the page |
PUT / PATCH / DELETE |
Create, modify, remove resources | Mostly on APIs you're authorized to use, not on public scraping |
OPTIONS |
Ask what a resource supports (used in CORS preflight) | You'll see these in a browser's network log; rarely something you send |
CONNECT / TRACE |
Tunnel establishment / diagnostics | Relevant to proxies, not to page fetching |
GET carries its parameters in the URL: /products?page=2&sort=price. There's a practical length ceiling — most browsers and servers keep URLs well under ~8,000 characters, and many CDNs cap lower — so anything large has to go in a body.
POST carries data in the request body, not the URL. The two encodings you'll meet constantly when reverse-engineering forms are application/x-www-form-urlencoded (classic HTML forms) and application/json (modern APIs and single-page-app back ends). Getting the Content-Type and body shape exactly right is the difference between a working POST and a 400. The fastest way to reproduce one is to open your browser's DevTools Network tab, find the request, and use Copy as cURL — it captures the method, URL, headers, cookies, and body verbatim, ready to translate into code. Many public sites feed their pages from JSON endpoints you can hit directly; see scraping hidden APIs for the technique.
Status codes: what the server is telling you
The response status line starts with a three-digit code. For a scraper, these codes are a diagnostic language — learn to read them and half your debugging is done.
2xx — success
- 200 OK — you got the resource.
- 201 Created / 204 No Content — common on API writes; 204 has no body by design.
3xx — redirection
- 301 / 308 (permanent) and 302 / 307 (temporary) — the resource is at another URL in the Location header. Most HTTP clients follow these automatically, but watch for redirect loops into login or consent pages, which usually mean a missing cookie.
- 304 Not Modified — your conditional request matched the cached version; no body is sent. Useful for polite re-crawling (see caching below).
4xx — client error (the request is the problem)
- 400 Bad Request — malformed request, often a wrong POST body or header.
- 401 Unauthorized / 403 Forbidden — authentication missing, or you've been blocked. A 403 on a page that loads fine in a browser is the classic anti-bot signature — your headers, TLS fingerprint, or IP gave you away.
- 404 Not Found — resource doesn't exist (or is hidden from you).
- 407 Proxy Authentication Required — your rotating proxy credentials are wrong.
- 429 Too Many Requests — you're being rate-limited. Respect the Retry-After header, slow down, and rotate identities.
5xx — server error (the server is the problem)
- 500 / 502 / 503 / 504 — internal error, bad gateway, service unavailable, gateway timeout. Sometimes genuine server trouble, sometimes an anti-bot layer returning a 503 challenge. Back off and retry with jitter rather than hammering.
A robust crawler treats status codes as control flow: follow 3xx, retry 429/5xx with exponential backoff, and route 403/401 into your anti-blocking logic (rotate proxy, refresh session, solve challenge) rather than simply erroring out.
HTTP headers: where scrapers win or lose
Headers are the metadata of a request and response, sent as Name: Value pairs. They are, for scraping purposes, the single most important part of HTTP, because the gap between a real browser's headers and a naive script's headers is exactly what bot detection looks for.
The request headers that matter most
Host— the domain you're addressing; required in HTTP/1.1.User-Agent— identifies the client software. Default clients announce themselves plainly (python-requests/2.x,Go-http-client,curl), which is an instant tell. Sending a current, real browser UA string is table stakes — but it must be consistent with everything else you send.Accept,Accept-Language,Accept-Encoding— what content types, languages, and compressions you'll take. Real browsers always send these; their absence flags a bot.Referer— the page you supposedly came from. Some sites gate content or images behind a matchingReferer.Cookie— replays stored state (see the next section). Critical for anything behind a login or a consent wall.Authorization— carries credentials/tokens for APIs (Basic ...,Bearer ...).Sec-Fetch-*andSec-CH-UA(Client Hints) — modern browsers send a family ofSec-headers describing the request's context and the browser's identity. Their absence, or values that contradict yourUser-Agent, is a strong automation signal on sites that check.
The deeper lesson: bot detectors don't just read the User-Agent — they check the completeness and ordering of your whole header set, and compare it against your TLS and HTTP/2 fingerprints. A "Chrome" UA arriving with three headers in the wrong order and a Python TLS signature is trivially caught. The goal is a coherent client, not just a spoofed string. This is a big topic in its own right; see how sites detect scraping and bypassing anti-scraping protection.
The response headers that matter most
Content-Type— the format and charset of the body (text/html; charset=utf-8,application/json). Tells your parser how to handle the bytes.Content-Encoding— the compression applied (gzip,br/Brotli,deflate,zstd). Your client must decompress it; mainstream libraries do this automatically, but hand-rolled ones sometimes forget and end up parsing garbage.Set-Cookie— the server handing you state to store and replay (below).Location— the target of a redirect.Cache-Control,ETag,Expires,Last-Modified— caching metadata (below).Retry-After— on a429or503, how long to wait before trying again. Respecting it is both polite and effective.
Cookies: adding state to a stateless protocol
Because HTTP forgets you between requests, cookies are how servers remember you. A server sends Set-Cookie in a response; your client stores it and echoes it back in the Cookie header on subsequent requests to that domain.
Set-Cookie: session=abc123; Domain=example.com; Path=/; Max-Age=3600; Secure; HttpOnly; SameSite=Lax
The attributes each matter to a scraper:
Domain/Path— which requests the cookie is attached to.Expires/Max-Age— a persistent cookie survives until then; without either, it's a session cookie that vanishes when the session ends. Long-running crawls need to handle both.Secure— only sent over HTTPS.HttpOnly— hidden from JavaScript, readable only by the server. Relevant when you're deciding whether a requests-based approach suffices or whether you need a real browser.SameSite(Strict/Lax/None) — controls cross-site sending; a modern addition the old HTTP guides predate entirely.
Cookies do the practical work of authentication, preserving login sessions, tracking, and A/B-test assignment. For scraping, the pattern is almost always: perform the login or consent step once, capture the resulting cookies, and replay them across the crawl. Most HTTP libraries give you a session/cookie jar object that does this automatically — use it rather than juggling headers by hand.
Caching: how to re-crawl politely and cheaply
HTTP has a built-in caching system that lets a client avoid re-downloading unchanged resources. For a scraper that re-crawls the same pages on a schedule, using it properly saves bandwidth and lightens the load you put on the target.
Expires: <date>— the resource is considered fresh until that timestamp. The legacy mechanism.Cache-Control: max-age=<seconds>— the modern, preferred control.max-age=600means the resource is fresh for 600 seconds (10 minutes) — note the value is in seconds, not milliseconds as some old tutorials wrongly state.Cache-Controlalso carries directives likeno-cache,no-store, andprivate.ETag+If-None-Match— the server tags each version of a resource with an opaque identifier (ETag: "a1b2c3"). On your next fetch you sendIf-None-Match: "a1b2c3"; if nothing changed, the server replies304 Not Modifiedwith no body, and you keep your cached copy.Last-Modified+If-Modified-Since— the timestamp equivalent of the ETag dance.
For change-detection scraping — "has this listing/price changed since yesterday?" — conditional requests with ETag/Last-Modified are exactly the right tool: a 304 is a cheap, definitive "nothing new." (Note that HTTPS responses are often marked non-cacheable by intermediaries, but end-to-end conditional requests still work.)
Chunked transfer and streaming
When a server generates a response whose length it doesn't know up front, it uses chunked transfer encoding:
Transfer-Encoding: chunked
The body arrives as a sequence of size-prefixed chunks, terminated by a zero-length chunk. Your HTTP library reassembles this transparently, so you rarely handle it by hand — but it's worth knowing when you stream large downloads, and it's why you can't always rely on a Content-Length header being present. (Transfer-Encoding can also apply gzip/deflate compression, though compression is more commonly signaled with Content-Encoding.)
Real-time transports: from Comet to WebSockets and SSE
Old HTTP write-ups spend time on "Comet" — hacks to push data from server to client over a request/response protocol, via polling (client asks repeatedly) and long-polling (client asks once, server holds the connection open until it has something). These techniques still exist, but the modern web has purpose-built transports:
- WebSocket — a full-duplex channel established by "upgrading" an HTTP connection; the client and server then exchange messages freely in both directions. This is what powers live chat, dashboards, trading tickers, and much live-updating content today.
- Server-Sent Events (SSE) — a one-way server-to-client stream over a single long-lived HTTP response (
text/event-stream).
For scraping, this matters because a growing share of dynamic data never arrives as a normal page — it streams in over a WebSocket after load. When the HTML looks empty and the data appears "live," check the DevTools Network → WS tab; you may be able to connect to the same socket directly, or you may need a real browser via Playwright or Puppeteer to capture the feed.
HTTPS: HTTP over TLS
HTTPS is ordinary HTTP wrapped in encryption. The encryption is provided by TLS (Transport Layer Security — the modern successor to the old SSL; you'll still hear "SSL" used loosely). Before any HTTP is exchanged, client and server perform a TLS handshake: they agree on a cipher, the server presents a certificate proving its identity, and they derive session keys. From then on the HTTP request and response travel encrypted.
Two consequences for scrapers:
- You can't sniff HTTPS off the wire without the keys. To inspect your own scraper's HTTPS traffic you use a man-in-the-middle proxy that presents its own certificate (mitmproxy, Fiddler, Charles) or log the session keys (
SSLKEYLOGFILE) for a tool like Wireshark. See our Wireshark as an HTTP sniffer walkthrough for the packet-level view. - The TLS handshake is itself a fingerprint. The exact list and order of cipher suites and extensions your client offers forms a signature (JA3/JA4). Python's default stack produces a different signature than Chrome, and anti-bot systems compare that signature against your
User-Agent. This is why simply setting a browser UA on arequestscall isn't enough on well-defended sites — the TLS layer betrays you before HTTP even starts.
HTTP versions and why they matter to scrapers
- HTTP/1.1 — text-based, one request at a time per connection (with keep-alive reuse). Simple to read and forge; still everywhere.
- HTTP/2 — binary framing, multiplexing many requests over one connection, header compression (HPACK). Faster, but the way a client negotiates and orders HTTP/2 frames and pseudo-headers is another fingerprint that detectors compare against your claimed browser.
- HTTP/3 — runs over QUIC (UDP) instead of TCP, folding the transport and TLS handshakes together for lower latency. Increasingly common on large sites.
The takeaway: a modern browser speaks HTTP/2 or HTTP/3 with a characteristic fingerprint, so a high-fidelity scraper aiming to blend in should ideally match the protocol version and behavior of the browser it's impersonating — not just its headers. Libraries and tools that emulate real browser TLS+HTTP/2 signatures exist precisely for this reason.
Tools for working with HTTP
To see, capture, and replay HTTP while building a scraper, from low-level to high-level:
- Browser DevTools (Network tab) — Chrome, Firefox, and Edge all show every request a page makes, with headers, cookies, timing, and the invaluable Copy as cURL. Your first stop for reverse-engineering a site.
- cURL and HTTPie — command-line HTTP clients for firing off and scripting requests. HTTPie is the friendlier, colorized one.
- Postman / Insomnia / Bruno — GUI API clients for building, saving, and iterating on requests, especially POSTs and authenticated APIs.
- mitmproxy, Fiddler, Charles, Proxyman — intercepting proxies that sit between your scraper (or phone) and the internet and decrypt HTTPS, so you can watch and modify the exact traffic your code produces. The most practical tools for debugging a scraper's requests.
- Wireshark and tcpdump — packet-level network analyzers. Overkill for everyday request inspection but unmatched for diagnosing TCP resets, TLS handshake failures, and non-HTTP protocols.
The old-guard tools some tutorials still list — Firebug, HttpFox, Opera Dragonfly — are long dead; the built-in DevTools replaced all of them.
Putting it together: the scraper's mental checklist
When a scrape fails, walk the HTTP conversation:
- Method and URL — are you sending the right method with the right query or body?
- Status code —
403/401means blocked or unauthenticated;429/503means slow down;3xxmeans follow the redirect (and check why). - Headers — do yours match a real browser in completeness and order? Is
Accept-Encodinghandled (are you decompressing)? - Cookies — did you capture and replay the session/consent cookies?
- HTTPS/fingerprint — on a stubborn
403that survives correct headers, suspect TLS/HTTP2 fingerprinting. - Content type — are you parsing HTML as HTML and JSON as JSON, at the right charset?
Master those and most scraping failures become quick reads rather than mysteries.
If you'd rather not maintain the HTTP plumbing, session handling, and anti-bot evasion yourself, this is exactly what our team does day to day — scraping.pro runs it as a managed web scraping service, delivering the clean data instead of the moving parts. When you are building it yourself, this page is here to keep the fundamentals within reach.
FAQ
Do I need to know HTTP to scrape with a high-level library? You can start without it, but you'll hit a wall the first time a page behaves differently for your script than for your browser. Every serious debugging session comes back to comparing HTTP conversations.
Why does a page load in my browser but return 403 to my script?
Almost always a client-fidelity gap: missing or mis-ordered headers, no session cookies, a giveaway User-Agent, or a TLS/HTTP2 fingerprint that doesn't match the browser you're claiming to be.
GET or POST for a search form?
Whatever the site uses — inspect the real request in DevTools. Search forms are often GET (parameters in the URL) but plenty of modern ones POST JSON to an API endpoint.
How do I inspect HTTPS traffic from my own scraper?
Run it through an intercepting proxy like mitmproxy, Fiddler, or Charles, which decrypt HTTPS by presenting their own certificate, or log TLS keys via SSLKEYLOGFILE and read them in Wireshark.
What's the single most important header for not getting blocked?
There isn't one — detectors look at the whole coherent picture. That said, a realistic, current User-Agent backed by a complete, correctly ordered header set and matching fingerprints is the baseline.