HTTP vs HTTPS: Key Differences for Web Scraping

Point Python's requests at an HTTPS page and it works on the first try. Point it at a site behind a bot-management edge and you get 403, while curl on the same machine, on the same network, sending the same headers, gets 200. Nothing about the HTTP request changed. What differs is a few hundred bytes sent before any HTTP exists at all: the TLS ClientHello. That is the difference between HTTP and HTTPS that costs scraping teams the most time in 2026, and it has almost nothing to do with the padlock.

This article covers the secure-versus-plain distinction and what falls out of it. For request structure, methods, headers and status codes, the companion piece on HTTP protocol basics covers that ground. Everything below was rechecked against the RFCs, vendor documentation and release pages on 13 August 2026.

The one-letter difference, and the documents behind it

HTTP moves bytes in the clear over TCP port 80. HTTPS moves identical HTTP semantics inside a TLS session on port 443. The protocol on top does not change: same methods, same header fields, same status codes. What changes is that a third party on the path can no longer read or alter them, and that the server has to prove who it is before you send anything.

The specification behind that sentence moved more recently than most write-ups admit. For twenty-two years the https:// scheme lived in RFC 2818, "HTTP Over TLS", an Informational document from May 2000. RFC 9110 obsoleted it in June 2022 and pulled the scheme into the main semantics document, where "secured" now has a precise definition: the server "has been authenticated as acting on behalf of the identified authority and all HTTP communication with that server has confidentiality and integrity protection."

The same June 2022 publication retired the whole RFC 7230-7235 series. If an article you are reading cites RFC 2616, RFC 2818 or 7230-7235 for anything about HTTP or HTTPS, it was written against documents that have been obsolete for four years.

Here is the comparison in the form people actually want it, with the rows that have changed since the older version of this article marked out below the table.

What TLS protects, and what it still leaks

Confidentiality. Nobody sharing the path reads the URL path, the query string, headers, cookies, form fields or the response body.

Integrity. Records carry an authentication tag. A flipped bit, an injected script, a swapped binary all break the check and the connection dies rather than delivering altered content.

Authentication. The certificate chain proves you are talking to the host you asked for. RFC 9110 puts the burden on the client: it "usually relies upon a chain of trust, conveyed from some prearranged or configured trust anchor, to deem a certificate trustworthy for the identified origin server."

The leaks matter more for scraping than the protections do. TLS hides content, not the existence of a connection. An observer still gets the destination IP address, the size and timing of every record, and, on most connections, the hostname in cleartext in the Server Name Indication extension. Response sizes alone identify which page of a small site you fetched.

RFC 9849, published in March 2026, closes the SNI hole. Encrypted Client Hello encrypts the inner ClientHello under a public key the client fetches from DNS, and it "protects the SNI and other potentially sensitive fields, such as the Application-Layer Protocol Negotiation (ALPN) list." The RFC is candid about its limits: the destination IP stays visible, DNS queries stay visible unless you also run DoH or DoT, and handshake timing and length still leak. ECH is not anonymity, it is one field fewer.

Where the knowledge stops. What fraction of the web has ECH actually switched on is not something we could establish from a primary source. Browser support has shipped and the standard is four months old. Treat any specific ECH adoption percentage you see quoted as unsourced.

"SSL" is a brand name for a protocol that is illegal to negotiate

People still order an "SSL certificate", and CAs still sell one under that name. The protocol has been gone for years. SSL 2.0 was prohibited by RFC 6176, SSL 3.0 deprecated by RFC 7568, and RFC 8996, published March 2021 as BCP 195, finished the job for TLS 1.0 and 1.1. That one document updates 84 other RFCs, changing every "SHOULD NOT negotiate" into "MUST NOT negotiate", and obsoletes RFC 7507 and RFC 5469 outright.

What secures connections today is TLS 1.2 or TLS 1.3. The certificate file is the same object either way: a public key plus identity information, signed by a Certificate Authority. Domain Validation proves control of the domain. Organisation and Extended Validation add vetted company details that no browser has surfaced in its address bar since 2019. For encryption, the three are identical.

Certificates: nearly everything about them changed in 2025 and 2026

This is the section that dates an article fastest, and most competing pages are still describing 2021.

Certificate lifetimes are collapsing on a published schedule. CA/Browser Forum ballot SC-081v3 passed in April 2025 and set three steps. From 15 March 2026 the maximum validity for a public TLS certificate is 200 days, down from 398. From 15 March 2027 it is 100 days. From 15 March 2029 it is 47. Domain validation data reuse falls on the same schedule, from 398 days to 10. DigiCert began issuing at 199 days on 24 February 2026, running one day under the cap so a clock skew cannot push an issuance over the line.

Renewal is now measured in days, not years. Let's Encrypt publishes three profiles. classic is the 90-day default. tlsserver is 45 days and drops the Common Name, Key Encipherment key usage and Subject Key ID, following current Baseline Requirements. shortlived is 160 hours, roughly six days, which qualifies as a Short-Lived Subscriber Certificate and therefore carries no revocation information at all. Their tlsclient profile was discontinued on 8 July 2026.

OCSP is dead at the largest CA. Let's Encrypt wound it down on a published timetable: 30 January 2025 for new Must-Staple requests, 7 May 2025 for dropping OCSP URLs from issued certificates, and 6 August 2025 for switching off the responders. CRLs replaced it, on the reasoning that they "have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration."

The cost argument is over. A DV certificate from Let's Encrypt costs nothing and issues over ACME in seconds. ZeroSSL's free tier gives three 90-day certificates with ACME and REST API access, and its cheapest paid plan is $14.99 a month billed yearly for unlimited ones. Its pricing page still advertised "annual certificates" on 13 August 2026, five months after the 200-day cap took effect, which is a fair warning about how far CA marketing copy lags the ballots.

Where this breaks for a scraper. Three failure modes follow directly from the shrinking lifetimes.

• Any code that pins a certificate fingerprint against a target now breaks several times a year instead of once. Pin the CA, or pin nothing.
• Stale CA bundles in old container images produce CERTIFICATE_VERIFY_FAILED against sites that are perfectly configured. The failure looks like the target's fault and is not.
• Container clock skew produces "certificate is not yet valid" errors that are indistinguishable from an expired certificate at the log level. With six-day certificates in the field, the window in which a wrong clock matters has narrowed from months to hours.

Certificate Transparency makes your target's hostnames public

This is the part of HTTPS that comparison articles skip, and it is the one with direct scraping value.

Chrome's Certificate Transparency policy states that "in CT-enforcing versions of Chrome, all publicly-trusted TLS certificates are required to be CT Compliant to successfully validate." Compliance means the certificate was submitted to public, append-only, Merkle-tree-backed logs before it was used. The Certificate Transparency project counts 2,563,437,540 certificates logged since 2013.

The consequence: every hostname a target has ever obtained a public certificate for is in a public database. Staging environments, internal admin panels, regional subdomains, forgotten API hosts and the exact date each one first appeared. Search interfaces exist at crt.sh and at Censys, both of which run as CT monitors. Neither is a link here because crt.sh disallows automated fetching in its robots.txt, and the right way to use it is by hand or through its documented API.

Two limits keep this honest. A wildcard certificate publishes *.example.com and hides every specific name under it, so heavy wildcard users leak far less. And a name in a CT log proves a certificate was issued, not that the host is reachable today. Treat CT output as a candidate list to resolve and probe, not as an inventory.

The handshake stopped being the tax it used to be

Old comparisons treat the TLS handshake as a performance penalty you accept for safety. That framing died with TLS 1.3.

• One round trip instead of two. TLS 1.3 sends the key share in the first flight, so the handshake completes in one RTT.
• Zero round trips on resumption, with a catch. 0-RTT early data can ride along with the ClientHello on a repeat connection. It is replayable by design, which is why it belongs on idempotent GETs and nowhere near a request that changes state. A scraper that puts a pagination cursor into 0-RTT data will eventually get duplicate rows and blame the target.
• Bulk encryption is close to free. AES-NI has been in ordinary CPUs since 2010, and ChaCha20-Poly1305 covers the machines without it.
• The fast protocols are the encrypted ones. HTTP/2 multiplexes many requests over one connection. HTTP/3 runs on QUIC over UDP, folding transport and cryptographic setup into a single handshake and removing head-of-line blocking at the transport layer. W3Techs put HTTP/3 at 40.1% of all websites in August 2026.

A correction to the earlier version of this article. It said HTTP/2 and HTTP/3 "only run over TLS." That is right for browsers and wrong for the protocols. RFC 9113 defines two ALPN identifiers, h2 over TLS and h2c over cleartext TCP, and notes that the h2c Upgrade mechanism "was never widely deployed and is deprecated by this document." No browser ever shipped cleartext HTTP/2, and prior-knowledge h2c does turn up between reverse proxies and origins. HTTP/3 is the absolute case: RFC 9114 runs only on QUIC version 1, and "QUIC version 1 uses TLS version 1.3 or greater as its handshake protocol." There is no cleartext HTTP/3 and no way to build one.

The newest way to look like a bot is to have no post-quantum key share

This is the change with the shortest half-life, and no HTTP-versus-HTTPS article we found mentions it.

Browsers now offer a hybrid post-quantum key agreement, X25519MLKEM768, combining classical X25519 with NIST-standardised ML-KEM. Cloudflare reported that over 60% of HTTPS traffic reaching its network in February 2026 was post-quantum encrypted, having crossed half of human-initiated traffic in late October 2025. Origins are the reverse: roughly 10% supported post-quantum key agreement in February 2026, up from under 1% at the start of 2025.

Client tooling caught up in stages. OpenSSL 3.5, released 8 April 2025, added ML-KEM, ML-DSA and SLH-DSA. Chrome introduced the X25519Kyber768 and later X25519MLKEM768 groups in versions 124 and 130, according to the curl-impersonate patch notes.

Put those two facts together and the scraping consequence is immediate. A ClientHello that advertises itself as recent Chrome through its cipher list and extension set, but offers no ML-KEM group, is describing a browser build that does not exist. That is not a subtle signal. It is a contradiction inside a single packet, available to any middlebox that parses the handshake, before your carefully rotated User-Agent is ever read. The same reasoning covers the other direction: a Python client on an old OpenSSL cannot offer the group at all, so upgrading the impersonation library is the fix, not tuning headers.

One side effect: an ML-KEM key share pushes the ClientHello past a single packet. Handshakes that used to fit in one datagram no longer do, which surfaces against old middleboxes as a hang rather than a clean error.

Where the knowledge stops. Whether any specific anti-bot vendor scores the absence of a post-quantum group is not published by any of them, and we found no independent measurement. The structural argument stands on its own: the mismatch is machine-checkable and cheap to check.

Where plain HTTP still lives

Browsers are finishing the job. Firefox has had HTTPS-First on by default since version 136, attempting an upgrade on every navigation and falling back quietly when no HTTPS version answers. Chrome announced that "Always Use Secure Connections" becomes the default in Chrome 154, releasing in October 2026, after a preliminary rollout in Chrome 147 in April 2026 for users on Enhanced Safe Browsing.

Google's numbers explain why the change is safe now and was not before. HTTPS usage for public sites runs from roughly 97% on Linux to over 99% on Android and Mac, and has sat in the 95-99% band since about 2020. In testing, "the median user sees fewer than one warning per week, and the ninety-fifth percentile user sees fewer than three warnings per week."

The default deliberately excludes private and local network addresses, which is where most surviving plain HTTP actually is: routers, printers, NAS boxes, lab equipment, internal dashboards, embedded devices whose firmware was frozen a decade ago. If your crawl targets are internal systems or IoT interfaces, HTTP is still your normal case. On the public web it is the exception, and often only a redirect hop on the way to 443.

What changes in your scraper when the target is HTTPS

TLS fingerprinting is the block you cannot see in your headers

During the handshake your client advertises a specific ordered set of TLS versions, cipher suites, extensions, elliptic curves and signature algorithms. Servers hash that into a fingerprint and compare it against known clients. A default requests or urllib handshake resembles nothing a human ever browses with, so header work alone does not save you. This layer sits alongside everything else in anti-scraping protection, and the full defensive stack is laid out in how sites protect themselves from scraping.

A second correction to the earlier version. It wrote "JA3/JA4" as though the two were interchangeable. They are not, and the older one is retired. Salesforce archived the JA3 repository on 1 May 2025, with the README stating that "the project is no longer being actively maintained by Salesforce" and pointing at FoxIO. The reason is technical, not administrative: Chrome began permuting ClientHello extension order, so one Chrome install emits many different JA3 hashes. JA4 sorts ciphers and extensions and survives that. If you are building detection or evasion against JA3 in 2026, you are working against a hash that stopped identifying anything.

Note the licensing split before you build on JA4+: JA4 itself is BSD 3-Clause, while JA4S, JA4H, JA4T and JA4X sit under the FoxIO License 1.1, which is permissive for internal use and requires an OEM licence to monetise.

The current impersonation tooling, checked on 13 August 2026:

• curl_cffi 0.16.0, released 1 August 2026. The Python binding most people use, with 37 fingerprint presets, HTTP/3 fingerprint support since 0.15.0, and the option to supply your own JA3 and Akamai HTTP/2 strings for targets it does not ship.
• curl-impersonate 2.0.0, the lexiforest fork, built on curl 8.21.0. Targets include Chrome 146, Firefox 147, Safari 26.0.1 and Tor 14.5, with QUIC and HTTP/3 fingerprints in the build. The original lwthiker repository stalled, so a stack pinned to it impersonates browsers nobody runs.
• tls-client v1.15.1, released 8 June 2026. Go, with profiles up to Chrome_144 and QUIC and HTTP/3 fingerprints intended to match real browsers.

# curl_cffi 0.16.0, released 1 August 2026, built on curl-impersonate 2.0.0.
# impersonate= replaces the entire TLS and HTTP/2 profile, not just the headers.
from curl_cffi import requests

r = requests.get(
    "https://example.com/api/products",
    impersonate="chrome",   # pin an exact build in production, not the moving alias
)
print(r.status_code, r.json())

The failure that catches people is incoherence. If you impersonate Chrome 146 and send a User-Agent claiming Chrome 131, you have handed the server a contradiction it can check in one comparison. Pin the impersonation target and generate the User-Agent from it.

The HTTP version is part of the fingerprint too

Above TLS, HTTP/2 has its own tell: the SETTINGS frame values, the window update increment, priority frames and pseudo-header order. A client stuck on HTTP/1.1 against a site that serves HTTP/2 to every real visitor stands out without sending a single suspicious header. Use a client that negotiates h2 through ALPN.

HTTP/3 is the softer spot on both sides. Fingerprinting QUIC transport parameters is possible, and anti-bot coverage of it lags HTTP/2 badly. That cuts both ways: less scrutiny, and less cover to blend into if the target does not offer h3 at all.

Certificate errors, and why verify=False is the wrong reflex

You will hit expired certificates, self-signed certificates, incomplete chains missing an intermediate, and hostname mismatches. Blanket-disabling verification turns all four into silence, and it also turns off the one protection that stops a transparent proxy on your own egress path from rewriting responses.

Diagnose first. An incomplete chain is the target's bug, fixable on your side by supplying the intermediate. An expired certificate on a site that renews every six days usually means their automation failed an hour ago and will recover. A hostname mismatch on a scraper running through a proxy often means the proxy, not the site. If an exception is genuinely needed, scope it to the one host, in code, with a comment naming the date you last checked it.

Reading encrypted traffic needs a proxy and a locally installed root

To find the JSON API a page calls quietly, route the browser through an intercepting proxy and install its CA certificate locally. Fiddler, Charles and mitmproxy all do this. The current mitmproxy is 12.2.3, released 12 May 2026, requiring Python 3.12 or later.

This works precisely because Chrome's CT requirement covers publicly-trusted certificates. A root you added yourself is not publicly trusted, is not subject to CT, and its certificates validate for you and nobody else.

Two constraints are worth planning around.

HTTP/3 interception is partial. mitmproxy's own documentation says its HTTP/3 functionality "is available in reverse proxy, local and WireGuard mode", supports QUIC version 1 only, and "has only been extensively tested with cURL. Other implementations are likely to exhibit bugs." The practical move when a site serves h3 is to disable QUIC in the browser so traffic falls back to TLS with HTTP/2, which every intercepting proxy handles well.

Mobile apps are a different problem entirely. Android's network security configuration documents that apps targeting API level 23 and lower trust the user-added CA store, and everything from API level 24 up does not. Installing your proxy root on the device changes nothing for a modern app. You need a network-security-config that trusts user certificates, which means repackaging the APK, or a debug-overrides block, which additionally bypasses pinning: "certificate pinning is not performed when the server's certificate chain uses one of these debug-only trust anchors." Against a pinned production app on an unrooted device, an intercepting proxy simply does not work.

Your proxies are still visible by address

TLS hides the URL and the payload from the network. The destination IP is not hidden, and neither is the SNI hostname unless ECH is in play on both ends. Encryption protects the content of your request, never the fact that you made it, and never which network you made it from. That is what rotating proxies for scraping address, and no amount of TLS work substitutes for them.

What breaks at ten thousand pages a night

A single HTTPS request costs nothing worth measuring. The differences show up when the request count does.

Connection reuse becomes the whole performance story. A fresh TCP plus TLS 1.3 handshake is two round trips before the first byte of your request leaves. At 80 ms of latency that is 160 ms of dead time per connection. Across ten thousand pages with no connection reuse, that is over 26 minutes spent on handshakes alone. Session pooling erases most of it, which is why a naive loop that builds a new client per URL is slower than the same loop rewritten around one session.

Session resumption is itself a fingerprint. Reusing a ticket across hundreds of exit addresses tells a server that those addresses are one client. Reusing nothing when a real browser would resume is equally distinctive. Both directions are checkable, and neither is fixed by rotating IPs.

HTTP/3 needs UDP egress. Plenty of datacenter networks and proxy providers pass TCP and quietly drop or fail to route UDP. Your client falls back to HTTP/2 without telling you, and starts diverging from the browser profile you spent time matching. Test that h3 actually negotiates through your proxy path rather than assuming it does.

Target certificates now change several times a year. Anything that caches a leaf certificate or alerts on certificate change generates steady noise once a target moves to short-lived issuance. Alert on issuer and validity failures instead.

Keeping a browser-accurate TLS stack current is a standing maintenance line, not a one-time setup: a new Chrome ships roughly monthly, and each one moves the profile you are matching. Some teams carry that in house through managed extraction; others buy the output as delivered datasets and skip the treadmill. The wrong answer is to build the stack once and assume it stays accurate.

Myths from the HTTP era, corrected

Several claims still circulate that were defensible a decade ago.

"Crawlers cannot handle encrypted pages." Every serious crawler speaks HTTPS, including Googlebot. Google announced HTTPS as a ranking signal in August 2014 and described it at the time as very lightweight, affecting fewer than 1% of global queries. Note what that is not: there is no documented penalty applied to HTTP. The earlier version of this article's table said "slight penalty", which overstates a tie-breaker into a punishment.

"You cannot cache HTTPS." Browser caches store decrypted responses like any other. CDNs terminate TLS at the edge and cache at enormous scale. The original observation applied to shared network proxies that cannot see inside the encryption, and those were a niche in 2014 and are close to extinct now.

"HTTPS breaks name-based virtual hosting." Server Name Indication has been a standard TLS extension since the early 2000s. One IP address serves thousands of HTTPS sites, the client names the host it wants during the handshake, and the server picks the right certificate. ECH now encrypts that name.

"The padlock means the site is safe." It means the connection is encrypted and the certificate validated for that hostname. Phishing operators get free DV certificates from the same automated CAs everyone else uses. The padlock says the channel is private, not that the other end deserves your data.

"HTTPS hides which sites you visit." It hides which pages. The destination IP is in every packet header, and the hostname is in cleartext in SNI unless ECH is deployed at both ends and you are also using encrypted DNS.

"HTTPS is slower." Measured end to end against a real site, usually the opposite, because the sites that terminate TLS are the ones serving HTTP/2 and HTTP/3.

What actually needs to change in your client

The scheme difference is one letter and one port. The operational difference is that on an HTTPS target your client introduces itself twice, once in the ClientHello and once in the HTTP request, and the two have to tell the same story.

Match the TLS handshake to a browser build that exists, using a library updated within the last few months. Offer a post-quantum key share if you are claiming to be a browser that offers one. Negotiate HTTP/2, and check whether HTTP/3 survives your proxy path. Keep your CA bundle current and resist verify=False as a diagnosis. Install a proxy root when you need to see inside the traffic, and expect that to fail against mobile apps that pin.

Everything else in this article is context for those five decisions.