Anti-Scraping Protection: How Modern Anti-Bot Systems Work

A request arrives. Before your application code runs, before anything touches a database, a scoring engine has already read the exit address, the shape of the TLS handshake, the order of the HTTP headers and whatever the browser volunteered about itself. It has decided how much of the page this client deserves. On Cloudflare that decision is a number between 1 and 99, computed at the edge, and the visitor never learns it happened.

The scale behind that machinery is no longer arguable. The 2026 Thales/Imperva Bad Bot Report, the thirteenth edition, puts bots at 53% of all internet traffic and bad bots alone at 40%, three points up on the year, and counts 17.2 trillion bad bot requests blocked during 2025. Daily AI-driven bot attacks in the same dataset went from 2 million to 25 million. Cloudflare, measuring its own network in March 2025, said AI crawlers by themselves generate more than 50 billion requests a day, just under 1% of every web request it sees.

Web scraping is the automated collection of data from websites: competitor prices, catalogue content, contacts, listings, stock levels. It is legal in most places when it targets publicly available data. It becomes somebody's problem when it hammers infrastructure, lifts unique content wholesale, feeds price dumping, or sets up fraud such as credential stuffing and inventory sniping. The vendor-neutral name for it is OAT-011 in OWASP's Automated Threats to Web Applications ontology, which reached version 1.3 on 17 March 2026 and catalogues 21 such events.

This article is about the machinery on the other side: what gets measured, what each measurement is worth, what the platforms cost, how each layer is defeated, and where each one stops working. Every price, version, release date and product claim below was read from the vendor's own pricing page, documentation or repository on 10 August 2026. Where something could not be checked, the text says so rather than guessing.

Three claims in the earlier version of this article did not survive that check, and they are corrected in place below: where GREASE comes from, who at HUMAN Security used to be called what, and how hiQ Labs v. LinkedIn actually ended.

What a scrape costs, and why that number decides everything

Every defensive choice below is an argument about unit cost. It helps to know what the units are before reading the arguments.

On the attacking side the dominant line item is bandwidth through addresses that do not look like a data centre. Oxylabs lists residential proxies at $6/GB on its 5 GB starter tier, sliding to $2.50/GB at a terabyte, with mobile from $3.50/GB and datacenter proxies priced per address at $0.70/IP. Bright Data advertises residential from $2.50/GB, half of a $5 list price at the time of reading, and its Web Unlocker from $1 per 1,000 requests. Challenge solving is cheap by comparison. 2Captcha's public price list puts plain image captchas at $0.50 to $1.00 per 1,000, reCAPTCHA v2 at $1.00 to $2.99, reCAPTCHA v3 at $1.45 when a score below 0.3 will do and $2.99 when it will not, and Cloudflare Turnstile, DataDome and Amazon tokens at $1.45 flat. Arkose Labs is the outlier at $1.45 to $50.

Now do the arithmetic on a real job. A million product pages at 400 KB of transferred markup each is roughly 400 GB. Through residential exits at $3/GB that is about $1,200. Suppose your defence fires a visible challenge on 2% of those requests: 20,000 solves at $1.45 per 1,000 adds $29. The bandwidth costs forty times what the captchas cost.

That ratio explains a great deal of behaviour that otherwise looks irrational. Defences that force a full browser render multiply the bytes and therefore multiply the bill. Defences that force a solve barely move it. A scraper will spend hours avoiding a JavaScript challenge and pay a solver without blinking.

Time is the second currency, and it is the one scrapers actually complain about. A solver that answers in three seconds instead of failing sounds fast until you multiply. Three seconds on 2% of a million-page crawl is 20,000 solves, which is sixteen hours of wall-clock delay unless the work is parallelised around it. One wasted second per page across a ten-thousand-page crawl is closer to three hours. Defences that cost the attacker seconds rather than cents are the ones that change schedules, and schedules are what a competitor's pricing feed actually runs on.

The defender's side of the ledger is public in exactly one place. DataDome publishes list prices, which almost nobody else in this market does: Essentials at $3,830 a month for under 100 million requests, Advanced at $8,670 for under 200 million, Premium at $10,160 for under 300 million, Enterprise from $13,270 above that. Divide the entry tier through and you get about $0.04 per thousand requests protected, against roughly $1 per thousand requests harvested through an unblocking API. The asymmetry runs the defender's way by a factor of twenty-five, and that asymmetry is the entire strategy. You are not trying to make scraping impossible. You are trying to make each page cost the other side more than it costs you.

The asymmetry inverts on small sites. That $3,830 floor is the same whether you serve 100 million requests a month or 100,000, and at the low end a subscription costs more per protected request than the scraping does. Sites below a few million monthly requests are the natural home of the free layers: Turnstile, a proof-of-work gate, a honeypot, a sane rate limit.

What signals get tracked

No serious system rests on one tell. They collect dozens of signals across the network stack and fold them into a single trust score. The lower the score, the better your odds of a challenge, a slowdown or a block. The signals sit in six rough layers, and the interesting property is not any individual layer but whether they agree with each other.

1. The address, and why it means less than it used to

The oldest layer. The system looks at reputation, at the owning ASN, and at what kind of network the address belongs to. Traffic from AWS, GCP or Azure ranges stands out instantly, because people do not browse from a hypervisor. On top of that sit request frequency and rhythm, sudden bursts, and hits on paths no human navigation would produce. Large vendors keep their own lists of addresses that misbehaved anywhere on their network, which is where the value of a big network actually shows.

Two developments have hollowed this layer out from both ends. Carrier-grade NAT means a single IPv4 address routinely fronts thousands of real mobile subscribers, so blocking one address can remove a city block of customers. On IPv6 the opposite problem applies, since one client is handed billions of addresses and rate limiting must work on the /64 prefix instead. Meanwhile residential proxy pools rent out the addresses of ordinary broadband subscribers by the gigabyte, so "looks residential" costs an attacker $2.50 and buys them the benefit of your doubt.

Address reputation is a prior, not a verdict. Treat it as evidence that shifts a score, never as a rule that decides one.

2. TLS fingerprint: JA3 is finished, JA4 is what runs now

This is the most reliable layer available, because it works before a single line of JavaScript executes and before your application sees the request. When a client opens an HTTPS connection it sends a ClientHello listing cipher suites, extensions, elliptic curves, protocol versions and ALPN values. That set, hashed, identifies the TLS stack that produced it.

The details here have moved, and most write-ups have not moved with them. Salesforce archived the JA3 repository on 1 May 2025. Its README now states plainly that "the project is no longer being actively maintained by Salesforce" and points readers to the original author's successor work at FoxIO-LLC/ja4.

JA4 is built to survive the thing that broke JA3. It composes a readable fingerprint from the protocol, TLS version, whether SNI was present, the cipher and extension counts and the ALPN value, followed by two hashes. Ciphers and extensions are sorted by hex value before hashing, so a client that shuffles their order emits the same fingerprint every time. Signature algorithms are deliberately left in their original order, because that order still carries information. GREASE values are ignored wherever they appear, which the specification repeats at every counting step.

Here is the correction. The earlier version of this article said TLS 1.3 introduced GREASE. It did not. GREASE is RFC 8701, published January 2020, and it applies to TLS 1.2, TLS 1.3 and DTLS alike. Its purpose has nothing to do with privacy or evasion: it makes clients advertise deliberately meaningless values so that servers and middleboxes which choke on unknown values get caught early, before the breakage ossifies the protocol. The reason a single browser install produced many different JA3 hashes is that browsers began permuting the order of ClientHello extensions, and JA3's MD5 ran over those fields in the order received. Sorting fixes that. Removing GREASE is housekeeping on top.

JA4 is one member of a suite of twelve: JA4S for server responses, JA4H for HTTP, JA4L and JA4LS for latency, JA4X for certificates, JA4SSH, JA4T and JA4TS for TCP, JA4TScan, and JA4D and JA4D6 for DHCP. If you plan to build on any of it, read the licensing first, because it is split. JA4 itself is BSD-3-Clause. Everything else in the suite sits under the FoxIO License 1.1, which permits academic and internal business use but requires an OEM licence to ship inside a commercial product. The repository's most recent release is dated 19 November 2025.

One layer up, the same idea applies to HTTP/2. A connection opens with a SETTINGS frame, and the values in it, the window-update increment, the presence and shape of priority information and the order of the pseudo-headers all vary between implementations in ways that are stable per browser build. A client can spoof a TLS handshake and still announce a Go or Python HTTP/2 stack one frame later. HTTP/3 over QUIC offers the same opportunity in principle, and JA4's format records which transport carried the handshake, but tooling on both sides is thinner there. That makes HTTP/3 the soft spot in the middle of this layer, for defenders and attackers alike.

The real power of this layer is not identification but contradiction. A User-Agent claiming Chrome on Windows, attached to a handshake that Python's requests produces, is not a suspicious client. It is a confessed one.

3. Headers, and the order they arrive in

Values matter less than composition. Systems check whether Accept, Accept-Language, Accept-Encoding and Referer are present and plausible, whether Client Hints such as Sec-CH-UA match the declared browser and platform, and whether the header order matches what that browser actually emits. Header order is stable per browser and per version, and hand-built HTTP clients get it wrong without knowing there was something to get right.

The cheapest catch on this layer is geography. An exit address in Warsaw, an Accept-Language of en-US and a timezone of America/Chicago describe a person who does not exist. Keeping four or five such fields mutually consistent is far harder than spoofing any one of them, which is why this check keeps earning its place.

4. The browser fingerprint

If the client runs JavaScript, the page can interrogate hundreds of environment properties:

• Canvas rendering, whose pixel-level output varies with GPU, driver and OS;
• WebGL and WebGPU, which name the graphics adapter and driver directly;
• AudioContext, where tiny differences in signal processing survive across sessions;
• installed fonts, screen geometry, timezone, language, hardware concurrency;
• the shape of the navigator object and which APIs exist at all.

Consistency rules here too. Canvas output that implies a discrete GPU, sitting next to a WebGL string reporting Intel integrated graphics, describes a machine nobody sells.

Randomisation is itself a signal. Anti-detect browsers that add noise to canvas output are caught by calling the API twice and diffing the results, since a real GPU is deterministic and the noise usually is not. A fingerprint that never changes across weeks and networks is equally odd. The detection question is not "is this fingerprint unusual" but "is this fingerprint the kind of thing a real machine produces."

5. Automation tells, including two that stopped working

Direct markers of headless browsers and automation frameworks still exist: traces of the Chrome DevTools Protocol, software rendering through SwiftShader where hardware acceleration should be, window geometry no display produces, and timings that are simply too good, such as a canvas fingerprint computed faster than a real browser could paint it.

Two tells that appear in nearly every article on this subject, including the earlier version of this one, no longer detect anything.

Missing plugins. The HTML Standard now fixes navigator.plugins to a hard-coded list of exactly five entries when the user agent supports PDF viewing, named "PDF Viewer", "Chrome PDF Viewer", "Chromium PDF Viewer", "Microsoft Edge PDF Viewer" and "WebKit built-in PDF", with two MIME types beside them. When PDF viewing is unsupported both arrays are empty. The spec says outright that these names were chosen from evidence of what websites historically searched for. There is no plugin inventory left to be missing, and code that tests for one is testing a constant.

navigator.webdriver. The flag reports that an automation protocol is driving the browser, not that the browser is headless. One command-line switch clears it, and modern headless Chrome is the same binary real users run. A check for it filters out beginners and nothing else.

What replaced them is subtler and decays faster: side effects of the DevTools Protocol itself, isolated-world naming leaks, serialisation quirks that only appear once a debugger is attached. These get patched on both sides in weeks. If your detection stack contains a hand-written list of automation tells, put a review date on it.

6. Behaviour, and the problem with it in 2026

The most human layer, and increasingly the one that settles the outcome. Models score mouse trajectories, scroll acceleration and rhythm, dwell times, keystroke cadence and navigation order against classifiers trained on real traffic for that specific site. People move a cursor untidily, overshoot targets, pause to read, and scroll in uneven bursts. Naive automation moves in straight lines at constant speed and never fumbles. No mouse movement at all during a multi-page session is suspicious by itself.

Faking this convincingly is the one bypass that cannot be bought for $1.45 per thousand, which is why the layer has held up. It has a new problem instead. A growing share of automation now acts on behalf of a specific person at that moment: an assistant fetching the page someone just asked about, a shopping agent comparing your prices, a booking agent filling your form. Those clients move a cursor because a human asked them to. A model trained to flag "no human hand here" will flag customers.

The vendors have noticed, and the category has been renamed around it. Kasada's own site cites Forrester's Q2 2026 wave for this market by its current title: Bot and Agent Trust Management. The first such wave, which Netacea still cites, was called Bot Management and nothing else. We have not read either report, so we make no claim about who placed where. The change of title alone tells you where the industry thinks the question has moved.

The defences, and what each one is actually worth

Protection is layered on purpose. Each layer removes a slice of unwanted traffic and raises the cost of the next step, and the layers are worth wildly different amounts.

Rate limiting and throttling. Caps on requests per address or session, with slower responses once thresholds are crossed. Cheap, mandatory, and stops nothing more sophisticated than a naive loop, because splitting the work across a proxy pool defeats it by construction. Profile your real traffic, find the 99th percentile, and start well above it. This is a floor, never a ceiling.

robots.txt. A declaration of which sections which crawlers may fetch, standardised as RFC 9309 in September 2022. The RFC is refreshingly blunt about its own limits: "These rules are not a form of access authorization," and the protocol "is not a substitute for valid content security measures." It works on crawlers that choose to read it. Adoption is worth knowing precisely, because the round number in circulation is soft: Cloudflare's June 2025 crawl of the top 10,000 domains found a robots.txt on 3,816 of them, roughly 38%, of which about 546 carried any directive aimed at AI bots at all. So the honest statement is not "only a third have one." It is that fewer than four in ten publish a policy, and roughly one in twenty has one that mentions the crawlers everyone is currently arguing about.

WAF and reputation lists. Known-bad patterns and known-bad networks, filtered before anything expensive happens. Blunt, cheap, and prone to collateral damage on shared addresses.

Challenges and invisible scoring. The visible puzzle is now a consequence of a low score rather than a first line of defence, and pricing reflects that. Cloudflare Turnstile carries no per-request charge at all, with the free plan capped at 20 widgets per account and 10 hostnames per widget, and an Enterprise tier lifting those to unlimited widgets and 200 hostnames. reCAPTCHA now sits inside Google Cloud Fraud Defense: Essentials is free to 10,000 assessments a month, Premium adds an $8 flat fee between 10,000 and 100,000 and $1 per 1,000 above that, and Enterprise is $1 per 1,000 on a twelve-month commitment. hCaptcha keeps a free tier with unlimited evaluations and charges $99 a month billed annually, or $139 monthly, for Pro with 100,000 evaluations and $0.99 per additional thousand.

Proof of work. The browser burns CPU on a puzzle before the page is served. Negligible for one reader, ruinous for a crawler fetching a million pages, and no perception test is involved, so the accessibility objections to visual challenges do not apply. Friendly Captcha builds on this and is priced by domain and volume: €9 a month for one domain and 1,000 requests, €39 for five domains and 5,000, €200 for fifty domains and 50,000, with a non-commercial free tier at starter limits and a WCAG 2.2 AA certification. The open-source counterpart is Anubis, which its author describes as "a Web AI Firewall Utility that weighs the soul of your connection using one or more challenges in order to protect upstream resources from scraper bots."

JavaScript challenges. The server returns a script that must run in a real browser to produce a valid token. HTTP clients without a JS engine are removed outright. There is an underrated bonus for defenders: any scraper that runs your JavaScript to pass also fires your analytics, so it appears in your own reports as an anomalous session.

Fingerprinting with ML scoring. Everything above, weighed continuously by models trained per site. DataDome claims "99.99% detection accuracy" from "1000s of AI models"; that is a vendor claim, unaudited, and we have no independent measurement to set against it.

Honeypots, mazes and tarpits. Links and form fields invisible to people, which only indiscriminate automation touches. The 2025 addition is the maze. Cloudflare's AI Labyrinth, launched 19 March 2025, serves misbehaving crawlers a set of AI-generated decoy pages linked invisibly from real ones, and the documentation notes the invisible links carry nofollow so search ranking is unaffected. It is opt-in on every plan including Free. The design point is that this is a detector as much as a punishment: anything that walks several pages deep into generated nonsense has identified itself.

Dynamic obfuscation and polymorphism. Rotating class names, markup structure and the defence code itself, so a scraper cannot be wired to a fixed template and an attacker cannot reverse the protection once and keep the result. Utility-first CSS and build-time hashing have made this much cheaper than it used to be, which is a reason to revisit it if you dismissed it years ago.

Edge versus application placement. Edge protection catches clients before the origin and saves the load. Application-level protection knows the business context and can reason about what this particular session is trying to do. Large estates run both, and the two see different attacks.

Verifying the bots you want. An allowlist keyed on a user-agent string is decorative, since anyone can send User-Agent: Googlebot. The durable method is reverse DNS: resolve the connecting address to a hostname, confirm the domain, then forward-resolve that hostname and check it returns the same address. The cryptographic successor is now real. Web Bot Auth, announced by Cloudflare on 15 May 2025 and initially in closed beta, has agents sign their requests using RFC 9421 HTTP Message Signatures, adding Signature-Input, Signature and a Signature-Agent header pointing at a public key directory. OpenAI was signing requests at announcement. The IETF has since chartered a working group for it, with two standards-track deliverables dated 30 April 2026 and an operational best-practice document due 31 August 2026. This is the most plausible long-term answer to the question of whose bot is at the door.

What breaks when you turn it on

Every article about anti-bot systems describes the detection. Far fewer describe what the detection does to the people it was not aimed at, which is where most of the money is lost.

False positives land on revenue, not on the vendor. The standing casualties of every technique above are the same groups: users behind corporate NAT and carrier NAT, VPN users, private browsing, assistive technology, keyboard-only navigation, old devices, and your own power users whose behaviour is genuinely repetitive because their job is repetitive.

Price it before you deploy it. Take 2 million sessions a month, a 1% false-positive rate, and 15% of those people giving up rather than solving whatever you put in front of them. That is 3,000 lost sessions a month; at a 2.5% conversion rate and a $70 average order it is about $5,250 a month walking out. That is more than DataDome's entry tier costs. These are illustrative figures, not measured ones, so substitute your own. The shape will not change: on a mid-sized commerce site the false-positive bill and the subscription bill are the same order of magnitude, and only one of them arrives as an invoice you can see.

Visible challenges are an accessibility problem and, in the EU, a compliance one. W3C's note Inaccessibility of CAPTCHA, republished 16 December 2021, puts the design rule in one sentence: "Users should not be forced beyond what is strictly necessary to keep a site secure, e.g., if a honeypot suffices, use a honeypot until evidence of robotic attacks dictates something else." That ordering, cheapest and least intrusive control first, escalating only on evidence, is the part most deployments skip.

Blocking the wrong crawlers is a revenue decision wearing a security costume. Shutting out AI crawlers wholesale also removes you from AI answer surfaces, and the fleets are split precisely so you do not have to make one decision for all of them. Training crawlers, search indexers and live user-triggered fetchers are three different questions with three different answers. Decide them separately and on purpose.

Your defence is now a dependency. An edge platform sits in front of every request you serve. Latency is genuinely small there, single-digit milliseconds in the normal case, but availability, false-positive policy and pricing power now belong to somebody else. Negotiate the false-positive terms harder than the price, because that is the clause that lands on your side of the ledger.

Where each layer stops working

Knowing the boundary of a control is worth more than knowing its mechanism.

TLS and header fingerprinting see nothing behind a login. A competitor with a real browser and a real account produces a perfect handshake. Every network-layer signal you own says "genuine Chrome," because it is genuine Chrome. Abuse behind authentication is an account-behaviour problem, not a fingerprinting one.

Behavioural models need traffic to learn from. Per-site classifiers are trained on your real users. A site with 30,000 monthly sessions cannot support the same model quality as one with 30 million, and vendors will not say this on a call. On low-traffic sites the behavioural layer is closer to a generic heuristic than to the tailored model in the marketing copy.

APIs and mobile endpoints have no browser to fingerprint. Data is frequently taken from the JSON endpoint your own front end calls, where there is no canvas, no WebGL and no mouse. Those paths need their own controls: signed requests, device attestation, mobile SDKs, per-key quotas. A stack that only protects HTML pages protects the least valuable copy of the data.

Per-request scoring is blind to patient, low-volume collection. Thirty requests an hour from residential addresses, for a year, never trips a rate limit and never looks like an attack. The instruments that apply to that adversary are watermarking, canary records and content-similarity monitoring, which detect after the fact rather than prevent.

Proof of work taxes the wrong people. The cost imposed on a funded crawler with a datacentre behind it rounds to nothing. The cost imposed on a five-year-old phone on a slow connection does not. Gates that require JavaScript also exclude text browsers and anyone with scripting disabled outright.

Mazes burn your compute to burn theirs. Generating decoy pages is not free, and content-similarity checks plus link-depth limits get careful scrapers back out of them.

Verified agents become a list you have to maintain. Every identity mechanism, from reverse DNS to signed requests, hardens the honest path and does nothing to the dishonest one. It also creates an allowlist, and allowlists rot. Operators split their fleets, rename bots, publish new address ranges and retire old ones, and an entry you added in 2024 may now wave through something you would not have chosen. Identity is a control worth having. It is not a control you can install once.

Everything above is defeated at once by paying somebody. Managed unblocking APIs sell exactly this: a coherent stack, maintained continuously, billed per successful request. That is the honest limit of the whole discipline. A defence stack does not stop a well-funded adversary. It converts them from free to metered, and it removes everyone who was not willing to pay.

How scrapers get around the controls

Understanding the offence is the fastest way to see what the defence is really built against. Professional scraping stopped being "change the User-Agent" a long time ago. It is a coordinated imitation of one specific real client across every layer at once.

Proxy rotation. Datacenter addresses are cheap and get flagged by ASN. Residential proxies route through real home connections, mobile proxies through carrier ranges, and both are priced by the gigabyte at the rates quoted earlier. Geolocation is matched to the site's actual audience. Swapping addresses alone accomplishes nothing, since it leaves the TLS and browser fingerprints untouched.

TLS impersonation. curl-impersonate reproduces a real browser's handshake so that JA3 and JA4 match the declared User-Agent. Which repository you use matters more than most people realise. The lexiforest fork describes itself as "an active fork of curl-impersonate" from the original lwthiker project, ships stable 2.0.0 built on curl 8.21.0, and covers Chrome 99 through 146, Firefox 133 through 147, Safari 15.3 through 26.0.1, Edge 99 and 101, and Tor 14.5. The Python binding most people actually use, curl_cffi, tracks it. A stack pinned to the stalled original is impersonating browser versions nobody runs, which is a worse fingerprint than not impersonating at all.

Fortified headless browsers. Vanilla Selenium and Puppeteer announce themselves. The current field:

• nodriver, the official successor to undetected-chromedriver from the same author, which talks CDP directly and abandons WebDriver altogether. It publishes no formal GitHub releases, so the version you get depends on when you installed it, and the README's claim is explicitly comparative: what makes it different is "the optimization to stay undetected for most anti-bot solutions." Most, not all, and no guarantee against any particular one.
• Camoufox, a Firefox fork injecting fingerprints at the C++ level so JavaScript inspection cannot see the seam. Version 150.0.2 shipped on 11 May 2026. Its own README is unusually honest about the state of it: "There has been a year gap in maintenance due to a personal situation. Camoufox has gone down in performance due to the base Firefox version and newly discovered fingerprint inconsistencies." It also states two hard limits: it does not fully support injecting Chromium fingerprints, and some WAFs test for SpiderMonkey engine behaviour, "which is impossible to spoof."
• SeleniumBase in its UC and CDP modes, plus stealth plugins for Puppeteer and Playwright. Plugins rot faster than anything else in this list, because they patch symptoms one at a time and detection moves in bulk.

Scrapfly's roundup of anti-bot bypass tools, updated 10 August 2026, covers the same field with hands-on notes, and is worth reading alongside this section for the tooling detail we have deliberately kept short here.

Anti-detect browsers. Multilogin, GoLogin, AdsPower, NestBrowser and similar create isolated profiles with fingerprints that are unique but internally consistent. They were built for multi-accounting and are now standard scraping equipment. Their weakness is the one described earlier: consistency across the whole API surface is much harder to maintain than any single spoofed value, and a deleted or extra API is a giveaway.

Behaviour imitation. Non-linear cursor paths, randomised pacing, realistic dwell times, occasional backtracking, warm-up navigation before the target action. Doing it well is expensive. Most off-the-shelf automation skips it, which is why the behavioural layer keeps earning its keep.

Solving challenges. Solver services return a token for reCAPTCHA, hCaptcha or Turnstile at the prices listed earlier. Professionals treat this as failure handling rather than strategy, because a solve costs both money and seconds, and the seconds hurt more. Keeping the trust score high enough that no challenge appears is cheaper than answering challenges quickly.

Managed extraction. Unblocking APIs and hosted stealth browsers sell the whole apparatus as a per-request service: one continuously patched browser stack, residential rotation, TLS and HTTP/2 profiles matched to real builds, challenge handling included. Sites that fight back are where a managed extraction service earns its keep, and where teams that only need the finished rows increasingly buy data as a service instead of maintaining a stealth stack whose whole purpose is to be rebuilt every quarter.

The lesson runs identically in both directions. The winner is not whoever disguises themselves most aggressively, but whoever's layers agree with each other. Detection is the search for disagreement.

We pointed an ordinary fetcher at thirteen vendor pages

A small measurement, run while writing this article, because it is easy to repeat and it says something the brochures do not.

Method. On 10 August 2026 we requested thirteen bot-management and challenge-vendor pages with a plain automated HTTP client from a datacenter address: no browser, no JavaScript execution, no cookie jar. The list was Cloudflare, DataDome, HUMAN, Akamai Bot Manager, Akamai Content Protector, Imperva, Kasada, Arkose Labs, Netacea, Fastly, hCaptcha and Friendly Captcha, plus the Anubis project site. Two pages needed a second attempt after an error on our own side; nothing else was retried.

Result. All but two returned their full page text to a client that was obviously not a person. Imperva's bot-management product page returned an interstitial instead, explaining that it had been flagged and asking for JavaScript and cookies. The Anubis project site returned its own proof-of-work gate, carrying the build string v1.27.1-0.20260808190809, dated two days before the request.

What it does and does not show. It does not measure detection quality. A marketing page is supposed to be crawlable, and a security vendor blocking Googlebot from its own product page would be making a strange trade. What it shows is that even in this industry the default posture on public content is open, and that the two exceptions are the ones whose product is content protection specifically. It also gave us a liveness signal better than any changelog: Anubis proved it was maintained by refusing us with a two-day-old build.

What the platforms cost and what they are for

The bot-management market is mature, crowded, and almost entirely priced by negotiation. Below is what could be established from vendor material on 10 August 2026. Where a vendor publishes no prices, this says so instead of inventing a range.

Cloudflare Bot Management. The largest deployment surface by a distance, and worth being precise about how large. Cloudflare's own page says its systems power "20% of the Internet." The independent survey at W3Techs, read the same day, puts Cloudflare on 24.3% of all websites and 84.1% of sites whose reverse proxy is known. Those denominators are not the same thing and the two numbers are not comparable, but both are larger than any competitor's, and network visibility is the actual product: models trained on that much traffic see a new attack pattern on somebody else's site first. Note the packaging, because it trips people up. The 1–99 bot score is an Enterprise add-on, where 1 is near-certain automation and 99 near-certain human, bucketed as automated at 1, likely automated at 2–29, likely human at 30–99, with 0 meaning not computed. Pro and Business plans see the buckets in analytics, not the number. Around it sit Turnstile, AI Labyrinth and AI Crawl Control, the last available on all plans, with pay-per-crawl still in private beta as of the April 2026 documentation update. Cloudflare's stated direction has been public since 1 July 2025, when it announced it was "changing the default to block AI crawlers unless they pay creators for their content."

DataDome. ML-first, real-time, strongest in e-commerce, marketplaces, travel, fintech and media, and the only major vendor here that publishes a rate card. Mobile and M2M API protection arrives at the second tier, agentic endpoint protection including MCP at the third. Detection claims are the vendor's own and unaudited.

HUMAN Security. Behavioural analytics plus an anti-fraud ecosystem covering ad fraud, account takeover, fake accounts and transaction abuse. This is the second correction. The earlier version of this article called HUMAN "formerly PerimeterX," which reverses the history. The company's own merger announcement of 27 July 2022 identifies it as "HUMAN Security, Inc. (formerly White Ops)" and describes a merger with PerimeterX, bringing 450 employees and 500-plus customers under the HUMAN name. White Ops became HUMAN; PerimeterX merged into it. The mistaken version is repeated across a great many roundups.

Akamai Bot Manager. Enterprise-scale, edge-first, described as taking action "at the edge server, forwarding only clean traffic to the origin." The scraping-specific sibling is Content Protector, which names its layers explicitly: protocol fingerprinting of how the connection is established, application-level evaluation of whether the client can run business logic in JavaScript, user-journey analysis, interaction metrics from touch, keyboard and mouse, and custom JavaScript that looks for headless indicators "even in stealth mode." Most defensible for teams already standing on Akamai.

Imperva. Mature bot management with strong API coverage, usually bought alongside the Imperva WAF, and its Bad Bot Report is now published under Thales branding. There is no link here for a reason worth stating: Imperva's own product page is one of the two in our thirteen that refused an automated request, so we could not open it to verify current claims.

Kasada. Cryptographic proof of work and heavy client-side obfuscation, aimed at making retooling expensive rather than making challenges visible. The site's framing is direct: "Kasada runs a full VM with custom bytecode," and "We stop the most sophisticated bot attacks without CAPTCHAs." Current capabilities are listed as Bot Defense, Account Intelligence, AI Agent Trust and Kasada IQ for Fraud. The claim to weigh sceptically, here and everywhere in this market, is any assertion about competitors' miss rates, which no independent test supports.

Arkose Labs. Risk scoring plus interactive challenges, built around making an attack economically pointless rather than technically impossible, and strongest in fraud and mass fake registration. The platform is now called Arkose Titan and includes an Agent Trust Manager to "classify and govern AI agents at scale." The site advertises "the industry's only $1M warranty" against bot and AI agent fraud.

Netacea. The one genuinely different deployment model: "no agents, no code, no hardware," working from server-side and log data rather than injected client JavaScript. That makes it the option when you cannot put a script on the page at all, and it means attackers cannot probe your detection by reading your front end.

Fastly Bot Management. Edge detection "performed in real time at the edge, where traffic can be evaluated before it reaches origin infrastructure," combining server-side and client-side signals, with deception tactics and AI-traffic monetisation in the current feature set. The natural choice for teams already delivering through Fastly.

Challenge providers as a standalone layer. Turnstile, reCAPTCHA inside Google Cloud Fraud Defense, hCaptcha and Friendly Captcha are all deployable on their own and all appear embedded inside larger platforms. For a site below the volume where a bot-management subscription pays for itself, this layer plus rate limiting plus a honeypot is the entire sensible stack.

Roughly, the market splits three ways. Cloudflare, Akamai and Fastly are strongest at the edge and when consolidating web security into one stack. DataDome and HUMAN go deeper at the application, API and mobile layers. Kasada and Arkose sell asymmetry: make the attacker's retooling cost more than the data is worth.

What to ask before you sign

Trials in this market are run by the vendor, on their terms, and they measure the thing the vendor is good at. Four questions move the conversation somewhere more useful.

"Show me the sessions you blocked, not the ones you caught." Detection rate is the easy half. Ask for a raw sample of blocked sessions from your own traffic during the trial, big enough to review by hand, and price the legitimate ones yourself. Every team that skips this learns the number later, from support tickets, at a worse moment.

"How much of my traffic do your per-site models need?" Per-customer models are the headline feature of most of this market, and they are trained on your data. A vendor who cannot say how many sessions a model needs before it beats a generic ruleset is telling you something.

"What happens to my score when a customer arrives with an agent?" This is the live question of 2026, and the answers differ enormously.

"Which contract term covers false positives?" Usually none. Ask anyway, and note the answer.

Where this is going

From signatures to behaviour, and from behaviour to identity. Static rules gave way to per-site behavioural models, and those models are now running into clients that behave like people because a person is driving them. The next control is not a better classifier. It is a signed request that says who this agent is and who it works for.

Network signals stay the foundation. They work before JavaScript, they are expensive to fake coherently, and they are where a mismatch shows up first. They are also not static. Handshakes change as browsers change, post-quantum key agreement is being rolled into browser TLS stacks, and every such change rewrites the fingerprint tables built on top. GREASE, from RFC 8701, exists precisely so those changes do not break the web on the way through. Any fingerprint table without an owner and a refresh cadence is decaying from the day it ships.

Soft challenges instead of puzzles. Proof of work and background scoring are displacing perception tests, for accessibility reasons as much as security ones, and the pricing above shows the market has already moved.

AI crawlers as their own category, with money attached. Pay-per-crawl, robots.txt compliance tracking with enforcement, crawler and agent monetisation tiers at DataDome, agent trust products at Arkose, Kasada and HUMAN. The question has genuinely shifted from "human or bot" to "which bot, acting for whom, and on what terms."

Cryptographic bot identity. Web Bot Auth is the piece to watch, because it is the only proposal in this space with an IETF working group, a signature format borrowed from an existing RFC, and shipped implementations. If it lands, verified crawlers stop being a DNS lookup and start being a key.

The legal layer is not a defence layer

This is the third correction. The earlier version of this article said that hiQ Labs v. LinkedIn affirmed that scraping public pages is not a Computer Fraud and Abuse Act violation. That is half right and stated far too confidently. Both Ninth Circuit rulings, September 2019 and April 2022, concerned a preliminary injunction and the likelihood of success, not the merits. The Supreme Court vacated the 2019 opinion on 14 June 2021 and sent it back in light of Van Buren v. United States. When the case reached the merits, hiQ lost: in November 2022 the district court found it had breached LinkedIn's User Agreement, and the parties settled after that. The durable lesson is narrower than the one usually drawn. The CFAA is a weak instrument against logged-out scraping, so platforms sue on contract, on circumvention, and on other theories instead.

Nothing in robots.txt changes any of this, and RFC 9309 says so itself. A declaration establishes what you permitted, which matters later, in an argument. It is not an access control and it never was. The practical consequence for defenders is that your legal position is strongest when your technical position documents itself: machine-readable permissions, real access controls on anything that matters, and records showing what was taken and when.

This is not legal advice, and jurisdictions differ sharply on the questions that follow from it.

What actually holds

Perfect protection does not exist. A motivated, funded operation with residential addresses, a patched browser stack and a solver account gets through nearly anything, and any vendor page that implies otherwise is selling.

What a good stack buys is arithmetic. At list prices, protecting a request costs roughly four cents per thousand and harvesting one costs roughly a dollar per thousand. Layering is what keeps that gap open, because each layer forces the attacker to buy something else and to keep it consistent with everything they already bought. One residential address does not help if the handshake says Python. A perfect handshake does not help if the mouse never moves. A perfect mouse does not help if the account was created ninety seconds ago.

Three things are worth doing before any of the tuning. Put the heaviest checks on the few endpoints that matter, which is pricing, search, product detail, login and checkout, rather than smearing expensive scoring across every page. Measure your false positives in currency, weekly, because that number decides whether the deployment was worth it and nobody else will produce it for you. And decide on purpose which automated clients you want, since a stack tuned to exclude every non-human client will eventually exclude the ones acting for your customers.

The realistic goal was never to make scraping impossible. It is to make it slow, expensive, brittle and visible, while the people and machines you actually want keep getting through.