In 2025 automated clients sent more requests to the web than people did. The Thales/Imperva Bad Bot Report put automated traffic at 53% of everything it observed, roughly three quarters of that classed as malicious. Cloudflare, counting HTTP requests across its own network, reached 57.5% by June 2026. The two figures use different denominators and are not comparable, but they break the same assumption, the one sitting underneath most anti-scraping advice: that bots are the exception and humans are the default.
The second change is more awkward for defenders. A growing share of that automation is acting for a specific person at that moment, whether it is an assistant fetching the page a user just asked about, a shopping agent checking your prices, or a booking agent filling your form. Cloudflare's telemetry shows user-triggered AI crawling growing roughly fifteenfold during 2025. "Is this a bot?" has stopped being a useful question. The useful ones are whose bot it is and what happens to the data afterwards.
What follows are the protection techniques that actually get deployed, oldest to newest, each with the countermeasure that defeats it. Every defence here has a documented bypass, most of them cheap, and a security page that does not say so is selling something. What a good stack buys is cost: enough of it, in enough places, that harvesting your data stops being worth the money.
If your question is the prior one, "is my site even being scraped?", that is a detection problem with its own answer. Start with How to detect your site is being scraped, and with the vendor-neutral taxonomy in OWASP Automated Threats to Web Applications, which catalogues scraping as OAT-011 alongside its neighbours.
Before you pick a defence, decide what you are defending against
Most bad anti-scraping decisions come from skipping this step and buying a general-purpose bot platform for a specific problem. Four adversaries show up in practice, and they respond to completely different controls.
A competitor scraping prices or catalogue data. Low volume, high value per record, patient, willing to spend real money. Rate limits will not touch them; they will happily run at thirty requests an hour from residential addresses for a year. Fingerprinting, behavioural scoring and watermarking are what apply here.
AI training crawlers. High volume, mostly declared, mostly compliant with robots.txt. Cloudflare measured training crawlers at 52% of all crawler requests in June 2026, up from 22% in spring 2025. They create a bandwidth bill and a licensing exposure, not a break-in. Wikimedia reported multimedia bandwidth up 50% between January 2024 and April 2025, with bots responsible for 65% of the most expensive traffic while accounting for only 35% of pageviews. A text file stops this adversary, if it chooses to read one.
AI answer engines and user-triggered fetchers. These arrive live, one page at a time, on behalf of someone who asked a question, and they are the most likely to ignore robots.txt, because their operators argue a user-directed fetch is not crawling. TollBit's late-2025 State of the Bots data found 30% of AI scrapes ignoring explicit permissions, with OpenAI's ChatGPT-User the least compliant at 42%. Blocking them is a revenue decision.
Content republishers, SEO spam and low-effort harvesting. High volume, low sophistication, beaten by the cheapest layers: a rate limit, a JavaScript requirement, a honeypot. Credential stuffing and account abuse often get filed here, but they are a different toolkit with a different urgency, and most of this article is the wrong list for them.
1. Rate limits, quotas and IP reputation
The oldest signal of automation is volume. If one address requests far more pages, far faster, than a person plausibly could, it gets throttled, challenged or blocked. A related control caps the amount of data any single client can pull over a period, which is the right instrument when the concern is bulk harvesting and not server load.
Calibration is the hard part, and it is harder than it looks. Set the threshold low and you punish power users, shared corporate networks and mobile carriers, which put thousands of real subscribers behind a handful of addresses; carrier-grade NAT has made a single IPv4 address a meaningless unit of identity across large parts of the mobile web. Set it high and scrapers stroll underneath. Most teams profile genuine traffic first, find the 99th percentile of real behaviour, then start well above it. On IPv6, rate-limit by /64 prefix; individual addresses are meaningless there, since a single client is routinely handed billions of them.
Google has policed query rates per address for two decades, escalating to interstitial challenges when a session looks automated. Hand-rolled rules have largely given way to managed services, and the platforms listed in Commercial bot management and WAFs below all ship rate limiting as one component of something larger.
Address reputation is blunt, which is the standing drawback. Legitimate users behind a flagged address sit inside the blast radius, and aggressive blocking degrades the experience for everyone sharing it. Quotas set carelessly frustrate the heavy, loyal users you would least like to lose, along with any public API consumer you have.
How it's bypassed. Traffic gets spread across many addresses using proxy pools. Datacenter proxies are cheapest and most easily flagged by ASN. Residential and mobile proxies share address space with real subscribers, which makes them far harder to ban without collateral damage; they are sold by the gigabyte, and that price is the real constraint on a scraper's side. Rotating services assign a fresh exit per request. The same manoeuvre defeats quotas: split the workload across enough identities and each one stays under the cap while the aggregate harvest proceeds on schedule. Rate limiting alone stops nothing more sophisticated than a naive loop, which is why it belongs underneath the fingerprinting and behavioural layers, never in front of them.
2. Account gating
Putting content behind a login moves the unit of control from the address to the account. You can then watch per-account behaviour, throttle a suspicious session, and ban the offender regardless of which network they reconnect from. Social platforms have run this model for years, scoring account activity continuously and disabling accounts that behave mechanically.
Gating is also the clearest legal line in the United States. After Van Buren v. United States (2021), the Computer Fraud and Abuse Act turns on a gates-up-or-down question: were you entitled to access this area at all? Public pages sit behind an open gate; a login does not. That distinction runs through nearly every scraping case decided since.
The cost is real. Gating reduces reach and indexability, adds friction to every first visit, and moves the arms race to account creation.
How it's bypassed. Scrapers build pools of accounts, increasingly created and warmed automatically. The strongest defensive addition is identity verification at signup: phone verification, email confirmation, payment-instrument checks, device attestation. None of these is airtight, since disposable numbers and account-resale markets exist for exactly this purpose, but each step raises the unit cost of a fake account measurably. If an account costs a scraper $2 instead of nothing, and your data needs a thousand accounts to harvest, you have put a price on your own catalogue.
3. CAPTCHAs and the challenges that replaced them
A challenge asks the visitor to prove they are human before continuing. Historically that meant transcribing distorted text or picking traffic lights out of a grid. That era is over, and it ended because the puzzles stopped working.
A 2023 USENIX Security study (UC Irvine, ETH Zürich, Microsoft and Lawrence Livermore) ran 1,400 participants through 14,000 CAPTCHAs and compared them with automated solvers. On distorted text, humans took 9 to 15 seconds at 50–84% accuracy while bots took under a second at 99.8%. On hCaptcha's image grids, humans scored 71–81% and bots 98%. In 2024 a group at ETH Zurich reported solving reCAPTCHA v2 image challenges at a 100% rate using off-the-shelf YOLO models, against 68–71% for prior published work, and found that reCAPTCHA v2's real discriminator was cookies and browser history, with the puzzle contributing little.
So the industry moved to invisible scoring. Google has dropped the v2/v3 branding and now distinguishes checkbox keys from score-based keys; both return a 0.0–1.0 risk score per assessment, and score-based keys simply never render a puzzle. The product has also moved house, with legacy keys migrated into Google Cloud projects and reCAPTCHA folded into Google Cloud Fraud Defense in April 2026. The free ceiling is 10,000 assessments per month per organisation, $1 per 1,000 beyond it.
The current field:
- Google reCAPTCHA — score-based keys return a risk score with no puzzle in the common case; Enterprise tiers add WAF integration and mobile SDKs.
- hCaptcha — still advertises a two-line swap from reCAPTCHA, but has repositioned as an enterprise fraud platform covering bot detection, account defence and user journeys. Pro starts at $99 a month billed annually for 100,000 evaluations. Its privacy positioning is contested, since it collects address, pointer, timing and hardware signals like everyone else.
- Cloudflare Turnstile — no per-request charge since general availability in September 2023, though the free plan caps you at 20 widgets and 10 hostnames each. Note that the recommended Managed mode does sometimes render a checkbox, chosen dynamically by risk; what Cloudflare guarantees is no visual puzzle, not zero interaction.
- Anubis and similar proof-of-work gates — these invert the model, charging CPU instead of testing perception. See Honeypots, mazes and tarpits.
Visible CAPTCHAs are friction, an accessibility barrier and, in the EU, a compliance exposure. W3C's Inaccessibility of CAPTCHA note recommends non-interactive alternatives first, and the European Accessibility Act (Directive (EU) 2019/882) has applied to EU-facing e-commerce since 28 June 2025, with a visual-only challenge in a consumer checkout being exactly the kind of barrier it targets. Cloudflare's own measurements put a classic CAPTCHA at roughly 32 seconds against about one second for a managed challenge, with visitors 31% less likely to abandon the latter. The USENIX study saw 18–45% of participants drop out after the first CAPTCHA.
How it's bypassed. By paying someone, or something, per solve. The old split between human farms and automated recognition has collapsed, and the economics have inverted: automated solving is now both cheaper and faster than human solving. As of August 2026, published price lists put CapSolver at roughly $0.80 per 1,000 reCAPTCHA v2 solves at 3–5 seconds each, against $2.00–$2.99 and 10–20 seconds at the human-backed services. Simple image CAPTCHAs run $0.50–$1.00 per 1,000 and Turnstile tokens around $1.20–$1.45. The clearest premium in the market is reCAPTCHA Enterprise at up to $5 per 1,000, which tells you roughly what the extra signal is worth.
One counterweight cuts against the doom framing. General-purpose LLM agents remain bad at multi-step interactive challenges, with the Open CaptchaWorld benchmark putting humans at 93.3% and the best agent at 40%. Purpose-built solvers are the threat model here, not chatbots. The move to invisible, behaviour-scored challenges has blunted both, because there is often no puzzle to solve, only a session that has to look human end to end.
4. JavaScript execution and client-side proof of work
The server requires the client to run JavaScript that computes a token, usually via deliberately convoluted code split across several loadable files, and refuses the response if the token is missing or wrong. A plain HTTP client that does not execute JavaScript simply fails. Large platforms have used this pattern as a gate for well over a decade.
A modern variant does not obfuscate at all; it charges. Proof-of-work gates make the browser burn CPU on a hash puzzle before the page is served, which is negligible for one reader and ruinous for a crawler fetching a million pages.
There is an underrated side effect for defenders: any scraper that runs your JavaScript to pass the challenge also fires your analytics, so it surfaces in your reports as an anomalous session. The costs are engineering complexity, page weight, and a hard dependency on JavaScript that excludes text browsers, some assistive setups and any user who has scripting disabled.
How it's bypassed. Scrapers drive real browser engines, so the JavaScript actually runs:
- Playwright — Microsoft's framework, with direct browser control, auto-waiting and network interception. The default choice for new work.
- Puppeteer — Chrome and Chromium automation for Node.js.
- Selenium — the long-standing, multi-language WebDriver standard, still ubiquitous in test estates.
For static or API-backed pages, lighter stacks remain the sensible tool: Scrapy and Beautiful Soup are both actively maintained and shipped releases in 2026. The older Mechanize library, cited in earlier versions of this discussion, does not execute JavaScript and has been superseded for this purpose.
The bigger change is a generation of LLM-driven scrapers that removes the brittle part of the job. Firecrawl and Crawl4AI return model-ready markdown from a URL. Stagehand wraps Playwright in act(), extract() and agent() calls written in English. Browser Use and Microsoft's Playwright MCP let a model drive a browser directly, the latter through the accessibility tree instead of screenshots. That generation is what kills selector brittleness as a defence, which is the subject of the next section.
5. Changing the page structure
Scrapers depend on stable selectors: element IDs, class names, DOM hierarchy. Change them regularly and brittle parsers break. In practice this ranges from rotating hashed class names at build time to reshuffling the layout.
It is expensive for you too, since it complicates your own maintenance, testing and caching, so it usually happens in periodic passes instead of continuously. Utility-first CSS and build-time hashing have made it cheaper than it was, which is a reason to reconsider it if you dismissed it years ago.
How it's bypassed. Resilient parsers anchor on the things you cannot rotate without breaking your own site: visible text, ARIA roles, structural relationships, JSON-LD blocks, or the JSON API your front end is already calling. The LLM-driven extractors above are close to immune, because they read the rendered page the way a person does, without matching a CSS path. Structure churn still raises maintenance cost on both sides and slows an unattended scraper. It no longer stops an attended one.
6. Rendering data as images
Replacing machine-readable text with images keeps content visible to humans while making naive extraction harder. Prices, email addresses and phone numbers are the usual candidates; some sites render whole passages, historically via Flash and today via canvas.
Treat this as the expensive trade it is. Image-rendered content is invisible to search engines, cannot be selected or copied, breaks translation and reflow, and is inaccessible to screen-reader users unless you duplicate it in an alt attribute, which of course hands the text straight back to the scraper. Use it for a handful of high-value fields or not at all.
How it's bypassed. OCR, running through the same recognition pipelines that solve CAPTCHAs and at broadly the same per-record prices. Modern vision-language models read a rendered price table about as reliably as they read a receipt. This raises the per-record cost of extraction without preventing it.
7. Browser and TLS fingerprinting
Long before a scraper touches your DOM, it has introduced itself. At the network layer, the TLS handshake and HTTP/2 settings produce a fingerprint that betrays whether the client is Python's requests or a genuine Chrome build.
The details here have moved, and the JA3 hash that everyone still cites was archived by Salesforce in May 2025. It is largely superseded by JA4, maintained by FoxIO, and the reason is instructive: Chrome 110 began permuting ClientHello extension order in early 2023, so a single Chrome install started emitting many different JA3 hashes. JA4 sorts ciphers and extensions and strips GREASE, which lets it survive that randomisation. JA4 is one member of the JA4+ suite, alongside JA4S for servers, JA4H for HTTP, JA4T for TCP and JA4X for certificates. If you are building on it, note the licensing split: JA4 itself is BSD-3-Clause while the rest of the suite sits under the FoxIO License 1.1.
Above TLS, the Akamai HTTP/2 fingerprint hashes the SETTINGS frame values, the WINDOW_UPDATE increment, PRIORITY frames and pseudo-header order, in the form S[;]|WU|P[,]#|PS[,]. Equivalent HTTP/3 fingerprinting, based on QUIC transport parameters, exists but is immature; anti-bot products have not reached HTTP/2-level coverage there, which makes HTTP/3 a soft spot on both sides.
At the browser layer, JavaScript can read hundreds of environment signals: screen geometry, installed fonts, WebGL, canvas and audio outputs, timezone, language, hardware concurrency, client hints. Two commonly repeated tells are no longer tells. "Missing plugins" stopped meaning anything when navigator.plugins became a fixed five-entry PDF list in every modern browser per the HTML spec. And navigator.webdriver is set by the automation flag, not by headlessness, so one command-line switch removes it. Chrome's old headless mode was split out of the main binary in Chrome 132 into a separate chrome-headless-shell download, which is still shipped and still fingerprintable; the --headless Chrome that most automation now runs is the same binary real users have.
Detection has therefore moved to side effects of the Chrome DevTools Protocol itself, principally the object-serialisation behaviour triggered by Runtime.enable and Runtime.consoleAPICalled, plus isolated-world naming leaks and pptr: source markers. Even these decay. The best-known variant of the CDP error-getter check stopped working when V8 changed its DevTools error handling in May 2025.
How it's bypassed. Every one of these signals gets patched. TLS-imitating clients reproduce a real handshake: curl-impersonate is now maintained as the lexiforest fork, impersonating current Chrome, Firefox, Safari, Edge and Tor builds, with curl_cffi as the Python binding most people actually use. The original lwthiker repository stalled in 2024, so a stack pinned to it is impersonating browsers nobody runs. On the automation side, undetected-chromedriver has not shipped since February 2024; its author's successor nodriver talks CDP directly with no WebDriver at all, and the current field also includes Patchright, rebrowser-patches, Camoufox and SeleniumBase's UC mode. Puppeteer's stealth plugin was last published in early 2023 and is now well detected, despite still being downloaded a million times a week.
Two things make this layer more interesting than a patch-and-counter-patch treadmill. The first is coherence: every layer has to agree. A residential address in Poland attached to a browser reporting en-US, an America/Chicago timezone and 64 CPU cores is not a fingerprint, it is a confession, and keeping geolocation, Accept-Language, timezone and client hints consistent is much harder than spoofing any one of them.
The second is that randomisation is itself a signal. Anti-detect browsers that add noise to canvas output get caught by calling the API twice and diffing the results, and the noise is often reversible. The general method goes back to the FP-Scanner work at USENIX Security 2018: check that the APIs actually present match the browser and operating system being claimed. A documented case from 2025 found one commercial anti-detect browser had deleted navigator.share, BarcodeDetector and CSPViolationReportBody, all of which must exist in the Chrome build it claimed to be.
8. Behavioural analysis
Even a perfectly disguised browser has to act like someone. Modern systems collect mouse coordinate sequences, scroll acceleration, click timing, keystroke cadence and navigation order, then score them against classifiers trained on real traffic for that specific site. Some platforms have shifted to intent-based detection, asking whether a navigation pattern looks like data collection rather than reading, which can flag a technically flawless client.
It costs data and tuning, and an over-aggressive model produces false positives on legitimate but atypical users: assistive technology, keyboard-only navigation, and anyone whose session is genuinely repetitive because their job is repetitive.
How it's bypassed. Scrapers simulate human input, with non-linear cursor paths, randomised pacing, realistic dwell times and occasional backtracking. Doing this convincingly at scale is hard, and most off-the-shelf automation either skips it entirely or moves the cursor in perfect straight lines that a classifier spots on the first sample. Behaviour remains one of the most durable layers precisely because faking it well cannot be bought for $1.45 per thousand.
The complication is the one from the introduction. Agentic browsers move a cursor because a person asked them to buy something, and behavioural models trained to flag "not a human hand" will increasingly flag customers. If you deploy this layer, budget for the false-positive review alongside the detection rate.
9. Honeypots, mazes and tarpits
A honeypot is bait only a bot will take: a link or form field hidden from people via CSS (display:none, visibility:hidden, zero opacity, off-screen positioning), or a path excluded in robots.txt that no compliant crawler should visit. A real user never sees it, while a scraper that indiscriminately follows links or fills every field walks in and identifies itself. Triggering one can get the address blocked, sometimes across a whole proxy pool. The email honeypot is the classic variant: a hidden address that proves a harvester took it the moment mail arrives.
The 2024–2026 additions are mazes. Cloudflare's AI Labyrinth, shipped in March 2025 and available on the free tier, serves misbehaving crawlers a set of pre-generated, plausible, no-index decoy pages linked invisibly from real ones. Open-source equivalents such as Nepenthes, iocaine and Quixotic generate endless Markov-chain gibberish. Proof-of-work gates belong to the same family of "make the bot pay": Anubis, released in January 2025, has been adopted by GNOME, FFmpeg, FreeCAD, Codeberg and UNESCO among others.
The drawbacks get less attention than the tools do. A robots.txt-based trap punishes the crawlers polite enough to read robots.txt, which is a strange selection pressure. Poorly implemented CSS traps snag accessibility tooling, since a screen reader may well announce a link you thought was invisible. Mazes burn your own compute and bandwidth to burn someone else's, and Tavis Ormandy's critique of proof-of-work gates is the one to read before you deploy one: the cost imposed on a well-funded crawler is close to nothing, while the cost imposed on a user with an old phone is not. Anubis also blocks anyone with JavaScript disabled outright.
How it's bypassed. Careful scrapers check computed styles before following a link or filling a field, and stay on well-linked, human-reachable paths. Generated mazes fall to content-similarity checks and link-depth heuristics. Codeberg reported scrapers solving Anubis challenges by August 2025, and sourcehut has since moved to a different gate. Honeypots remain excellent value against unsophisticated bots, which is still most of them.
10. Crawler identity: verify what you allow
Every allowlist keyed on a user-agent string is decorative. User-Agent: Googlebot is a header anyone can send, and treating it as identity leaves your carefully tuned stack with a hole that has public instructions. The standard fix is old and boring: reverse-resolve the connecting address, confirm the hostname ends in the operator's domain, then forward-resolve that hostname and check it returns the same address.
$ dig -x 66.249.66.1 +short
crawl-66-249-66-1.googlebot.com.
$ dig crawl-66-249-66-1.googlebot.com +short
66.249.66.1Google documents this method and also publishes machine-readable ranges (common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json and others) so you can skip the DNS round trip. Bing offers an equivalent tool. OpenAI publishes separate JSON ranges per bot, and Anthropic now publishes a single combined file at claude.com/crawling/bots.json covering all three of its agents, having previously declined to publish any.
Knowing which declared bot is at the door then lets you make a decision instead of applying a blanket rule, and the vendors have split their fleets specifically so you can. OpenAI runs GPTBot for training, OAI-SearchBot for its search index and ChatGPT-User for live user-triggered fetches. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User on the same pattern. Google-Extended and Applebot-Extended are not crawlers at all; they are usage opt-out tokens, and disallowing them changes what your content is used for without removing you from Search, Spotlight or Siri. The practical consequence is that blocking GPTBot stops training and does nothing about live retrieval, and that blocking Google-Extended never removed anyone from AI Overviews.
The controls available to express this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /internal/robots.txt itself was finally standardised as RFC 9309 in 2022. Everything layered on top of it is a draft or a policy, and the difference matters:
- Content Signals, Cloudflare's September 2025 policy, adds
search,ai-inputandai-traindeclarations torobots.txt. Cloudflare pushed a default ofsearch=yes, ai-train=noacross 3.8 million managed domains, deliberately leavingai-inputunset. It is a stated preference with contractual framing behind it, and no enforcement mechanism. - IETF
aiprefis the standards-track attempt at the same thing. The vocabulary draft is atdraft-ietf-aipref-vocab-06(April 2026) and the attachment draft, which would update RFC 9309, has lapsed. Nothing is ratified. - Web Bot Auth flips verification from network to cryptography: agents sign requests with Ed25519 keys using RFC 9421 HTTP Message Signatures and a
Signature-Agentheader pointing at a public key directory. Cloudflare shipped signed agents in August 2025 with ChatGPT agent, Block's Goose, Browserbase and Anchor Browser in the first cohort, and AWS WAF added support that November. The IETF chartered awebbotauthworking group in 2026. This is the most likely long-term answer to "whose bot is this". llms.txtis not an access control. It is a Markdown index of your content for models that choose to read it, with thin adoption and no engine committed to honouring it. It does not belong in a security review.
How it's bypassed. By not declaring at all. In August 2025 Cloudflare reported that when it blocked declared Perplexity crawlers on test domains, an undeclared generic Chrome user-agent kept fetching them at 3–6 million requests a day from ASNs outside Perplexity's published ranges, and it delisted Perplexity from its verified bot list. Perplexity's rebuttal the same day attributed the traffic to a third-party cloud browser and called the analysis mistaken. The dispute is unresolved, but the structural point survives either version: an undeclared client with a residential address and a real browser fingerprint is indistinguishable from a reader, and no identity layer catches it. Identity verification hardens the honest path and does nothing to the dishonest one.
11. Commercial bot management and WAFs
Most sites with a real problem now buy the stack instead of building it. These platforms combine address reputation, TLS and HTTP fingerprinting, JavaScript challenges, behavioural ML and challenge issuance into a single score computed at the edge in single-digit milliseconds. Many run per-customer models, so each protected site is a separate problem for an attacker, and solving one does not solve the next.
- Cloudflare Bot Management — the widest deployment surface by some distance, since Cloudflare proxies about a quarter of all websites. Note the packaging: the scored product is an Enterprise add-on, and lower plans get Bot Fight Mode or Super Bot Fight Mode without access to the score. That score runs 1–99, where 1 is near-certain automation and 99 near-certain human, bucketed as "automated" at 1, "likely automated" at 2–29 and "likely human" at 30–99, with 0 meaning not computed.
- DataDome — ML-first, classifying in real time across its whole network, strong on device and behavioural signals.
- HUMAN Security — formerly White Ops, merged with PerimeterX in 2022. The common claim that HUMAN "was formerly PerimeterX" gets this backwards.
- Kasada — CAPTCHA-free by design, and the engine behind Vercel's BotID, launched in 2025.
- Akamai Bot Manager — sold in Standard and Premier tiers alongside App & API Protector and the scraping-specific Content Protector.
- Imperva — a Thales company since December 2023; absorbed Distil Networks in 2019.
Several others are worth knowing about beyond that list. F5 Distributed Cloud Bot Defense is built on Shape Security. AWS WAF Bot Control adds browser interrogation in its Targeted tier and now supports Web Bot Auth plus a Monetize action returning HTTP 402. Fastly and Radware both sell bot managers of their own. Netacea works from server logs instead of client-side JavaScript, which makes it the option when you cannot inject a script at all. Forrester's Q2 2026 Wave for the category placed DataDome, HUMAN and Kasada in the leaders quadrant, with Arkose Labs, CHEQ and Netacea as strong performers.
One frequently repeated claim deserves scepticism: that Akamai is unmatched at unmasking patched headless browsers. It traces to vendor marketing and to scraper vendors selling the bypass; no independent testing supports it.
The drawbacks are cost, false positives against unusual real users, and a dependency you cannot easily unwind. Latency is genuinely small at the edge. The false-positive question is the one to negotiate hard on, because it lands on your revenue and not the vendor's.
How it's bypassed. This is the expensive tier, with no universal bypass. Defeating it means defeating every layer coherently and at once: authentic TLS fingerprint, patched browser environment, residential or mobile addresses, plausible behaviour. That work has been commoditised into managed unblocking APIs and hosted scraping browsers (Bright Data Web Unlocker, Oxylabs Web Unblocker, Zyte API, ScrapingBee, ZenRows and others) which sell exactly this stack as a per-successful-request service. The economics are the point: the defender pays a subscription, the attacker pays per page, and the contest is over whose unit cost moves first. Vendors retrain continuously and globally, so any specific workaround has a short shelf life.
12. Watermarking and canary tokens
This is a detective control rather than a preventive one, and it belongs in every serious stack for exactly that reason. Seed your dataset with unique, harmless markers: a listing that does not exist, a distinctively misspelled entry, a one-off email address, a canary token. When your data surfaces on someone else's site, you can prove where it came from, which is what turns a suspicion into a takedown notice or a claim. Thinkst Canarytokens makes planting the tripwires trivial.
Design matters more here than most people assume. Markers need to be numerous, individually plausible, and distributed so that no single one is load-bearing. A single fake listing tells you that you were copied; a per-customer or per-session watermark tells you who copied you, which is worth considerably more. What it will not do is stop the initial scrape, and sophisticated republishers strip obvious markers.
How it's bypassed. Cross-referencing several sources and discarding outliers, or aggregating hard enough that individual records lose identity. Neither is free, and both degrade the quality of the stolen dataset, which is itself a form of defence. Well-hidden watermarks stay valuable precisely because the scraper cannot tell which records are bait, so removing them means throwing away good data too.
13. Policy, licensing and the legal layer
Not every defence is technical, and the non-technical ones have moved faster since 2024 than anything in the stack above.
Establish intent. A clear robots.txt (RFC 9309), explicit terms of service, documented rate-limit headers and, if relevant, Content Signals create a record of what you permitted. That record is what every later argument depends on.
Understand what the US case law actually says. hiQ Labs v. LinkedIn is cited constantly for the proposition that scraping public data is not computer fraud, a reading that is half right and usually stated too confidently. Both Ninth Circuit rulings were preliminary-injunction decisions about likelihood of success, and the 2019 opinion was vacated by the Supreme Court and remanded after Van Buren. On the merits, hiQ lost. In November 2022 the district court granted LinkedIn summary judgment on breach of contract, and in December 2022 hiQ accepted a $500,000 judgment and a permanent injunction barring it from scraping LinkedIn at all. The durable lesson is that the CFAA is a weak instrument against logged-out scraping, so platforms litigate other things.
And they do. Meta v. Bright Data (January 2024) held that platform terms bind users rather than logged-out visitors, and that Meta had left the gate open by not requiring a password. X Corp. v. Bright Data (May 2024) dismissed scraping-based contract claims as preempted by the Copyright Act, though a later ruling let server-access claims through on a trespass theory. Ryanair v. Booking.com produced a CFAA jury verdict in July 2024 for scraping behind a login, then had it overturned on JMOL in January 2025 because Ryanair could not prove the statute's $5,000 loss threshold. LinkedIn v. Proxycurl ended in July 2025 with a permanent injunction and the defendant shutting down. Reddit's 2025 suits are the ones to watch: the Anthropic case is proceeding in state court after a March 2026 ruling that its contract claims are not copyright-preempted, and the Perplexity case survived dismissal in August 2026 on a DMCA §1201 anti-circumvention theory, a materially different weapon from anything in the hiQ line.
The EU runs on different levers. Article 4(3) of the DSM Copyright Directive lets rightsholders reserve works from commercial text and data mining, but only through an appropriate, machine-readable reservation. The Hamburg court in Kneschke v. LAION doubted a plain-language terms page qualified, and the appeal court went further in December 2025: the reservation must be machine-actionable, not merely machine-legible. That is the strongest argument for putting your opt-out in robots.txt rather than your footer. The AI Act's Article 53 makes respecting those reservations an obligation on general-purpose model providers, applicable since August 2025, with enforcement powers from August 2026.
GDPR does not care that data is public. Sixteen data protection authorities restated this jointly in October 2024, and the EDPB's Opinion 28/2024 sets out what a legitimate-interest basis for scraping personal data would actually have to satisfy. Clearview AI has collected fines across at least five jurisdictions, including €30.5 million in the Netherlands in September 2024, for building a product on the theory that public means free.
How you obtained it is a separate question from what you did with it. That is the lesson of Bartz v. Anthropic, where training was held to be fair use while acquiring pirated copies was not, and which settled for $1.5 billion with final approval in July 2026. Jurisdictions differ sharply. Japan's Article 30-4 broadly permits AI training, but the Agency for Cultural Affairs has said scraping that circumvents robots.txt or authentication falls outside it, and the UK's Getty v. Stability ended with the training claim abandoned largely on territorial grounds.
How it's bypassed. By operating from somewhere that will not enforce, and by never identifying itself in the first place. A scraper running through residential proxies under a shell company in a jurisdiction with no reciprocal enforcement is not deterred by your terms of service, because you will never learn who to sue. Policy works on the actors you can name, which in practice means funded companies with something to lose, and that turns out to include most of the ones scraping at scale.
None of this is legal advice, and all of it moves. The operational takeaway is narrow and stable: your legal position is strongest when your technical position documents itself. Machine-readable permissions, enforced access controls and watermarks that prove provenance are worth more in a dispute than any argument made after the fact.
Knowing whether any of it works
Four numbers are enough to start, and most teams track none of them.
Challenge and block rate, split by outcome. How many sessions you challenged, how many passed, how many converted afterwards. A block rate that rises while conversions fall is not a win.
False-positive cost, in currency. Sample your blocks weekly and price the legitimate ones. Every team that skips this discovers the number eventually, and the discovery usually arrives as a support ticket.
Honeypot and canary hit rate. Traps are your only unambiguous signal, since nothing legitimate should ever touch them. A honeypot that has never fired is either perfectly hidden or badly placed, and you should know which.
Verified versus unverified crawler traffic. Run the reverse-DNS check across your logs and see how much traffic claims an identity it cannot prove. That single query usually reframes the whole problem.
Add one qualitative check: search for your own distinctive strings quarterly, because watermarks only work if somebody looks.
The defence nobody sells you: make the data cheap to get legitimately
Every technique above raises the cost of taking your data. Exactly one strategy lowers the demand for taking it, and no vendor has an incentive to mention that one.
If your data is scraped because it is genuinely useful, a rate-limited API, a bulk dump, a sitemap or a licensed feed converts an adversarial relationship into a metered one. You get identity, quotas, terms, an audit trail and possibly revenue; they get a stable interface and stop paying for proxies. Wikimedia did this explicitly in April 2025, publishing structured Wikipedia datasets on Kaggle so that consumers would stop parsing raw article text, and ran it alongside their blocking work. The pay-per-access experiments point the same way, from Cloudflare's pay-per-crawl beta in July 2025 through its pivot toward per-answer payment in 2026, and the RSL licensing standard launched in September 2025 by Reddit, Yahoo, Medium, O'Reilly and others.
The counter-argument deserves airtime, because blocking has costs that land on the other side of the ledger. AI search traffic is small but unusually high-intent, and Ahrefs measured it in 2025 at 0.5% of traffic and 12.1% of signups. Blocking search and answer bots is therefore the self-harm case, while blocking training crawlers is a different decision entirely, and conflating them is how sites accidentally remove themselves from the surfaces their customers are moving to. There is collateral damage too. Through 2025 and into 2026, publishers hardening against AI scraping also blocked the Internet Archive: Gannett across 241 sites, Reddit in August 2025, The New York Times by the end of that year. Whatever you make of that trade, it should be a decision and not a side effect of a managed rule.
Choosing a strategy
Layering is the whole mechanism. Every individual method here has a published bypass costing between nothing and a few dollars per thousand requests, and what makes scraping uneconomic is having to defeat four of them at once while keeping them consistent with each other.
Match the instrument to the data. Image rendering and visible challenges belong on rare, high-value, low-traffic interactions and nowhere else. Fingerprinting and behavioural scoring suit high-traffic public pages where friction has to stay invisible. Watermarking belongs on anything you might one day need to prove was yours.
Tune around real users before you tune against bots. Shared addresses, accessibility tooling, corporate VPNs and your own power users are the standing casualties of every technique in this article.
Then accept what prevention is. The realistic goal is to make harvesting slow, expensive and detectable enough that it stops being worth doing, backed by detective controls for when someone tries anyway and by a documented policy for when you need to act on it.
The web is now majority-automated, and a growing share of that automation works for your customers. A stack tuned to exclude every non-human client will eventually exclude business along with it. So the work that holds up is unglamorous: identify what is at the door, decide on purpose which of it you want, and publish that decision somewhere a machine can read.