Web scraping is the fastest way to turn public web pages into structured data, and as more teams rely on it, more sites push back. Anti-scraping techniques are the defenses a site deploys to tell bots from humans and shut the bots down — sometimes to protect infrastructure, sometimes to guard content or pricing. This guide surveys the main defenses you'll meet in 2026 and the practical, responsible ways scrapers get around each one. It's the offense-side companion to our deep dive on how anti-scraping protection works under the hood.
A note before the tactics: bypassing an anti-scraper is a technical exercise, not a license to ignore terms of service, hammer someone's servers, or collect personal or copyrighted data you have no right to. Scrape public data, throttle politely, and respect the law in your jurisdiction.
How anti-scraping systems decide you're a bot
Every defense boils down to one question the server asks before it hands over content: is this a real person on a real browser, or a program? Modern anti scraping stacks look at four broad signal groups:
- Network / IP — the address, its reputation, and how many requests come from it.
- Request shape — headers, header order, HTTP version, and TLS handshake.
- Client fingerprint — what the browser (or lack of one) reveals about itself.
- Behavior — timing, mouse movement, navigation patterns, and honeypot interactions.
Beat a site and you're beating one or more of these. Here are the specific techniques and their counters.
1. Rate limiting and speed detection
The defense. Bots fetch pages far faster than any human could click. Sites track requests per IP over time and flag anyone who reads dozens of pages a second or marches through a paginated list at machine speed.
The bypass. Slow down and randomize. Add jittered delays between requests instead of a fixed sleep, cap your concurrency, and respect Retry-After headers and robots.txt crawl-delay hints. Randomizing the order in which you visit pages also helps you blend in.
import time, random
def polite_delay(base=2.0, jitter=1.5):
time.sleep(base + random.uniform(0, jitter))
Pacing costs you throughput, so combine it with rotation (below) rather than trying to be both fast and invisible from one address.
2. IP bans and blocklists
The defense. The oldest and most common control: too many requests from one IP, or an IP from a known datacenter range, gets throttled or blocked outright.
The bypass. Route requests through a pool of rotating proxies so no single address carries the whole load. For tough targets, prefer residential or mobile proxies, whose IPs look like ordinary home and cellular users; datacenter IPs are the easiest to identify and ban. Rotate per request for large crawls, or hold a sticky IP per session when you need to stay logged in.
3. HTTP header and TLS fingerprinting
The defense. A default requests or curl call sends a tell-tale User-Agent and a header set that doesn't match any real browser. Beyond headers, sites inspect the TLS handshake (JA3/JA4 fingerprints) and the HTTP/2 frame order — signatures a scripting library produces differently from Chrome or Firefox.
The bypass. Send a complete, consistent set of browser-like headers (User-Agent, Accept, Accept-Language, Sec-CH-UA, Referer) and keep them coherent with each other. To defeat TLS fingerprinting, use a client that mimics a real browser's handshake — libraries such as curl_cffi (with impersonate="chrome") or tls-client in Python, or simply drive a real browser. Rotate User-Agents from a small set of current, real strings, not random junk.
from curl_cffi import requests
# Impersonates a real Chrome TLS + HTTP/2 fingerprint
r = requests.get("https://example.com", impersonate="chrome")
4. JavaScript challenges and encrypted content
The defense. Many sites render data with JavaScript or hide it behind a JS challenge: the page ships a script that must run and compute a token before real content loads. Some obfuscate or "encrypt" payloads that only client-side code can unpack. A plain HTTP client sees an empty shell.
The bypass. Use a real browser engine. Headless Playwright, Puppeteer, or Selenium execute the JavaScript exactly as a user's browser would, so the challenge resolves and the data appears. Pair headless browsers with stealth plugins that patch the obvious automation tells (like navigator.webdriver). When the site ultimately fetches data from an internal JSON endpoint, watch the network tab — calling that API directly is often faster and sturdier than rendering the whole page.
5. Browser fingerprinting
The defense. JavaScript can profile the client deeply: canvas and WebGL rendering, installed fonts, screen size, timezone, hardware concurrency, and the presence of automation flags. A headless browser with default settings has a fingerprint that screams "bot."
The bypass. Make the automated browser look ordinary. Run in non-default (or headful) mode, set a realistic viewport and timezone that match your proxy's geolocation, and use anti-detect tooling or stealth patches. Keep the fingerprint consistent with everything else — a US residential IP paired with a browser reporting a Moscow timezone is an instant flag. Consistency across IP, headers, and fingerprint matters more than any single value.
6. CAPTCHAs
The defense. When a request looks suspicious, the site interrupts with a challenge — click-to-verify, distorted text, image grids (reCAPTCHA), or invisible scoring (reCAPTCHA v3, hCaptcha, Cloudflare Turnstile).
The bypass. First, avoid triggering them: good proxies, clean fingerprints, and human-like pacing prevent most challenges from ever appearing. When one does appear, route it to a CAPTCHA-solving service — services such as 2Captcha, Anti-Captcha, or CapSolver return a solved token via API in seconds for a small fee, which your scraper submits with the form. Solving CAPTCHAs in bulk is a smell, though: if you're hitting them constantly, fix the upstream signals rather than paying to solve thousands.
7. Honeypot traps
The defense. A honeypot is a link or field invisible to humans (hidden with CSS, zero size, or display:none) but present in the HTML. A naive crawler that follows every link or fills every field walks straight into it, and the site instantly knows it's a bot.
The bypass. Don't blindly follow all links or fill all inputs. Use precise selectors (specific XPath or CSS) that target the real navigation, and skip elements that are hidden or styled off-screen. If you drive a headless browser, check element visibility before interacting.
8. Login walls
The defense. Sites gate valuable data behind authentication, then watch logged-in accounts for bot-like behavior and rate-limit or ban per account.
The bypass. Automate the login to obtain session cookies, then reuse that session for subsequent requests. Keep one proxy IP sticky per account so the session doesn't jump countries mid-visit, and pace account activity conservatively — a banned account is costlier to replace than a banned IP. See scraping login-protected sites for the full workflow.
9. Layout variation and structural traps
The defense. Some sites vary page layouts deliberately — pages 1–10 of a listing render differently from pages 11–20 — so a scraper hard-coded to one structure breaks.
The bypass. Write resilient parsers. Prefer stable attributes (IDs, data-* attributes, semantic tags) over brittle absolute paths, branch on which layout you detect, and fail loudly with logging so you notice when a template changes instead of silently collecting empty rows.
Commercial anti-bot platforms (and the Distil case)
Many sites don't build defenses in-house; they buy an anti-bot product that bundles all of the above. If you've seen references to Distil Networks in older scraping write-ups, note that Distil was acquired by Imperva in 2019 and now lives on as Imperva Advanced Bot Protection. The current landscape of managed anti-scraping tools includes:
- Cloudflare Bot Management / Turnstile
- Imperva Advanced Bot Protection (formerly Distil)
- DataDome
- HUMAN (formerly PerimeterX)
- Akamai Bot Manager
These platforms combine IP reputation, TLS and browser fingerprinting, behavioral scoring, and adaptive CAPTCHAs, and they update their models continuously. There's no magic one-liner to defeat them. The reliable approach against any of them is the same stack of fundamentals working together: quality residential/mobile proxies, a real browser with a consistent fingerprint, human-like pacing, and CAPTCHA handling only as a fallback. When you hit a JS or interstitial challenge from one of these vendors, a properly configured headless browser (with stealth) that lets the challenge script run is usually what gets you through — brute force does not.
Putting it together
No single trick beats a modern anti scraper; layered defenses require layered answers. A durable scraper generally combines:
- Rotating residential/mobile proxies for the network layer.
- A browser-accurate client (real headers, matching TLS fingerprint, or a full headless browser) for the request and fingerprint layers.
- Human-like behavior — jittered timing, sane concurrency, honeypot avoidance.
- CAPTCHA solving as a last resort, not a crutch.
- Resilient parsers that survive layout changes.
Get those five right and most sites stop noticing you. And always weigh whether an official API or a licensed data source solves the problem without the cat-and-mouse at all.
If maintaining this stack against sites that fight back isn't how you want to spend engineering time, scraping.pro runs it as a done-for-you data extraction service — proxies, anti-bot handling, and delivery included — or ships the finished dataset on a schedule as a data feed.