If you have spent any real time around web scraping, you already know there is no single switch that makes a site "unscrapable." Instead, there is a toolbox of defensive techniques, each with its own strengths, costs, and ways of being defeated. People ask me all the time which method works best, so this post pulls the most common approaches together in one place and explains, honestly, where each one helps and where it falls short.
The short version: scraping prevention is not a wall you build once. It is an ongoing trade-off between how much friction you are willing to impose on real visitors and how much effort you want to force on whoever is collecting your data. If you think I have left something important out, let me know.
Before you start defending, it helps to confirm you actually have a problem. If you want to first determine whether your pages are being harvested, see the companion guide on how to detect that your site is being scraped.
The techniques below fall into three broad families, depending on where in the request lifecycle they intervene: at the server, in the browser, or in the content itself.
Type A — Server-Side Filtering Based on Incoming Requests
These measures inspect raw HTTP traffic and decide what to allow before a single byte of your page is rendered. They are cheap to reason about but easy for a determined scraper to route around.
1. Blocking suspicious IP addresses
The most basic defense is to ban IP addresses that behave abnormally — for example, a single address firing hundreds of requests a minute, or a cluster of requests all originating from the same geographic location. On its own this can knock out lazy bots quickly.
The problem is that almost all serious scraping today runs through rotating proxy pools. Block one address and the scraper simply hops to the next, staying effectively IP-independent. You can automate the banning with a tool like Fail2Ban, which watches your logs and drops offenders into firewall rules, but as a long-term strategy IP blocking buys you very little against anyone using residential or datacenter proxies at scale.
2. Filtering at the DNS / edge layer
A more sophisticated option is to put a filtering service in front of your origin server so bad requests are dropped before they ever reach your infrastructure. In practice this usually means routing traffic through a content delivery and security network that profiles visitors at the edge.
Cloudflare is the canonical example. Its Bot Management product scores every request in real time using machine learning, TLS fingerprinting, and behavioral signals, while the older Scrape Shield feature set — now folded into Cloudflare's Web Application Firewall — adds email obfuscation, hotlink protection, and invisible tracking beacons that reveal where your content is being republished. (This is the same kind of edge-level protection described in the earlier write-up on ScrapeShield.) The upside is that you outsource the hardest part — maintaining detection logic against an adversary that constantly evolves — to a provider that sees a huge slice of global traffic. The downside is cost and dependency on a third party.
3. A custom script that tracks behavior and drops bad requests
In some cases you can fingerprint the pattern a crawler uses to walk your URLs — the order it requests pages, the intervals between hits, the absence of asset requests a real browser would make — and trigger defensive measures when that pattern appears. This typically means running a shell or application-level script on the server that watches request URLs and flips on protection when something looks automated.
Two caveats. First, every extra check adds latency, so this kind of inline analysis can measurably slow your service for everyone. Second, the moment you detect a crawl pattern and defend against it, the operator can change the pattern, and your carefully tuned rule is worthless again. It is a maintenance treadmill.
4. Limiting request frequency (rate limiting)
You can cap how many requests — or how much data — a given client may pull within a time window. The art here is choosing thresholds that throttle a relentless scraper without punishing a genuinely active human reader. Most web servers and proxies support this natively; for example, nginx offers the limit_req module out of the box.
Rate limiting works well against crude scripts. It works far less well once a scraper is reconfigured to imitate human pacing using browser-automation tools such as Selenium (see also What is Selenium WebDriver?), Playwright, Puppeteer, or the Python mechanize library. Note that older tools from this category, such as iMacros, have since been discontinued (Progress declared it end-of-life in 2023, though the earlier iMacros tutorial still illustrates the technique). Once requests arrive at a believable human cadence, frequency caps stop distinguishing friend from foe.
5. Capping maximum session length
A close cousin of rate limiting: terminate any session that stays active beyond a fixed duration. The reasoning is that bots tend to grind for hours where humans do not. In reality, modern scrapers routinely perform proper session authentication and break long jobs into short, human-looking sessions, so simply cutting off long sessions rarely accomplishes much on its own.
Type B — Browser-Level Identification and Prevention
Server-side filtering only sees the request. The next family of techniques forces the client to prove it is a real browser by executing something a plain HTTP client cannot.
1. CAPTCHAs on sensitive pages
The CAPTCHA is the classic gate: ask the visitor to solve a challenge a machine supposedly cannot. Deployed on key pages — login, checkout, search — it does stop a large share of casual automation. Modern, low-friction implementations such as Google reCAPTCHA, hCaptcha, and Cloudflare Turnstile try to verify visitors invisibly so most humans never see a puzzle.
There are two real costs. First, any visible challenge adds friction and can measurably reduce conversions and traffic — every box you put in front of a human is a chance for them to leave. Second, an entire industry of CAPTCHA-solving services exists specifically to defeat these challenges, combining cheap human labor with automated solvers. If your opponent is willing to pay fractions of a cent per solve, the CAPTCHA becomes a speed bump rather than a wall.
2. Injecting JavaScript logic into the response
Here you embed JavaScript that must run before or alongside your HTML, perform some computation, and return a value the server checks. If the script does not execute correctly — as happens with a naive HTTP client that does not run JavaScript — the server can withhold the real content, deliberately serve malformed markup, or refuse the request entirely. You can scope this to an entire page or to just the valuable parts (prices, contact details, listings).
Because plain requests-style scrapers don't run JavaScript, this raises the bar meaningfully: to bypass it, an operator must switch to a full headless browser (Selenium, Playwright, Puppeteer) or to a JavaScript runtime that evaluates your injected logic — all of which are slower, costlier, and more fragile to run at scale. It does not make scraping impossible, but it makes the cheap path unavailable.
Type C — Content-Level Protection
The final family doesn't try to identify the client at all. Instead it changes the content so that even a client that fetches the page successfully cannot easily parse the data out of it.
1. Rendering important data as images
A long-standing trick is to deliver sensitive values — prices, phone numbers, email addresses — as images rather than selectable text. A scraper that reads the HTML finds nothing useful where the number should be.
The trade-offs are significant. Text baked into images is invisible to search engines, so this directly harms your SEO and accessibility. And the defense is only as strong as the attacker's patience: an optical character recognition engine such as Tesseract can read text back out of those images, restoring the data the obfuscation was meant to hide.
2. Frequently changing the page structure
One of the more effective content-level tactics is to keep rewriting your markup — not just shuffling element IDs and CSS class names, but reorganizing the entire DOM hierarchy. Scrapers depend on stable selectors; every time the structure changes, their extraction logic breaks and has to be rewritten.
The catch is symmetry of effort. A meaningful restructuring forces you to redo styling and templating each time, which is expensive and error-prone on your side too. You are deliberately spending your own engineering time to impose engineering time on the scraper. Against a casual data collector that is often enough to make them give up; against a well-resourced competitor it is just a recurring nuisance for both of you.
Combining Defenses and a Note on Reality
No single item above is sufficient, because each one defeats a different class of scraper. IP blocking stops the unsophisticated; JavaScript challenges stop plain HTTP clients; CAPTCHAs raise the per-request cost; structural churn breaks brittle parsers. The practical approach is layering: stack measures so that defeating all of them at once is more expensive than your data is worth to the person trying to take it.
For sites where this matters commercially, it is usually more economical to adopt a managed bot-mitigation platform — Cloudflare Bot Management, DataDome, Imperva, or Akamai Bot Manager — than to maintain detection logic yourself. These services pool signal across enormous traffic volumes, which is exactly the advantage an in-house rule set can never match.
Two final, honest points. First, you should also state your intentions clearly: a well-formed robots.txt won't stop a malicious bot, but it tells well-behaved crawlers (including search engines and increasingly AI crawlers) what they may and may not access, and it strengthens your position if a dispute ever becomes a legal one. Second, accept the underlying truth of this whole field: anything a browser can render, a sufficiently motivated and funded operator can eventually extract. The realistic goal of scraping prevention is not perfect immunity — it is making your data costly enough to collect that most people decide it isn't worth the trouble.
If you think I've missed a technique worth covering, I'd genuinely like to hear it.