How to Detect If Your Website Is Being Scraped

The modern web is full of automated data collectors. Some are welcome, like the crawlers that index your pages for search engines or fetch a preview when someone shares a link. Others are not: competitors lifting your catalog and prices, content farms re-publishing your articles, or bots hammering your endpoints hard enough to slow the site down for real visitors. Add to this the recent surge in AI training crawlers, and most sites now see a meaningful share of their traffic coming from machines rather than people.

If you run a website, the first practical question isn't "how do I stop scraping?" but "how do I know it's happening, and how is it being done?" You can't defend against something you can't see. This guide is a field manual for detection. It catalogs the signals that reveal automated collection, organized from the ones a non-technical owner might notice down to the deep network- and browser-level fingerprints that modern anti-bot systems rely on.

This article is about spotting scraping. If you're more interested in stopping it once you've found it, see the companion piece on how to prevent web scraping, and for the cat-and-mouse view from both sides, 7 ways to protect a website from scraping (and how those protections get bypassed).

A useful way to think about the signals

Before the list, one organizing idea worth keeping in mind: does the signal survive a proxy?

Most amateur scrapers give themselves away through their IP address: too many requests, too fast, all from one place. Those signals collapse the moment the scraper rotates through a pool of proxies or residential IPs. The signals that keep working are the ones tied to what the client is and how it behaves: its TLS handshake, its HTTP header order, the way it navigates, whether it executes JavaScript, and whether your content suddenly shows up somewhere else. Throughout this guide, I'll flag which signals hold up against a proxied attacker, because that distinction is what separates a quick detection win from a durable one.

The categories below move roughly from "no code required" to "deep technical fingerprinting."

1. Business and operational signals (no code required)

These are the symptoms a site owner or marketing team can notice without ever opening a log file.

Your content appears somewhere it shouldn't

The clearest sign that data has been lifted is finding it republished elsewhere. Search your distinctive product descriptions, article sentences, or unique strings in a search engine and watch for new pages with identical content. Duplicate copies of your material competing for the same keywords can quietly siphon off visitors and damage your own search rankings. This is work your SEO or content team should be doing routinely. Survives a proxy: yes — it has nothing to do with the scraper's IP.

Bandwidth and throughput anomalies

Aggressive scraping can saturate your bandwidth or push response times up, degrading the experience for genuine users. If you watch server load, request volume, and egress bandwidth with proper monitoring, an unusual sustained spike is often your earliest warning that something is pulling pages at machine speed. The traffic shape — flat, relentless, around the clock — looks nothing like the daily rhythm of human visitors. Survives a proxy: partly — the aggregate load is visible even if it's spread across many source IPs.

2. Server-side and log-based signals (no JavaScript needed)

These signals live in your access logs and edge/WAF telemetry. You don't need the client to run any code to catch them, which makes them cheap and always-on.

High request rate from a single IP

The classic tell. A human browses a handful of pages a minute; a naive scraper fetches dozens or hundreds. Rate limiting at the server or CDN can throttle this, but it only addresses the unsophisticated case — a scraper distributing requests across a proxy pool stays under any per-IP threshold. Survives a proxy: no.

Suspiciously regular timing

Humans are irregular. They pause, read, get distracted, click in bursts. Automated clients often request pages at near-identical intervals, producing an unnaturally even rhythm. Even when request volume looks reasonable, machine-perfect cadence from one source is a strong indicator. Survives a proxy: only if you can still attribute the requests to one actor (e.g., via a session cookie or fingerprint); pure IP-based timing analysis breaks under rotation.

Telltale or headless user-agents

Every request carries a User-Agent header. Default HTTP clients announce themselves plainly — strings containing python-requests, Go-http-client, curl, Scrapy, Java, or HeadlessChrome stand out immediately against the Chrome, Safari, and Firefox strings real visitors send. You should be logging and reviewing the distribution of user-agents hitting your site; an automated or development-language agent looks nothing like your typical user. The catch: user-agents are trivially spoofable, so a plausible-looking user-agent proves nothing — but it does need to be checked for consistency against the deeper signals below. Survives a proxy: yes, until the scraper bothers to forge a realistic string.

Missing or inconsistent request headers

A real browser sends a rich, predictable set of headers in a particular order: Accept, Accept-Language, Accept-Encoding, Referer, and modern Client Hints such as Sec-CH-UA, Sec-Fetch-Site, and Sec-Fetch-Mode. Simple scrapers omit many of these, send them in the wrong order, or present a header set that doesn't match the browser their user-agent claims to be. The order and completeness of headers is harder to fake convincingly than the user-agent string alone. Survives a proxy: yes.

Support files that never get requested

When a browser loads a page, it automatically pulls in its dependencies: stylesheets, scripts, fonts, images, and favicon.ico. A bare-bones scraper that only wants the HTML never requests any of these. A session that fetches hundreds of pages but never once asks for a CSS or JS file is behaving unlike any browser. Caveat: this is softer than it used to be — ad blockers, privacy tools, and text-only browsers can legitimately skip dependencies, and image-heavy assets are often lazy-loaded only on interaction. Treat it as one weak signal among many, not proof. Survives a proxy: yes.

Abnormal navigation paths

Humans arrive somewhere and click their way deeper, leaving a breadcrumb trail in the Referer header and a logical page-to-page sequence. Scrapers frequently jump straight to pages that aren't linked from anywhere a user would naturally start — paginated URLs, internal IDs, or deep detail pages requested with no referring page. A client that lands on "page 247" of a listing cold, with no path leading there, is almost certainly working from a generated URL list. Survives a proxy: yes — it's about behavior, not origin.

HTTP/2 fingerprinting

Beyond headers, the way a client speaks HTTP/2 — its frame settings, stream priorities, and pseudo-header ordering — varies between real browsers and automation libraries. Edge providers compare this against the user-agent's claim, and a mismatch (a "Chrome" client whose HTTP/2 behavior matches a Go library) is a high-confidence automation signal. Survives a proxy: yes — it's a property of the client software, not the network path.

TLS fingerprinting (JA3 / JA4)

This is one of the most powerful server-side signals available today, and it's worth understanding in detail. When any client opens an HTTPS connection, it sends a TLS ClientHello message containing its supported cipher suites, extensions, elliptic curves, and protocol versions — in a specific order. That combination forms a fingerprint that identifies the underlying software stack before a single byte of HTTP is exchanged. Python's requests (via OpenSSL) produces a different fingerprint than Chrome, even when both send an identical user-agent.

The original JA3 method, created at Salesforce in 2017, became less reliable once Chrome began randomizing its TLS extension order. Its successor, JA4+ from FoxIO (by JA3's original author), sorts extensions before hashing to defeat that randomization and has since become the de facto standard. As of 2026 it's integrated into major edge platforms — Cloudflare documents JA3/JA4 as bot-profiling inputs, and it's used by Akamai, AWS, and others.

The key detection insight: look for a mismatch between layers. If a request's HTTP user-agent says "Chrome 120" but its TLS fingerprint matches a default Python or Go stack, you've caught a spoofed client with very high confidence. Survives a proxy: yes — rotating IPs does nothing to change the TLS handshake. This is precisely why proxies alone don't keep determined scrapers hidden.

3. Client-side and JavaScript signals

If you're willing to run JavaScript in the visitor's browser, a whole new layer of signals opens up. These catch automation that has already passed the network-level checks.

Automation flags: navigator.webdriver and CDP traces

Browsers driven by automation expose navigator.webdriver, which is set to true when Selenium, Playwright, or Puppeteer are in control. Scrapers routinely suppress this flag (with the --disable-blink-features=AutomationControlled launch argument, or by overriding the property), so the modern approach is to look for the side effects of automation rather than the flag itself:

• Property tampering. In a genuine browser, webdriver is defined on Navigator.prototype with a native getter. Stealth plugins that redefine it on the instance leave a non-native getter — the act of hiding is itself detectable.
• CDP artifacts. Puppeteer and Playwright drive Chrome through the Chrome DevTools Protocol. Enabling its domains produces observable traces, and injected execution contexts can surface in error stack traces (__puppeteer_evaluation_script__, __playwright_evaluation_script__).
• Injected globals. Automation frameworks leave variables in the page scope, such as window.__playwright__binding__ or window.__pwInitScripts.

Survives a proxy: yes — these are properties of the automated browser, entirely independent of IP.

Environment and fingerprint inconsistencies

A headless or automated browser rarely looks exactly like the real thing. Detectors compare what the browser claims against what it can actually do:

• Canvas, WebGL, and AudioContext fingerprints rendered by automation often differ from those of a real GPU-backed browser, or use a software renderer where hardware would be expected.
• Missing plugins, empty navigator.plugins, odd navigator.languages, or a timezone that contradicts the IP's geolocation.
• Default viewport and screen dimensions that match a headless launch profile rather than a physical display.

A useful mental model: a cluster of small lies that together match a known automation tool reads as a bot, even when no single property is conclusive. Open-source projects like CreepJS demonstrate how many of these vectors can be combined.

Behavioral analysis

Real users generate organic input: mouse paths with curvature and jitter, variable scroll speed, hesitation, focus and blur events. Automated sessions tend to produce no mouse movement at all, perfectly straight or instantaneous interactions, zero idle time, and page-to-page timing far faster than a human could read. Tracking these interaction patterns over a session is one of the harder signals for a scraper to fake convincingly, which is why it's increasingly central to commercial detection. Survives a proxy: yes.

Honeypot traps

A honeypot is a link or form field that's invisible to humans (hidden with CSS, positioned off-screen, or marked aria-hidden) but present in the DOM. A human never sees it and never interacts with it; a scraper that indiscriminately follows every link or fills every field walks straight into it. Any request to a honeypot URL or submission touching a honeypot field is a near-certain bot. It's cheap to deploy and produces almost no false positives. Survives a proxy: yes.

Note on the arms race. None of the JavaScript signals above is foolproof. Anti-detect frameworks like puppeteer-extra-plugin-stealth, undetected-chromedriver, and newer CDP-minimal tools (nodriver, selenium-driverless) exist specifically to patch these tells. Interestingly, patch-based stealth often replaces one detection signal with another — a recognizable "this is a stealth tool" fingerprint — so layering multiple independent signals remains the reliable strategy.

4. Infrastructure and reputation signals

Where a request comes from still carries information, even in a proxy world.

• Datacenter vs. residential origin. Traffic from known cloud/hosting ASNs (AWS, GCP, OVH, DigitalOcean) is far more likely to be automated than traffic from residential ISPs. Determined scrapers counter this with residential proxy networks, which is why IP reputation is a contributing score, not a verdict on its own.
• ASN and IP reputation databases. Known proxy ranges, VPN exit nodes, and previously flagged addresses can be scored against threat-intelligence feeds.
• Geographic and ASN clustering. A flood of requests spread across hundreds of unrelated residential IPs that nonetheless share timing, headers, or a fingerprint points to a single coordinated actor behind a proxy pool.

Survives a proxy: partially — this category is specifically about characterizing the proxy, so it degrades against high-quality residential networks but still adds signal.

5. Off-the-shelf detection services

You don't have to build all of this yourself. Commercial and open-source tools bundle the network, fingerprint, and behavioral signals above into managed scoring systems:

• Cloudflare Bot Management and the free Turnstile challenge
• DataDome
• HUMAN Security (formerly PerimeterX)
• Akamai Bot Manager
• Google reCAPTCHA and hCaptcha
• FingerprintJS BotD, an open-source bot-detection library

These services assign each request a bot score from many simultaneous signals and let you decide what to do at each threshold — allow, challenge, slow, serve alternate content, or block.

Practical notes for site owners

Watch your logs — that's where most of this lives. Track the distribution of user-agents, per-IP request rates, header completeness, and which clients never touch your static assets. A surprising amount of scraping is caught by simply looking at access logs that most teams never read.

Consider offering an API. If you're being scraped heavily for data you'd be willing to share anyway, a documented API is often the pragmatic answer. Most consumers would rather hit a clean, rate-limited, authenticated endpoint than maintain a brittle scraper, and an API gives you far better visibility into who is using your content and how much. It also dramatically reduces the load that ad-hoc scraping puts on your servers.

Set expectations with robots.txt — but don't rely on it. The Robots Exclusion Protocol lets you state your crawling preferences and is respected by well-behaved bots like search engines. Malicious scrapers ignore it entirely, so treat it as a signpost for good actors and a baseline — a client that requests pages you've disallowed is showing its hand.

No single signal is decisive. Every individual tell here has a false-positive story: privacy tools skip assets, corporate proxies share IPs, accessibility software moves the cursor strangely. Robust detection comes from combining signals into a score, not from any one rule. The strongest setups stack an IP-independent layer (TLS/HTTP fingerprinting, behavioral analysis, honeypots) on top of the cheap IP-based checks, precisely so that a scraper hiding behind a proxy still trips something.

In closing

Detecting scraping is less about a single silver-bullet check and more about layered observation: knowing what normal human traffic looks like, watching for the network and browser fingerprints that automation can't fully hide, and combining weak signals into a confident verdict. Start with your logs and the cheap server-side signals, add fingerprinting and behavioral checks as your needs grow, and reach for a managed service when the volume justifies it.

Once you've confirmed your site is being scraped and understood how, the next step is deciding what to do about it. For that, read up on preventing web scraping and the broader picture of protections and the techniques used to bypass them.

Know a detection method that isn't covered here? The field moves fast — additions and corrections are always welcome.