Scraping JavaScript-Protected Content

You request a product listing, parse the HTML, and find nothing in it. No titles, no prices, no product names. What came back is a skeleton of empty <div>s and a pile of <script> tags. Your browser was showing forty products a minute ago, and the bytes you just parsed contain none of them.

That wall has two sides that look identical from where you stand. On one side is a page that assembles itself in the browser and never gave you a thought. On the other is a page that examined you and decided you were not a browser. The diagnosis is the same for both. The tools overlap. The bill does not.

The scale of the first problem is easy to state. The HTTP Archive's 2024 crawl, published as the Web Almanac JavaScript chapter in March 2025, put the median page at 558 KB of JavaScript over 22 requests on mobile and 613 KB over 23 on desktop. Fetching a document and fetching its content stopped being the same operation some years ago, and most scraping guidance has not caught up.

Everything below was rechecked against vendor documentation, package indexes and the relevant specifications on 13 August 2026. Code was written against Playwright 1.62.0 (released 31 July 2026) and Selenium 4.47.0 (released 10 August 2026), both on Python 3.13.

What a rendered page costs

Start with the price, because the price decides the method.

Zyte publishes per-request rates for Zyte API broken out by how hard the target is. On 13 August 2026 the plain-HTTP tiers ran $0.13, $0.23, $0.44, $0.70 and $1.27 per 1,000 requests, from Simple to Advanced. The browser-rendered tiers for those same five buckets ran $1.01, $2.01, $4.02, $8.04 and $16.08. Same targets, same data, roughly eight to thirteen times the money for asking a browser to do it.

ScrapingBee prices the identical decision in credits: 1 for a plain fetch, 5 with JavaScript rendering, 25 for a premium proxy with it, and 75 for the stealth proxy, which only works with rendering switched on. On the $49 Freelance plan and its 250,000 credits, that is about $0.20, $0.98 and $14.70 per thousand requests. The vendors selling rendering have priced it at up to seventy-five times a plain GET. Believe them.

Self-hosting moves the cost from an invoice to a machine, and the shape survives. Browserless sizes its own Docker deployments at 2 cores and 4 GB of RAM for 5 to 10 concurrent sessions, 4 cores and 8 GB for 10 to 20, and 8 or more cores with 16 GB for 20 to 50. That is somewhere between 300 and 800 MB of memory for every page you want open at once, plus a core per handful of them.

The arithmetic that picks your architecture. One wasted second per page across a ten-thousand-page crawl is just under three hours. A rendered page that takes four seconds instead of a 300 ms HTTP fetch is not 13 times slower in wall time only, it is 13 times more infrastructure for the same nightly window. And the median page's JavaScript alone, at 558 KB, costs about $1.40 per thousand pages in pure bandwidth through residential proxies at Bright Data's advertised $2.50 per GB, before a single image is downloaded.

Rendering is not the hard part. Rendering everything is.

Rendered, protected, or refused

Three situations produce an empty parse, and they need different work.

The page is JavaScript-rendered. A single-page app in React, Vue, Angular or Svelte ships an HTML shell, then fetches its data and paints it into the DOM. Nobody is trying to stop you. The data is often sitting in the document already, serialised into a <script> block, and the fastest scraper for this case never opens a browser.

The page is JavaScript-protected. The site runs a script that measures your browser and only releases content if the measurements pass. This is anti-bot machinery, not rendering. Cloudflare's own documentation describes the mildest form plainly: a non-interactive challenge makes the determination "based on the limited information attained from their browser signals via an injected JavaScript," typically inside five seconds, with no visible puzzle. Above that sits the managed challenge, which picks its severity per request. A plain HTTP client fails these before it reaches any content. A browser passes or fails depending on details covered further down.

You were refused before any JavaScript ran. The content is in the HTML, and you never receive the HTML, because the TLS handshake and HTTP/2 settings gave you away at the edge. No amount of rendering helps here. The fix is a client that handshakes like a real browser.

The first case is a parsing problem. The second is a fingerprinting problem. The third is a networking problem. Guessing wrong costs you a week.

Step 1: prove it before you fix it

Open the target in Chrome and press F12. Two panels answer the question.

1. Network, filtered to Fetch/XHR. Reload and watch what the page fetches for itself. Chrome's DevTools network reference documents the resource-type filter and the copy menu you will need in a minute. A background call that returns JSON full of your data is the shortest path in this article.
2. View source against Elements. View Page Source shows the bytes the server sent. The Elements panel shows the live DOM after scripts ran. Data present in the second and absent in the first is data assembled in the browser.

Then confirm it in code, because the eye is a poor diff.

import requests

r = requests.get("https://example.com/products", timeout=30)
html = r.text

print(r.status_code, len(html))
print("Dell Latitude" in html)          # the visible product name
print("application/ld+json" in html)    # structured data in the document
print(html.count("<script"))            # how much work happens client-side

Presence is not visibility. A False on the product name does not prove the data is missing, and this is where the first half-day disappears. The string may be there with & in the middle of it, split across markup as <span>Dell</span> Latitude, written as 1 299,00 where you searched for $1,299.00, or withheld because you have no consent cookie and got a banner instead. A 2 KB body is a shell. A 300 KB body that lacks your string usually holds your data in another shape.

A 403, a 503 with a challenge page, or a body that mentions "Just a moment" is not the JavaScript-rendering problem. That is the third case, and Step 2B will not fix it.

Step 2A: the data is usually already in front of you

Scrapy's documentation states the priority order about as directly as anyone can: "the recommended approach is to find the data source and extract the data from it." Their guide to dynamically-loaded content puts a headless browser last, after finding the source and after reproducing the requests by hand.

There are two sources, and people forget the first one.

The data is embedded in the HTML. Server-rendered frameworks serialise their state into the document so the client can hydrate without a second round trip. That state is your dataset, sitting in a <script> tag, already parsed by the server. Look for <script type="application/ld+json"> blocks, and for any large script that assigns a JSON-shaped object to a global. The catch is that JavaScript object literals are not JSON: unquoted keys, single quotes, trailing commas and undefined all break json.loads. Scrapy's own docs point at chompjs (1.4.1, released 17 May 2026) for exactly this, and it is worth having installed before you need it.

import json, re
import chompjs

# 1. Standards-track structured data, if the site publishes it.
for m in re.finditer(
    r'<script[^>]+type="application/ld\+json"[^>]*>(.*?)</script>',
    html, re.S):
    block = json.loads(m.group(1))

# 2. Framework state: valid JavaScript, invalid JSON.
m = re.search(r'window\.__INITIAL_STATE__\s*=\s*(\{.*?\});', html, re.S)
if m:
    state = chompjs.parse_js_object(m.group(1))

The data comes from an endpoint. In the Network panel, find the Fetch/XHR request whose response holds your records, right-click it, and use Copy as cURL or Copy as fetch. Chrome also offers Copy all as cURL for a filtered set, which is how you find the paginated sibling of the call you just captured. Import that into your HTTP client of choice and you have structured records with no parsing at all.

import requests

s = requests.Session()
# Load the real page first: it sets the cookies and hands you the token.
s.get("https://example.com/products", headers={"User-Agent": UA})

resp = s.get(
    "https://example.com/api/v1/categories/112/products",
    headers={
        "User-Agent": UA,
        "Accept": "application/json",
        "Referer": "https://example.com/products",
        "X-Requested-With": "XMLHttpRequest",
    },
    params={"page": 1, "per_page": 100},
    timeout=30,
)
resp.raise_for_status()
for product in resp.json()["products"]:
    print(product["name"], product["price"])

The usual complication is proof that you came from the site: a CSRF token, a session cookie, a Bearer header. Load the ordinary page first, harvest what it sets, then call the endpoint. Our longer treatment of that pattern lives in scraping dynamic content.

The endpoint can refuse you before it reads a single header. Anti-bot products fingerprint the TLS handshake and the HTTP/2 SETTINGS frame, and Python's requests looks nothing like Chrome at that level. The symptom is a 403 on a URL that works perfectly when pasted into a browser, with identical headers on both sides. The fix is not a browser, it is a client that handshakes like one. curl_cffi (0.16.0, released 1 August 2026) wraps curl-impersonate and reproduces browser TLS/JA3 and HTTP/2 fingerprints, with HTTP/3 fingerprints added in 0.15.0.

from curl_cffi import requests as cffi

resp = cffi.get(
    "https://example.com/api/v1/categories/112/products",
    impersonate="chrome",     # or a pinned build, e.g. "chrome124"
    timeout=30,
)
print(resp.status_code, len(resp.content))

That one substitution rescues a large share of the endpoints people give up on and go render instead. It costs a few milliseconds. A browser costs a few hundred megabytes.

Where this path genuinely ends: when the request carries a signature computed by obfuscated JavaScript, or when the endpoint only opens after a challenge has been solved in a real browsing context. Then you render.

Step 2B: render, and know what you actually launched

Two things in the previous version of this article were wrong, and both are wrong in most tutorials on this subject.

Correction one: networkidle is discouraged by the people who wrote it. Playwright's API reference marks the wait_until="networkidle" option DISCOURAGED, in those capitals, with the note: "Don't use this method for testing, rely on web assertions to assess readiness instead." On a page with polling, analytics beacons or a websocket, the network never goes idle for 500 ms and you wait out the full timeout on every URL. On a fast page it returns before the second render pass. It is a timer wearing a lab coat.

Correction two: headless Playwright does not run Chrome by default. Playwright's browsers documentation says it plainly: "Playwright ships a regular Chromium build for headed operations and a separate chromium headless shell for headless mode." That headless shell is the old headless implementation, which Chrome split out of the main binary at version 132.0.6793.0 into a standalone chrome-headless-shell download. Chrome's own headless documentation describes the trade: the shell is "a lightweight wrapper around Chromium's //content module" tuned for screenshotting and scraping, while new headless "is the real Chrome browser, and is thus more authentic, reliable, and offers more features." Playwright added the opt-in channel in 1.49. If you are being fingerprinted, channel="chromium" is the single highest-value line in your setup.

from playwright.sync_api import sync_playwright, expect

with sync_playwright() as p:
    # channel="chromium" selects new headless (real Chrome engine).
    # Without it, headless=True runs the old chrome-headless-shell binary.
    browser = p.chromium.launch(headless=True, channel="chromium")
    context = browser.new_context(
        locale="en-GB",
        timezone_id="Europe/London",
        viewport={"width": 1366, "height": 900},
    )
    page = context.new_page()

    # Drop what you will never parse. Images and fonts are most of the bytes.
    page.route("**/*.{png,jpg,jpeg,gif,webp,woff,woff2,svg}",
               lambda route: route.abort())

    page.goto("https://example.com/products", wait_until="domcontentloaded")

    cards = page.locator(".product-card")
    expect(cards.first).to_be_visible(timeout=20_000)   # wait for content
    expect(cards).not_to_have_count(0)                  # not just the container

    for row in cards.all():
        name = row.locator(".name").inner_text()
        price = row.locator(".price").inner_text()
        print(name, price)

    browser.close()

Waiting on an assertion rather than a clock is the whole point. Playwright's actionability checks already auto-wait for an element to resolve uniquely, be visible, be stable, receive events and be enabled before any interaction. Asserting on the content, not the container, catches the case that eats an afternoon: the grid element exists at 200 ms and stays empty until 2,400 ms.

Selenium remains a reasonable choice, especially where a team already runs a WebDriver estate. Version 4.47.0 shipped on 10 August 2026, and the official docs still show --headless=new for Chrome. Since Chrome 132 a bare --headless means the new mode anyway, so the flag is belt and braces rather than magic.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1366,900")
driver = webdriver.Chrome(options=options)

driver.get("https://example.com/products")
WebDriverWait(driver, 20).until(
    lambda d: len(d.find_elements(By.CSS_SELECTOR, ".product-card")) > 0
)
for card in driver.find_elements(By.CSS_SELECTOR, ".product-card"):
    print(card.text)
driver.quit()

The wait condition matters more than the flag. EC.presence_of_element_located, the recipe in every tutorial including the earlier version of this one, resolves the moment the node enters the DOM. On a hydrating SPA that is several hundred milliseconds before the node has any text in it. Counting children is cheap and correct.

Worth tracking: WebDriver BiDi, the standards-track answer to the DevTools Protocol, reached W3C Working Draft dated 29 June 2026. It is what lets Selenium subscribe to browser events instead of polling. Our Selenium WebDriver tips and the Java scraping guide cover the surrounding API.

What the page is actually checking

Three claims travel through every article on this subject, and all three are dead. We checked each against the specification rather than against other articles.

"Automation is detected by navigator.webdriver, so headless gets caught." The WebDriver specification defines the attribute as returning "true if webdriver-active flag is set, false otherwise," and defines that flag as "set to true when the user agent is under remote control. It is initially false." It reports remote control, not headlessness, and it is switched off by a command-line flag. Patchright (1.61.2, 5 July 2026) does exactly that, adding --disable-blink-features=AutomationControlled and removing --enable-automation.

"Headless browsers have no plugins, which gives them away." Not since the HTML Standard fixed the list. The spec defines a PDF viewer plugin objects list of exactly five hard-coded names: "PDF Viewer", "Chrome PDF Viewer", "Chromium PDF Viewer", "Microsoft Edge PDF Viewer" and "WebKit built-in PDF". The spec even explains why: "These names were chosen based on evidence of what websites historically search for." Every modern browser reports the same five. An empty navigator.plugins today means someone patched it badly, which is a different tell entirely.

"Headless Chrome is a distinct, detectable browser." It was, and the distinct part was moved out of the way. Since Chrome 132.0.6793.0 the old implementation ships only as the separate chrome-headless-shell binary, and --headless on the ordinary Chrome binary runs the same engine a human uses. The catch is the one from the last section: Playwright's default headless still downloads and runs that shell, so a scraper following the standard tutorial is running the detectable one while reading that it is not detectable.

What does get checked, in rough order of how much trouble it causes:

• Artefacts of the DevTools Protocol. Playwright, Puppeteer and Selenium's CDP paths leave traces in the JavaScript environment. Patchright's own documentation names the specific ones it works around: the Runtime.enable leak, which it dodges "by executing Javascript in (isolated) ExecutionContexts," and the Console.enable leak, which it kills by disabling the Console API outright. That is the honest cost of stealth in one sentence: your console.log stops working.
• Network-layer fingerprints. Covered above. This is why an impersonating HTTP client sometimes beats a browser.
• Coherence across layers. A residential exit in Poland, a browser reporting en-US, an America/Chicago timezone and 64 CPU cores is not a fingerprint, it is a confession. Setting locale and timezone_id on the context, as in the code above, costs nothing and fixes the most common mismatch.
• Behaviour. Mouse paths, scroll acceleration, dwell time. Hard to fake well, hard to detect cheaply, and the layer that is still holding.

The current tools, with what they actually are on 13 August 2026:

Beyond the fingerprint layer, the remaining levers are the familiar ones. Rotating residential proxies address the reputation half of the score, and a residential proxy pool priced from $2.50 per GB at Bright Data is the usual entry point. A CAPTCHA solving service clears the visible challenges when they appear, at published rates from 2Captcha on 13 August 2026 of $0.50 to $1 per 1,000 image captchas, $1 to $2.99 for reCAPTCHA v2 and $1.45 for Cloudflare Turnstile. Our analysis of a commercial anti-scrape product walks through what the defending side buys.

Debugging case: Splash renders it, the spider gets an empty page

A developer scrapes a grocery chain's category page, an SPA that loads products by JavaScript. They wire up Scrapy with Splash, point a browser at the Splash instance, and watch the page render correctly. The spider then receives a response with no product list and no search form. The render works when watched and fails when automated, which is the most demoralising failure mode there is. Three things are usually happening at once, and the fixes generalise to any headless setup.

A fixed wait is a race you lose at random. The original code passed args={'wait': 3}, meaning render then sleep three seconds. On an SPA that hydrates in stages, three seconds is enough on a warm cache and a fast morning, and not enough on a cold one. The scrape captures a half-built DOM and reports success. Wait for the content instead, as in the Playwright example above, and assert that the container has children rather than that it exists.

Splash stopped moving a long time ago. The Splash documentation still describes version 3.5, released 16 June 2020, over a footer reading "© Copyright 2019, Scrapinghub" and a rendering stack of Python 3, Twisted and Qt5 WebKit. The Docker image everybody actually runs tells the same story: the latest tag was last pushed on 19 August 2020 and master on 21 February 2021. The repository has no published releases at all and 373 open issues. Scrapy's current documentation on dynamic content does not mention Splash anywhere. A Qt WebKit engine frozen in 2020 will fail on any page using a browser API newer than that, which by now is most of them.

Give the nuance its due, because the internet flattens it: the Scrapy-side client scrapy-splash did ship 0.10.0 in January 2025 and is not abandoned. The renderer it talks to is the frozen part. A maintained client for a stalled server is still a stalled stack.

The modern replacement for a Scrapy project is scrapy-playwright. It drops a real Chromium into the spider with proper wait conditions, and it reached 0.0.48 on 10 July 2026. Read its limitations before adopting it: proxies are configured per browser or per context, never per request, which matters the moment you want rotation.

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_LAUNCH_OPTIONS = {"channel": "chromium"}   # new headless, not the shell
PLAYWRIGHT_MAX_CONTEXTS = 8
PLAYWRIGHT_ABORT_REQUEST = lambda req: req.resource_type in ("image", "font")

# spider.py
import scrapy
from scrapy_playwright.page import PageMethod

class ProductsSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/categories/112",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", ".product-cell .price"),
                ],
            },
        )

    def parse(self, response):
        for cell in response.css(".product-cell"):
            yield {
                "title": cell.css(".name::text").get(),
                "price": cell.css(".price::text").get(),
            }

There was probably an API the whole time. Storefront SPAs, grocery catalogues very much included, populate themselves from a JSON endpoint. Thirty seconds in the Network panel before any of this would have found a .../categories/112 call returning the same records, at a tenth of the cost and with no rendering race to lose.

What breaks at ten thousand pages

A script that works once and a crawl that runs nightly are different programs.

Memory is the first wall. By the Browserless sizing above, a 4-core, 8 GB box holds 10 to 20 concurrent pages. Create a fresh context per job and reuse the browser, not the other way round. One context shared across targets carries cookies and storage between them, which is how a scraper ends up recording one signed-in customer's personalised prices for every record.

The tail decides the schedule, not the median. A page that renders in 2 seconds at p50 and 45 at p99 spends most of a ten-thousand-page night inside 1% of its URLs. Set per-page timeouts aggressively, treat a timeout as retry-later rather than failure, and log the distribution. An average tells you nothing about whether you finish by morning.

Blocking resources is the cheapest speedup available. The page.route abort in the example above drops images, fonts and often a third to a half of the bytes. Billed through metered residential proxies, that is a line item rather than an abstraction.

Selectors rot faster than code. Anchor on text, ARIA roles and structural relationships rather than hashed class names, which build tools rotate on every deploy. Then add a canary: assert that a known-good page yields a plausible record count, and fail loudly at zero. A scraper that silently writes empty rows for six weeks costs more than one that crashes on night one.

Fixed concurrency plus a shared proxy pool is how you get banned on schedule. Rate is a per-target property. One global setting across forty domains means you are polite to thirty-nine and hammering one.

Where this stops working

Some doors do not open with the toolkit above, and recognising them early saves months.

Per-customer models. Commercial bot management trains site-specific classifiers, so solving one protected target teaches you almost nothing about the next. Effort does not amortise the way it does with parsing.

Signed request parameters. When the endpoint requires a value computed by obfuscated JavaScript that reads browser state, you either run that JavaScript in a real context or reverse-engineer a bundle that is re-minified weekly. Both are jobs, not tasks.

The buy-versus-build line. Managed unblockers price this exact decision. Bright Data's Web Unlocker was $1.50 per 1,000 requests pay-as-you-go on 13 August 2026, with a $499 monthly plan covering 383,000 requests and $1.30 per 1,000 beyond it. ZenRows listed $16 for 45,000 credits up to $456 for 5 million. Zyte's Advanced browser tier was $16.08 per 1,000. Compare those against a week of engineering time and the answer usually writes itself, which is the whole argument for handing the hard targets to a managed extraction service and keeping the easy ones in-house.

The legal boundary moved, and it moved specifically around this article's subject. Bypassing a technological measure that controls access is a different legal question from reading a public page, and it is being litigated now. Reddit sued Perplexity AI and several scraping intermediaries in the Southern District of New York in October 2025 under DMCA §1201(a), the anti-circumvention provision, rather than for copyright infringement. In a ruling reported at the start of August 2026, Judge Paul A. Engelmayer let the §1201(a) claims proceed against both Perplexity and SerpApi, holding that a third-party anti-bot system controlling access to search results can qualify as a technological measure, while dismissing the §1201(b) trafficking claim against SerpApi along with the unfair competition and unjust enrichment counts. Nothing here is legal advice and the case is at an early stage. The structural point stands regardless of outcome: "I only read what was on the page" is a weaker position once you have defeated something built to stop you.

Choosing your approach

The bottom line

The decision tree is short. Look in the HTML for serialised state, because it is there more often than anyone expects. Look in the Network panel for the endpoint the page is already calling. Try an impersonating HTTP client before you conclude that you need a browser. Render when the token you need is generated by code you cannot reproduce, and when you do render, launch the real engine and wait on content rather than on a clock.

The two mistakes that cost the most are both defaults. wait_until="networkidle" is marked DISCOURAGED in Playwright's own reference and still appears in most tutorials, including the earlier version of this one. And headless=True in Playwright runs the old chrome-headless-shell binary, not the Chrome that Google unified in version 132, so the scraper following the standard example is the detectable one. Both are one-line fixes. Neither is discoverable from a search result.

Skip the abandoned layer entirely. PhantomJS was archived in May 2023, dryscrape has not shipped since 2015, requests-html targets a Python that reached end of life in 2021, pyppeteer's own README says it is unmaintained, and Splash's Docker image was last pushed in August 2020. Any tutorial recommending them was written before the problem changed.

Past a certain level of protection the work stops being extraction and becomes maintenance of an evasion stack against a vendor with more engineers than you. That is where teams either accept a per-request price from an unblocker or take delivery of the records instead, which is what data as a service means in practice. The ceiling is real, and it is priced.