Scraping JavaScript-Protected Content

You write a simple scraper, request the page, parse the HTML — and the data you can plainly see in your browser simply isn't there. The response is a near-empty shell of <div>s with no products, no prices, no text. You've hit JavaScript-protected content: a page whose real content is assembled by JavaScript after it loads, and sometimes deliberately withheld from anything that doesn't look like a real browser.

This is one of the most common walls in web scraping dynamic websites, and it's entirely beatable once you understand what's happening. This guide covers how to detect JS-driven pages, the two reliable ways through them, modern Python tooling for scraping dynamic content, and a real debugging case that trips up almost everyone the first time.

"JS-rendered" vs "JS-protected" — they're not the same

It's worth separating two things that look identical from the outside:

• JS-rendered content. The page is a single-page app (React, Vue, Angular, Svelte). The HTML skeleton arrives first, then JavaScript fetches the data — usually as JSON from a backend API — and paints it into the DOM. There's no intent to block you; the data just isn't in the initial HTML.
• JS-protected content. The site actively checks whether a real browser is present before it releases content. It runs JavaScript that measures browser fingerprints, timing, and environment quirks, and only serves the real HTML if those checks pass. This is anti-bot protection, not just rendering.

The good news is that the same toolkit handles both, and the diagnosis step is identical. The difference matters mainly for how hard you'll have to work: a plain SPA yields to a headless browser immediately, while a fingerprinting defense may need a stealth-hardened browser or a specialized unblocker.

Step 1 — Confirm the page is JavaScript-driven

Before changing your approach, prove the problem. Open the target in Chrome, press F12, and use two tabs:

1. Network → Fetch/XHR. Reload and watch the background requests. If the page fires off calls that return JSON full of the data you want, that's your fastest way in (see Step 2A).
2. View source vs. Elements. Right-click → View Page Source shows the raw HTML the server sent. If your target data is absent there but present in the Elements inspector (the live DOM), the content is being injected by JavaScript.

A quick programmatic tell: fetch the page with a plain HTTP client and search the response for a value you know is on the page.

import requests
html = requests.get("https://example.com/products").text
print("Dell Latitude" in html)   # False -> content is JS-rendered

If that prints False while your browser clearly shows the item, you're dealing with dynamic content.

Step 2A — The fast path: find the hidden API

Here's the counterintuitive truth about scraping dynamic content: the harder the page leans on JavaScript, the easier it often is to scrape — because all that JavaScript is usually just fetching clean JSON from an endpoint you can call directly. Rendering a whole browser to scrape a page that's quietly downloading a tidy JSON feed is doing it the slow way.

In the Network tab, find the XHR/fetch request whose response holds your data, then right-click → Copy → Copy as cURL. Import that into Postman or convert it to code. You get structured data with none of the parsing, and it's dramatically faster and lighter than driving a browser.

import requests

# The endpoint the page's JavaScript calls in the background
resp = requests.get(
    "https://example.com/api/v1/categories/112/products",
    headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
        "Accept": "application/json",
        "X-Requested-With": "XMLHttpRequest",
    },
)
data = resp.json()
for product in data["products"]:
    print(product["name"], product["price"])

The usual complication is that these endpoints expect proof you came from the real site — a CSRF token, session cookies, a Bearer token, or headers like Referer and X-Requested-With. The pattern is: load the normal page first, harvest the tokens and cookies, then attach them to the API call. We cover this in depth in scraping hidden APIs and, more broadly, scraping dynamic content. When the endpoint's parameters or signatures are generated by obfuscated JavaScript in the browser, this path gets hard — and that's your cue to render.

Step 2B — The universal path: render with a headless browser

When there's no reachable API, or the page's protection generates tokens you can't reproduce, you let a real browser do the work. A headless browser runs Chromium or Firefox without a visible window, executes all the JavaScript exactly as a user's browser would, and hands you the fully rendered DOM.

The modern default is Playwright; Selenium remains a solid, widely used choice. (If you're following an old tutorial that recommends dryscrape, PhantomJS, or the PyV8/v8js plugins — those are all abandoned; don't build on them.)

Playwright (the recommended choice in 2026)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products", wait_until="networkidle")

    # Wait for the actual content, not a fixed number of seconds
    page.wait_for_selector(".product-card")

    names = page.locator(".product-card .name").all_inner_texts()
    prices = page.locator(".product-card .price").all_inner_texts()
    for name, price in zip(names, prices):
        print(name, price)

    browser.close()

Playwright's key advantage for dynamic pages is wait_for_selector and wait_until="networkidle" — you wait for the thing you need to exist, not for an arbitrary timer. It also auto-waits on interactions, handles infinite scroll and "load more" buttons cleanly, and can block images and fonts to run faster.

Selenium (Python)

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://example.com/products")
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
)
for card in driver.find_elements(By.CSS_SELECTOR, ".product-card"):
    print(card.text)
driver.quit()

See our full guides to Playwright and Selenium web scraping for the deeper API. If you're scraping at scale, you'll want to pool browsers and reuse contexts rather than launching one per URL — headless browsers are memory-hungry.

Step 3 — When the page fights back (true anti-bot protection)

Genuinely JS-protected sites go further than rendering. They run challenge scripts that check for the tells of automation — the navigator.webdriver flag, headless-Chrome fingerprints, missing plugins, mismatched timezone/WebGL data — and block or serve a challenge page if anything looks off. Straight Playwright or Selenium will get caught. The countermeasures, in rough order of effort:

• Stealth-hardened browsers. Tools like undetected-chromedriver and Playwright stealth patches remove the obvious automation fingerprints. Start here.
• Rotating residential proxies. Many defenses are IP-reputation based; a good residential proxy pool plus user-agent rotation gets you far.
• CAPTCHA handling. When a challenge appears, a CAPTCHA solving service can clear it. See our overview of anti-scraping protection for the full landscape.
• A managed unblocker or service. Past a certain point the cat-and-mouse isn't worth your engineering time — which is where a done-for-you pipeline earns its keep.

Debugging case: "Splash renders it in the browser, but my spider gets an empty page"

Here's a real, representative problem. A developer tries to scrape a grocery store's category page — an SPA that loads its products via JavaScript. They set up Scrapy with Splash, point a browser at the Splash instance, and see the full page render correctly. But when the spider runs, the response object comes back missing the product list and the search form. The rendered HTML the spider receives is incomplete.

This confuses everyone because the render clearly works when viewed manually. Three things are almost always going on, and the fixes generalize to any headless setup:

1. A fixed wait is a race condition. The original code used args={'wait': 3} — "render, then wait 3 seconds." On a slow SPA that hydrates in stages, three seconds isn't always enough, and the scrape captures a half-built DOM. The fix is to wait for a specific element to appear rather than for a clock:

# Instead of a blind wait, wait for the product grid to exist.
# In Playwright:
page.goto(url, wait_until="networkidle")
page.wait_for_selector(".product-cell", timeout=20000)

2. Splash itself is legacy. Splash is no longer the recommended tool — Zyte, its maintainer, moved on and it now sees minimal maintenance. The modern equivalent for a Scrapy project is scrapy-playwright, which drops a real Chromium into your spider with proper wait conditions:

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# spider.py
import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/categories/112",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    {"method": "wait_for_selector", "args": [".product-cell"]},
                ],
            },
        )

    def parse(self, response):
        for cell in response.css(".product-cell"):
            yield {
                "title": cell.css(".name::text").get(),
                "price": cell.css(".price::text").get(),
            }

3. There was probably an API all along. Most storefront SPAs — grocery catalogs very much included — populate themselves from a JSON endpoint. Before rendering anything, the developer should have checked the Network tab for a .../products or .../categories/112 call returning JSON. Nine times out of ten, hitting that endpoint directly (Step 2A) is faster and more robust than any amount of browser rendering, and it sidesteps the rendering race entirely.

The lesson: when a headless render returns incomplete HTML, don't add more sleep time — wait for a concrete selector, move off deprecated tooling, and check whether a clean API makes the whole browser unnecessary.

Choosing your approach

The bottom line

Scraping JavaScript-protected content comes down to a simple decision tree: look for the hidden API first because it's faster and cleaner, render with a modern headless browser when there isn't one, and add stealth and proxies when the site actively fights back. Skip the abandoned tools from a decade of old tutorials — Playwright and scrapy-playwright are what actually work now.

When the protection is aggressive and keeping scrapers alive turns into a full-time cat-and-mouse game, that's usually the point to hand it off. Our web scraping service runs the headless-browser, proxy, and anti-bot stack as a managed pipeline and delivers the clean data — so JavaScript defenses become our problem, not yours.