Scraping Dynamic Content: Two Approaches That Work

Modern websites rarely hand you a finished HTML page with the data already in it. Product catalogs, feeds, prices, reviews — these are usually loaded after the page renders, via JavaScript. For a scraper, that means a plain GET of the page and a parse of the HTML returns an empty shell with no data inside.

There are two fundamentally different ways to solve this, and scraping dynamic content almost always comes down to choosing between them:

1. Intercepting API requests — find the exact requests the browser itself uses to fetch the data, and replay them directly, without a browser.
2. Full browser emulation — spin up a real (or headless) browser, let it render the JavaScript, and interact with the page like a user: click, scroll, drag elements.

The first is faster and far lighter on resources; the second is more universal and more resilient to unusual logic. In practice, teams often combine them.

Approach 1. Intercepting and replaying API requests

The idea

When a page "fills in" its data, it almost always makes background HTTP requests (XHR/fetch) to a private backend API that returns JSON. If you find that endpoint and reproduce the request with the right headers, cookies, and tokens, you can pull the data directly — skipping rendering entirely. That's tens of times faster and needs no browser at all.

How to find the API

1. Open DevTools (F12) → the Network tab.
2. Filter by Fetch/XHR.
3. Scroll the page, click "Show more," switch categories — in short, trigger a data load.
4. Find the request whose response contains the JSON you want (products, prices, and so on).
5. Study it: URL, method, query parameters, headers, body, cookies.
6. Right-click → Copy → Copy as cURL — an excellent starting point: import it into Postman, or convert it straight to code.

Tokens: CSRF, sessions, authorization

The main difficulty here is that the request is almost never "bare." The server expects a set of proof values, and without them it returns 401, 403, or 419.

CSRF token (Cross-Site Request Forgery). Protection against forged cross-site requests. The server issues a random token that the client must return on modifying (and sometimes even reading) requests. Where it usually lives:

• in <meta name="csrf-token" content="..."> in the page HTML;
• in a hidden form field <input type="hidden" name="_token" value="...">;
• in a cookie (often XSRF-TOKEN) that you then need to echo back in an X-CSRF-Token or X-XSRF-TOKEN header.

The workflow: first load the normal page, extract the token and session cookies, then attach them to the API request.

Session cookies. After your first visit the server sets Set-Cookie (for example, sessionid, PHPSESSID, laravel_session). You need to persist these between requests — that's what a session object (requests.Session, httpx.Client) does automatically.

Authorization (Bearer / JWT / API key). If the data sits behind a login, the header is usually Authorization: Bearer <token>. JWT tokens are obtained through the login endpoint, then attached to every subsequent request.

Other protective fields. X-Requested-With: XMLHttpRequest (often mandatory for AJAX endpoints), Referer, Origin, and sometimes signed parameters (signature, nonce, timestamp) generated by the front-end JavaScript.

When this approach breaks down. If a token or request signature is generated by obfuscated JavaScript right in the browser (or in WASM), reproducing it server-side is extremely hard. That's your cue to switch to the second approach — browser emulation — where the JavaScript runs itself.

Example: Python + requests (with CSRF and pagination)

import re
import requests

session = requests.Session()
BASE = "https://example-shop.com"

# 1. Load the page to get the CSRF token and session cookies
resp = session.get(f"{BASE}/catalog", headers={
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
})

# The CSRF token may live in a meta tag...
m = re.search(r'name="csrf-token"\s+content="([^"]+)"', resp.text)
csrf = m.group(1) if m else session.cookies.get("XSRF-TOKEN")

headers = {
    "X-CSRF-Token": csrf,
    "X-Requested-With": "XMLHttpRequest",
    "Referer": f"{BASE}/catalog",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept": "application/json",
}

# 2. Hit the internal API page by page
all_items = []
page = 1
while True:
    r = session.get(
        f"{BASE}/api/products",
        params={"category": "phones", "page": page, "per_page": 48},
        headers=headers,
    )
    r.raise_for_status()
    payload = r.json()
    items = payload.get("items", [])
    if not items:
        break
    all_items.extend(items)
    page += 1

for it in all_items:
    print(it["title"], it["price"])

Example: Python + httpx (async, faster at scale)

import asyncio
import httpx

async def fetch_page(client, page):
    r = await client.get("/api/products", params={"page": page, "per_page": 48})
    return r.json().get("items", [])

async def main():
    async with httpx.AsyncClient(base_url="https://example-shop.com",
                                 headers={"X-Requested-With": "XMLHttpRequest"}) as client:
        tasks = [fetch_page(client, p) for p in range(1, 11)]
        results = await asyncio.gather(*tasks)
    items = [x for chunk in results for x in chunk]
    print(len(items))

asyncio.run(main())

Example: Node.js + fetch

const csrf = "..."; // extracted from HTML/cookies beforehand

const res = await fetch("https://example-shop.com/api/products?page=1&per_page=48", {
  headers: {
    "X-CSRF-Token": csrf,
    "X-Requested-With": "XMLHttpRequest",
    "Accept": "application/json",
    "Cookie": "sessionid=abc123; XSRF-TOKEN=" + csrf,
  },
});

const data = await res.json();
data.items.forEach(item => console.log(item.title, item.price));

Approach 2. Full browser emulation

The idea

You launch a real engine (Chromium, Firefox, WebKit); it downloads the page, executes all the JavaScript, and builds the DOM. From there you work with the page exactly like a human: wait for elements to appear, click, scroll, drag sliders. All the tokens, signatures, and anti-bot scripts run on their own — you don't have to reproduce any of them.

The downsides: an order of magnitude slower, hungry for CPU/RAM, and easier for anti-bot systems to detect (though there are dedicated "stealth" modes to fight that).

What to emulate with

• Selenium — the oldest standard, supporting Python, Java, C#, JavaScript, and Ruby. It drives real browsers through WebDriver.
• Playwright — a modern framework from Microsoft. Python, JavaScript/TS, .NET, Java. Chromium, Firefox, and WebKit out of the box, smart auto-waiting for elements, and convenient network interception.
• Puppeteer — Node.js, originally Chromium-only (there's an experimental Firefox). Very fast and mature for Chrome.

Actions on the page

Below are the same four actions (click, scroll, scroll to element, hold mouse button + move) across different stacks. Hold + move is the basis for drag-and-drop, range sliders, and slider CAPTCHAs.

Playwright (Python)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example-shop.com/catalog")

    # CLICK
    page.click("button.load-more")

    # SCROLL with the wheel
    page.mouse.wheel(0, 2000)

    # SCROLL to a specific element
    page.locator("footer").scroll_into_view_if_needed()

    # HOLD button + MOVE (drag / slider)
    box = page.locator(".slider-handle").bounding_box()
    start_x = box["x"] + box["width"] / 2
    start_y = box["y"] + box["height"] / 2
    page.mouse.move(start_x, start_y)
    page.mouse.down()                                   # press and hold
    page.mouse.move(start_x + 200, start_y, steps=25)   # drag smoothly (25 steps)
    page.mouse.up()                                     # release

    browser.close()

Playwright (JavaScript/Node)

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://example-shop.com/catalog');

  // CLICK
  await page.click('button.load-more');

  // SCROLL
  await page.mouse.wheel(0, 2000);

  // SCROLL to element
  await page.locator('footer').scrollIntoViewIfNeeded();

  // HOLD + MOVE
  const box = await page.locator('.slider-handle').boundingBox();
  await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2);
  await page.mouse.down();
  await page.mouse.move(box.x + 200, box.y, { steps: 25 });
  await page.mouse.up();

  await browser.close();
})();

Selenium (Python) — via ActionChains

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example-shop.com/catalog")
wait = WebDriverWait(driver, 10)

# CLICK (waiting until clickable)
btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more")))
btn.click()

# SCROLL
driver.execute_script("window.scrollBy(0, 2000)")

# SCROLL to element
footer = driver.find_element(By.CSS_SELECTOR, "footer")
driver.execute_script("arguments[0].scrollIntoView({block:'center'})", footer)

# HOLD + MOVE
handle = driver.find_element(By.CSS_SELECTOR, ".slider-handle")
(ActionChains(driver)
    .click_and_hold(handle)     # press and hold
    .move_by_offset(200, 0)     # move 200px to the right
    .pause(0.3)
    .release()                  # release
    .perform())

driver.quit()

Selenium (Java)

WebDriver driver = new ChromeDriver();
driver.get("https://example-shop.com/catalog");

// CLICK
driver.findElement(By.cssSelector("button.load-more")).click();

// SCROLL
((JavascriptExecutor) driver).executeScript("window.scrollBy(0, 2000)");

// HOLD + MOVE
WebElement handle = driver.findElement(By.cssSelector(".slider-handle"));
new Actions(driver)
    .clickAndHold(handle)
    .moveByOffset(200, 0)
    .release()
    .perform();

Puppeteer (Node.js)

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();
  await page.goto('https://example-shop.com/catalog');

  // CLICK
  await page.click('button.load-more');

  // SCROLL
  await page.evaluate(() => window.scrollBy(0, 2000));

  // SCROLL to element
  await page.$eval('footer', el => el.scrollIntoView());

  // HOLD + MOVE
  const handle = await page.$('.slider-handle');
  const box = await handle.boundingBox();
  await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2);
  await page.mouse.down();
  await page.mouse.move(box.x + 200, box.y, { steps: 25 });
  await page.mouse.up();

  await browser.close();
})();

Stealth modes: evading automation detection

A headless browser launched out of the box is trivially distinguishable from a real one. Anti-bot systems (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai, and others) check dozens of signals, and if even a few of them scream "automation," you get a CAPTCHA, a challenge, or an outright block. Stealth mode is a set of patches and tricks that mask those tells.

The signals they catch you by

• navigator.webdriver === true — the most obvious flag, which a browser under WebDriver/CDP control sets automatically.
• Headless artifacts. A missing window.chrome, an empty plugin list (navigator.plugins), unusual navigator.languages, a WebGL renderer like SwiftShader/Google Inc. instead of a real GPU.
• Fingerprinting. Canvas, WebGL, AudioContext, and the installed font set produce a stable "fingerprint" of the environment; on default headless it looks suspiciously generic.
• TLS/JA3 fingerprint. At the HTTP-connection level, the handshake of a Python client or Node differs from Chrome's — and that's caught before any JavaScript even runs (relevant for Approach 1 too).
• Behavior. Instant clicks with no mouse movement, perfectly even timing, jumping straight to an inner page with no navigation — all of it reads as inhuman.
• IP reputation. Data-center ranges (AWS, Hetzner, and so on) are flagged; residential and mobile addresses draw far less suspicion. This is where rotating proxies earn their keep.

Ready-made tools

• puppeteer-extra + puppeteer-extra-plugin-stealth (Node) — the best-known bundle; hides navigator.webdriver, fixes WebGL/plugins/languages and dozens of other "leaks."
• playwright-extra with the same stealth plugin — the Playwright-on-Node equivalent.
• undetected-chromedriver (Python, on top of Selenium) — a patched ChromeDriver that passes many Cloudflare checks. Its successor is nodriver (no WebDriver protocol at all, pure CDP).
• SeleniumBase in UC mode (--uc) — a Selenium wrapper with built-in anti-detection.
• rebrowser-patches — low-level runtime patches for Puppeteer/Playwright that close subtler CDP leaks.

Important: no stealth plugin is a guarantee. Anti-bot systems update constantly, and what slipped through yesterday may be caught tomorrow. This is an arms race, not a one-time setting.

Examples

Python — undetected-chromedriver:

import undetected_chromedriver as uc

options = uc.ChromeOptions()
options.add_argument("--lang=en-US")
# for anti-detection you usually do NOT run headless, or you use the new mode:
# options.add_argument("--headless=new")

driver = uc.Chrome(options=options)
driver.get("https://example-shop.com/catalog")
print(driver.title)
driver.quit()

Node — puppeteer-extra + stealth:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

(async () => {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();
  // a believable UA and matching headers
  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
  );
  await page.goto('https://example-shop.com/catalog');
  await browser.close();
})();

Node — playwright-extra + stealth:

const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();

chromium.use(stealth);

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://example-shop.com/catalog');
  await browser.close();
})();

Manual tricks (on top of, or instead of, plugins)

• Hide webdriver and fix the environment. Via CDP / an init script, before the page loads:

# Playwright (Python): runs in every new document BEFORE the site's own scripts
page.add_init_script(
    "Object.defineProperty(navigator, 'webdriver', {get: () => undefined});"
)

• New headless mode. In recent Chrome, the --headless=new flag is closer to a normal browser than old headless; sometimes it's better to run non-headless under a virtual display (Xvfb) instead.
• Persistent profile. Launching with user_data_dir keeps cookies and a "warmed-up" session between runs — less like a freshly minted bot.
• Human-like behavior. Random pauses, mouse movement along a curve (Bézier), scrolling in jerks rather than one leap. Libraries like pyautogui/bezier help generate trajectories, or use the built-in steps argument on mouse.move.
• Proxies. Residential and mobile proxies with rotation noticeably cut the share of challenges compared with data-center IPs.

Stealth for Approach 1 (no browser)

API interception is detectable too — by the HTTP client's TLS fingerprint. To make the request look like real Chrome at the handshake level, use clients that spoof the TLS fingerprint:

# curl_cffi can mimic the TLS/JA3 fingerprint of a specific browser
from curl_cffi import requests

r = requests.get(
    "https://example-shop.com/api/products?page=1",
    impersonate="chrome124",   # spoof the Chrome 124 handshake
)
print(r.json())

Alternatives: tls-client (Python/Go), curl-impersonate (a system binary). This often fixes the case where plain requests gets a 403 while the same page opens fine in a browser. If you keep hitting challenge pages, pairing this with CAPTCHA solving and rotating proxies is usually the difference between a stable scraper and a blocked one.

Which libraries do what

The key dividing line: does the tool execute JavaScript and can it emulate mouse actions? HTTP clients and HTML parsers do neither — they're only suitable for static pages or for Approach 1 (API replay).

The short version:

• Just need to replay an API request → requests/httpx (Python), fetch/got (Node), Colly (Go).
• Parse HTML you already have → BeautifulSoup/lxml, cheerio, jsoup.
• Need JS rendering and mouse actions → Playwright, Selenium, Puppeteer, chromedp/rod.
• HtmlUnit handles a little JavaScript, but it stumbles on modern SPAs — for serious rendering, reach for Playwright/Selenium.

Case study: monitoring online stores and infinite scroll

This is arguably the single most common real-world task. In catalogs, products usually aren't all rendered at once: sites use lazy loading / infinite scroll — new cards load as you scroll (or when you click "Show more"). A plain HTML request returns only the first "batch." It's the classic challenge in web scraping dynamic websites like ecommerce catalogs.

There are two paths, both widely used in price and assortment monitoring:

Path A (preferred): intercept the pagination API

While scrolling, the store almost always fires something like /api/catalog?page=2&offset=48. If so — forget the browser and collect data page by page directly (see Approach 1). It's fast, stable, and scales to thousands of products. That's how most industrial monitoring is built: the browser is used once — to reverse-engineer the API structure and tokens — while the actual collection runs through an HTTP client.

Path B: render + scroll, when the API is locked down

If the endpoint is protected by a nontrivial signature, or the data is generated purely on the client, you're left scrolling with a browser and collecting cards from the DOM. The algorithm: scroll down → wait for the load → count the cards → repeat until the count stops growing.

from playwright.sync_api import sync_playwright

def scrape_catalog(url):
    products = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url)

        prev_count = -1
        stable_rounds = 0

        while stable_rounds < 2:  # 2 empty scrolls in a row = reached the end
            # scroll to the very bottom
            page.mouse.wheel(0, 4000)
            page.wait_for_timeout(1500)  # give it time to load

            cards = page.locator(".product-card")
            count = cards.count()

            if count == prev_count:
                stable_rounds += 1
            else:
                stable_rounds = 0
            prev_count = count

        # collect all cards after fully scrolling
        cards = page.locator(".product-card")
        for i in range(cards.count()):
            card = cards.nth(i)
            products.append({
                "title": card.locator(".title").inner_text(),
                "price": card.locator(".price").inner_text(),
                "url":   card.locator("a").get_attribute("href"),
            })

        browser.close()
    return products

data = scrape_catalog("https://example-shop.com/catalog/phones")
print(f"Products collected: {len(data)}")

The same trick for a "Show more" button — click while the button still exists:

while page.locator("button.load-more").count() > 0:
    page.click("button.load-more")
    page.wait_for_timeout(1200)

Hybrid: the best of both worlds

Playwright and Puppeteer can listen to network traffic. You can open the page in a browser (so all the tokens and anti-bot checks pass), but pull data not from the DOM — from the responses of that very API the browser calls as you scroll:

def handle_response(response):
    if "/api/products" in response.url:
        data = response.json()
        # save the ready JSON — no need to parse HTML
        save(data["items"])

page.on("response", handle_response)
page.goto("https://example-shop.com/catalog")
# then just scroll — the data flows into the handler on its own

This is often the optimal choice for monitoring: the resilience of browser emulation plus clean structured JSON instead of brittle markup parsing.

Practical monitoring tips

• Wait for data, not for time. Instead of "sleep 1.5 seconds," wait for an element to appear (wait_for_selector) or for network quiet (wait_for_load_state("networkidle")) — more reliable and often faster.
• Deduplicate. With infinite scroll, some cards may be read twice — collect by a unique id/url.
• Rate-limit yourself. Overly aggressive requests overload the site and get you blocked fast. Add delays, use proxy pools carefully, and stay within the law.
• Cache your recon. Work out the API structure and tokens once; in production, run a lightweight HTTP collector and keep the heavy browser as a fallback.

How to choose an approach

The practical rule: always check first whether you can get by with API interception — it's faster, cheaper, and more stable. Move to browser emulation only when the API is hidden behind client-side cryptography, protected by complex anti-bot logic, or when you need to reproduce nontrivial interaction (drag-and-drop, sliders, multi-step forms).

If you'd rather not build and maintain this yourself — the DevTools recon, the token handling, the proxy rotation, the anti-bot arms race — scraping.pro runs it as a done-for-you data extraction service, delivering the structured data (or a live data feed) without you owning the infrastructure.