JavaScript Rendering for Web Scraping: Libraries That Work

A modern, practical guide to JavaScript rendering for web scraping: when you actually need to render a page, which library or service to reach for in 2026 (Playwright, Puppeteer, Selenium, and hosted rendering APIs), and working code for each — plus the trick that lets you skip the browser entirely.

A decade ago, a scraper was simple: fetch the HTML, parse it, done. Then the web moved to React, Vue, Angular, and Svelte, and the HTML that arrives over the wire is often an empty shell — a <div id="root"></div> and a bundle of JavaScript that builds the real page in your browser. Point a plain HTTP client at one of those pages and you get nothing useful back. That is the problem JavaScript rendering for web scraping solves: you run the page's JavaScript the way a browser would, wait for the content to appear, and then extract it.

This guide covers the current libraries that do the rendering, when each one fits, and — just as important — how to tell whether you need a rendering step at all.

Do you actually need to render JavaScript?

Rendering is the most expensive way to scrape. A headless browser can use 50–300 MB of RAM per tab and is an order of magnitude slower than an HTTP request. Before you reach for one, check whether you can avoid it.

Open DevTools → Network → Fetch/XHR and reload the page. Most "JavaScript-heavy" sites don't paint their data with raw DOM writes; they fetch it from a JSON endpoint like /api/products?page=2 and render the response. If you can see that request, you can usually call it directly:

import httpx

# The internal API the page's JavaScript calls — no rendering needed
r = httpx.get("https://quotes.toscrape.com/api/quotes?page=1")
for q in r.json()["quotes"]:
    print(q["author"]["name"], "—", q["text"])

Hitting the internal JSON API is faster, lighter, and more stable than rendering. Reach for a rendering library only when the data is genuinely painted into the DOM by client-side code, is guarded behind interaction (infinite scroll, clicks, tabs), or the endpoint is signed in a way that's harder to reproduce than to just drive a browser.

A historical note: older tutorials on this topic pointed you at Splash, a scriptable rendering service from Scrapinghub that you ran in Docker and drove with Lua. Splash still runs, but it's effectively in maintenance mode and no longer the recommended path. The techniques below have replaced it.

The three ways to render JavaScript today

1. A headless browser library you run yourself — Playwright, Puppeteer, or Selenium. Maximum control, you host it.
2. A rendering plugin for your scraping framework — e.g. scrapy-playwright, which bolts a browser onto Scrapy so you keep your pipelines and just render the pages that need it.
3. A hosted rendering API — you POST a URL, the service runs a browser in its cloud and returns the rendered HTML, handling proxies and blocks for you.

Let's walk through each with real code.

Playwright — the modern default JavaScript rendering library

Playwright (from Microsoft) is the tool most new projects should start with. One API drives Chromium, Firefox, and WebKit; it auto-waits for elements before acting (which kills most flaky-timing bugs); and it has first-class bindings for Python, Node.js, Java, and .NET. For rendering-based scraping it's the best balance of power and ergonomics available in 2026.

Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    # The data is written by JavaScript; it exists only after rendering
    for quote in page.query_selector_all("div.quote"):
        text = quote.query_selector("span.text").inner_text()
        author = quote.query_selector("small.author").inner_text()
        print(f"{author}: {text}")

    browser.close()

Node.js:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'networkidle' });

const quotes = await page.$$eval('div.quote', els =>
  els.map(el => ({
    text: el.querySelector('span.text').textContent,
    author: el.querySelector('small.author').textContent,
  }))
);
console.log(quotes);
await browser.close();

Two habits pay off immediately:

• Wait for the thing, not for time. Prefer page.wait_for_selector("div.quote") over sleep(3). Playwright's expect/auto-wait makes scrapers far less brittle.
• Render once, parse fast. Grab the rendered HTML with page.content() and hand it to a fast parser like selectolax or BeautifulSoup if you prefer familiar selectors — you get JS rendering plus lightweight extraction.

Handling infinite scroll

Many feeds load more items over AJAX as you scroll. With a browser you just scroll and wait:

import time

prev = 0
while True:
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(1000)
    count = len(page.query_selector_all("div.quote"))
    if count == prev:          # nothing new loaded — reached the end
        break
    prev = count

But remember the first rule: if scrolling just fires ?page=N API calls, loop over those pages with an HTTP client instead — it's far cheaper.

Puppeteer — Chrome-first automation in Node

Puppeteer is Google's Node.js library for driving Chrome/Chromium over the DevTools Protocol. It predates Playwright, shares much of the same API design, and is still an excellent choice if you're in the Node ecosystem and only need Chromium.

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'networkidle2' });

const data = await page.evaluate(() =>
  [...document.querySelectorAll('div.quote')].map(el => ({
    text: el.querySelector('span.text').innerText,
    author: el.querySelector('small.author').innerText,
  }))
);
console.log(data);
await browser.close();

For scraping specifically, pair it with puppeteer-extra and its stealth plugin to reduce the fingerprints that give an automated Chrome away. Playwright vs. Puppeteer in one line: choose Playwright for multi-browser support and a cleaner cross-language API; choose Puppeteer if you're Node-only and want the most mature Chrome-specific ecosystem.

Selenium — the veteran, still everywhere

Selenium is the original browser-automation tool and still the most widely deployed, with bindings in nearly every language. Selenium 4 modernized it considerably: the W3C WebDriver protocol is standard, and Selenium Manager now downloads the right driver automatically, so the old "download chromedriver and match versions by hand" chore is gone.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)  # driver auto-managed in Selenium 4

driver.get("https://quotes.toscrape.com/js/")
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote"))
)

for quote in driver.find_elements(By.CSS_SELECTOR, "div.quote"):
    text = quote.find_element(By.CSS_SELECTOR, "span.text").text
    author = quote.find_element(By.CSS_SELECTOR, "small.author").text
    print(f"{author}: {text}")

driver.quit()

Selenium is a safe pick when your team already knows it, when you need a language binding Playwright doesn't cover as well, or when you're integrating with an existing Selenium Grid. For brand-new scraping projects, though, most developers now find Playwright faster to write and less flaky. There's a fuller walkthrough in our Selenium scraping guide.

scrapy-playwright — rendering inside Scrapy

If you already run Scrapy, you don't have to abandon your spiders, pipelines, and middleware to render JavaScript. The scrapy-playwright plugin is the modern successor to the old scrapy-splash setup: mark the requests that need a browser, and Scrapy routes just those through Playwright while everything else stays a normal fast HTTP request.

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        yield scrapy.Request(
            "https://quotes.toscrape.com/js/",
            meta={"playwright": True},   # render this one in a browser
        )

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

This "render only what you must" pattern keeps a large crawl fast: you pay the browser tax on the handful of pages that need it, not on the whole site.

Hosted rendering APIs — when you don't want to run browsers

Running a fleet of headless browsers at scale is real work: they crash, leak memory, need rotating proxies to avoid IP bans, and get flagged by anti-bot systems. A rendering API (also called a scraping API or web-unblocking API) offloads that: you send a URL plus options like "wait for this selector" and "use a residential proxy in the US," and the service returns fully rendered HTML.

import httpx

resp = httpx.get(
    "https://api.example-render.com/v1",
    params={
        "url": "https://quotes.toscrape.com/js/",
        "render_js": "true",
        "wait_for": "div.quote",
        "country": "us",
    },
    auth=("API_KEY", ""),
    timeout=60,
)
html = resp.text   # already rendered — hand it to your parser

The trade-off is straightforward: you swap infrastructure headaches and per-request cost for not maintaining browsers, proxy pools, or anti-bot evasion yourself. These services shine on heavily defended targets where the hard part isn't rendering but not getting blocked — sometimes in combination with CAPTCHA solving. For a small, well-behaved site, self-hosted Playwright is cheaper.

Which JavaScript rendering approach to choose

The decision tree in practice:

1. Can you hit an internal API? Do that.
2. Building fresh and self-hosting? Playwright.
3. Already on Scrapy? Add scrapy-playwright for the pages that need it.
4. Fighting aggressive blocking or scaling past what you want to operate? A hosted rendering API.

FAQ

Is a headless browser always required for JavaScript sites? No. If the content comes from a JSON/XHR endpoint — which is common — you can call that endpoint directly and skip the browser. Rendering is only mandatory when data is constructed in the DOM by client-side code or gated behind interaction.

Playwright or Selenium for scraping in 2026? For new work, Playwright is usually the better default: auto-waiting makes scrapers less flaky, and one API covers three browser engines. Selenium remains a strong choice when you already use it or need a specific language binding or Selenium Grid.

Why is my rendered page still missing data? Usually a timing issue. Wait for a specific selector or a network-idle state rather than a fixed delay, and confirm the content isn't loaded only after a scroll, click, or user action you haven't triggered yet.

How do I render JavaScript at scale without getting blocked? Combine a headless browser with rotating proxies and realistic browser fingerprints, or offload the whole problem to a hosted rendering API that manages proxies and unblocking for you.

Rendering a page in a browser is the easy part; keeping hundreds of them alive, un-blocked, and fed with fresh proxies as sites change is the part that eats engineering time. If you'd rather have the extracted data than operate the browser farm, scraping.pro runs JavaScript-heavy scraping as a managed service — rendering, proxies, and monitoring handled, delivered as a clean data feed.