Scrape a JavaScript Website with Python: Full Guide

On 13 August 2026 we asked Python for the PyPI page of the requests package. The browser shows a long project page. Python got back 3,036 characters, HTTP status 200, and a <title> element reading Client Challenge. Inside it, a <noscript> block: "Please enable JavaScript to proceed." No exception. raise_for_status() passed without complaint. Ninety seconds later the same one-line script returned the real page, all 181,815 characters of it.

That is the whole problem. The HTML you get is not the HTML you see, nothing in the response says so, and the reason is not always the one everybody names.

Three reasons the HTML comes back short, and only one is JavaScript

Most guides on this topic, including the earlier version of this one, treat the short response as one condition with one cause. It is three conditions that look identical from inside your script.

The page really is rendered client-side. The server sends a small shell plus a script bundle. The browser runs the bundle, which fires XHR or fetch calls, receives JSON, and builds the DOM. An HTTP client downloads the shell and stops. This is the case everybody writes about, and it is real: in HTTP Archive's 2024 Web Almanac, the median page shipped 613 KB of JavaScript on desktop and 558 KB on mobile, with React present on 10% of pages, up from 8% the year before.

You were served a bot challenge with a 200 status. This is what happened above. An edge network decided your client was not a browser and sent an interstitial instead of the page. It carries a success status, so every status check in your code passes. The <noscript> text usually mentions JavaScript, which sends people off to install a headless browser when the problem was the request, not the rendering.

The bytes arrived intact and you decoded them wrong. Covered below, because it deserves its own section and because requests and httpx disagree about it today.

Diagnose which one you have before you choose a tool. Picking the wrong cause costs you a browser you did not need.

What happened to urllib2

Nothing you can use. urllib2 was Python 2 only. Python 2.7.18, released 20 April 2020, was the final Python 2 release, and its download page states flatly that "Python 2.7.18 reached end-of-life on 2020-01-01. It is no longer supported and does not receive security updates." In Python 3 the module was split apart: what was urllib2 now lives in urllib.request and urllib.error.

So the original comparison has no answer any more, because one of its two options no longer exists. Here is the question it turned into. Four clients are worth knowing, and their differences are real but nothing to do with JavaScript. Every version below was read from the package index on 13 August 2026.

On httpx and the twenty-month gap. The last stable httpx release is 0.28.1, dated 6 December 2024. The project is not dead and not archived: the repository is public, its changelog carries an [UNRELEASED] section at the top, and three pre-releases have shipped on the way to 1.0, the latest being 1.0.dev3 on 15 September 2025. Read that how you like. It means pip install httpx today installs software whose last stable build predates a lot of what your targets now run.

On curl_cffi. It exists for one reason: pure-Python clients cannot forge a browser's TLS handshake, and some sites check. HTTP/3 fingerprints landed in 0.15.0. It is the only one of the four that changes what your connection looks like below the HTTP layer. Whether that helps depends on your target, and the next section shows a case where it did not.

The difference that survives is not in your headers

We measured the default request each client sends, against a local server that recorded the headers verbatim. Same machine, same Python 3.11.15, 13 August 2026.

Two things in that table are worth more than the version numbers.

urllib.request asks for identity, so every page arrives uncompressed. On the JSON document measured below, that is 192,973 bytes instead of 43,299, a factor of 4.5. Across a hundred thousand pages you have downloaded fifteen extra gigabytes for nothing.

And curl_cffi in impersonation mode sends sec-ch-ua, Sec-Fetch-Dest, Sec-Fetch-Mode, Upgrade-Insecure-Requests and Priority. Those are what a browser sends and what your handwritten {"User-Agent": "Mozilla/5.0"} does not.

Headers are the part you can fix. The part you cannot fix is underneath. We captured the raw TLS ClientHello each client sends, by pointing them at a socket that recorded the handshake bytes, and parsed the extension list by hand. Three clients, one machine, one OpenSSL build, three fingerprints:

requests (urllib3)   extensions: 11 10 16 22 23 49 13 43 45 51 21
urllib.request       extensions: 11 10 35 16 22 23 49 13 43 45 51 21
httpx                extensions: 11 10 35 16 22 23 13 43 45 51 21

Extension 35 is session_ticket; urllib3 suppresses it, the standard library does not. Extension 49 is post_handshake_auth. Every one of those lists produces a different JA3 or JA4 hash. No amount of header spoofing changes any of it.

There is a smaller tell above the TLS layer. urllib.request normalises header names through str.capitalize(), so User-Agent goes out as User-agent and X-Custom-Thing as X-custom-thing. Header names are case-insensitive by specification. Their casing is still a signal, and no browser produces that pattern.

Where our knowledge stops. In one run against the PyPI project page, requests and httpx got the real page while urllib.request was challenged, with identical header values on all three. That looked like proof the TLS fingerprint decided it. Repeat the run and the result moves: sometimes everything is challenged, curl_cffi with full Chrome impersonation included, and after a ninety-second pause everything gets through. The edge is adaptive and its rules are not published. What we can state is what we measured. The clients are distinguishable at the network level whether you want them to be, and the fix that worked was waiting, not switching libraries.

Step 1: Work out which of the three you have

Three checks, in ascending order of effort.

Check the response for a challenge before anything else. A challenge page is short, arrives with status 200, and mentions JavaScript in a <noscript> block.

import requests

r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20)
print(r.status_code, len(r.text))
if len(r.text) < 20_000 and "noscript" in r.text.lower():
    print("likely an interstitial, not the page you asked for")
    print(r.text[:400])

Under twenty thousand characters for a page that should be long is the tell. So is Just a moment, Client Challenge, Checking your browser, or Attention Required in the title.

Check the encoding. If the text is there but reads as café where it should read café, no browser is required and the section below fixes it in one line.

Then compare View Source with Inspect. In the browser, View Page Source shows the raw HTML the server sent; Inspect shows the live DOM after scripts ran. Data present in Inspect and absent from View Source is client-side rendered. That is the only one of the three a headless browser solves.

Three checks, and they cover almost everything.

The encoding trap in requests

Set up a server that returns Content-Type: text/html with no charset parameter and a UTF-8 body. Then ask three clients for it. Measured against requests 2.34.2 and httpx 0.28.1 on 13 August 2026:

requests: r.encoding = 'ISO-8859-1'
requests: r.text     = '<p>café â\x80\x94 naïve</p>'
httpx:    encoding   = 'utf-8'
httpx:    text       = '<p>café — naïve</p>'
urllib:   get_content_charset() -> None

This is not a bug and it is documented, after a fashion. requests.utils.get_encoding_from_headers contains the line if "text" in content_type: return "ISO-8859-1", and the docstring on Response.text says the encoding is "determined based solely on HTTP headers, following RFC 2616 to the letter."

RFC 2616 is from 1999. It was replaced in 2014, and the current specification, RFC 9110 of June 2022, defines no default charset for text media types at all. So requests faithfully implements a rule withdrawn twelve years ago, on a web that has been overwhelmingly UTF-8 for longer than that.

httpx chose otherwise: its Response.encoding falls back to a default_encoding that is the literal string "utf-8". urllib.request declines to guess and hands you None, which is honest and leaves the work to you.

The fix is one line, and you should write it whether or not you have seen the symptom.

r = requests.get(url)
r.encoding = r.apparent_encoding   # charset_normalizer sniffs the bytes
text = r.text

# or skip the guessing entirely when you know the answer:
text = r.content.decode("utf-8", errors="replace")

apparent_encoding returned utf-8 for the mojibake case above, so one assignment repairs it. Note what breaks without it: nothing raises, nothing logs, and the damage is confined to non-ASCII characters. An English page passes every test you wrote. A German or Japanese page poisons your dataset silently, and you find out when somebody asks why the product names have question marks in them.

Step 2 (best): Find the hidden API

That JavaScript fetches your data from somewhere, and it is almost always a JSON endpoint. Call it directly and you skip rendering entirely.

This is not a shortcut we invented. Scrapy's documentation on dynamically loaded content puts it plainly: "the recommended approach is to find the data source and extract the data from it," and treats a headless browser as what you do after that fails.

How to find it:

1. Open DevTools (F12) and go to the Network tab.
2. Filter to Fetch/XHR.
3. Reload the page and interact with it: scroll, paginate, click.
4. Watch for responses that contain your data. Click one and read its Headers (URL, method, query parameters) and its Response.

Then reproduce it:

import requests

api_url = "https://example.com/api/v2/products"
params  = {"page": 1, "per_page": 50, "sort": "popular"}
headers = {
    "User-Agent": "Mozilla/5.0",
    "Accept": "application/json",
    # copy anything DevTools shows as required, e.g. Referer or a bearer token:
    # "Referer": "https://example.com/",
}

data = requests.get(api_url, params=params, headers=headers, timeout=20).json()
for item in data["products"]:
    print(item["name"], item["price"])

Three places to look that the Network tab will not show you. The data is often already in the HTML you downloaded, embedded by the framework so the page can hydrate without a round trip.

• Next.js Pages Router puts the entire page payload in <script id="__NEXT_DATA__" type="application/json">. One json.loads on that element's text and you have the whole record.
• Next.js App Router streams React Server Component data through repeated self.__next_f.push([1,"..."]) calls in inline scripts. Uglier, still parseable, and it is there in the first response.
• Structured data in <script type="application/ld+json"> covers products, articles, recipes and events across most commercial sites, and it is deliberately machine-readable.

GraphQL changes the shape of the work. If the Network tab shows POSTs to a single /graphql endpoint, you are not looking for many endpoints. You are looking for one plus a query document. Copy the request body verbatim, get it working, then trim fields.

For anything you found in DevTools, right-click the request and choose Copy as cURL, then paste it into curlconverter. It converts to 39 targets including Python requests and http.client, and it converts in your browser rather than on a server, which matters when the command holds a session cookie.

More on both techniques in scraping dynamic content and API scraping.

Step 3 (when you must render): headless browsers

Some data really is assembled in the browser with no clean endpoint behind it. Render it. First, what to stop reaching for, with the dates that make the case:

Playwright

Playwright 1.62.0 shipped on 31 July 2026. Reading browsers.json inside the installed package gives you exactly what it will download: Chromium 151.0.7922.34, Firefox 153.0, WebKit 26.5. One API across all three.

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # wait for the element you actually need, not for the network to go quiet
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    page.wait_for_selector(".product-card")

    names  = page.locator(".product-card .name").all_inner_texts()
    prices = page.locator(".product-card .price").all_inner_texts()
    for n, pr in zip(names, prices):
        print(n, pr)

    browser.close()

The hybrid is where the value is. Let Playwright load the page and log in, then hand its cookies to requests for everything after that. Better still, let the page fetch its own JSON and take it in flight:

with page.expect_response(lambda r: "/api/v2/products" in r.url) as caught:
    page.goto("https://example.com/products")
data = caught.value.json()

You have the structured payload without parsing a single tag, and you have learned the endpoint for next time.

Selenium

Selenium 4.47.0 shipped on 10 August 2026 across all five language bindings, with support for Chrome DevTools Protocol versions 149 through 151. Since 4.6 it manages drivers itself, so setup is a pip install.

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

opts = webdriver.ChromeOptions()
opts.add_argument("--headless")      # not --headless=new, see below
driver = webdriver.Chrome(options=opts)

driver.get("https://example.com/products")
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
)
for card in driver.find_elements(By.CSS_SELECTOR, ".product-card"):
    print(card.text)
driver.quit()

The flag changed and most tutorials still have the old one. Chrome unified its headless and headful modes in Chrome 112, and Chrome's documentation states that "since Chrome 132.0.6793.0 the old Headless mode is only available as a standalone binary named chrome-headless-shell." Plain --headless is now the flag. --headless=new survives as legacy syntax; --headless=old selects something no longer in the browser.

For waits, locators and anti-bot notes in depth, see Selenium web scraping.

Waiting: a correction to the previous version of this article

The earlier version of this guide used page.goto(url, wait_until="networkidle"). Playwright's own API documentation, in the version installed today, marks that value:

DISCOURAGED consider operation to be finished when there are no network connections for at least 500 ms. Don't use this method for testing, rely on web assertions to assess readiness instead.

The reasoning holds outside testing. networkidle waits for silence, and a page with analytics beacons, a chat widget or a polling socket never goes silent, so you either time out or wait far longer than you needed. Waiting for the element you are about to read is faster and deterministic. Use domcontentloaded for the navigation and wait_for_selector for the thing you want.

Never use a fixed time.sleep(). It is wrong in both directions at once: too short on a slow response, too long on every fast one.

What breaks at ten thousand pages

A script that runs once can be careless. A crawl that runs nightly cannot, and the numbers below are why.

Parsing costs more than people think. We timed three ways of pulling one field out of the same live PyPI project page, 181,815 characters of real HTML, against the same record delivered as 192,973 bytes of JSON. All three return the identical string, requests 2.34.2.

Measured on a shared two-core Intel Xeon at 2.80 GHz, Python 3.11.15, beautifulsoup4 4.15.0, lxml 6.1.1, on 13 August 2026. Your absolute numbers will differ. The ordering will not.

BeautifulSoup here is already using the lxml backend, which its own documentation recommends: "If you can, I recommend you install and use lxml for speed." The remaining eighteenfold gap is the cost of BeautifulSoup's object model, not of parsing. Across ten thousand pages that is ten minutes against thirty-three seconds. Across a hundred thousand it is an hour and a half spent building Tag objects you throw away.

Rendering costs money, and vendors publish the number. Zyte prices its API by target difficulty and by whether a browser is involved. At the "simple" tier, HTTP responses run $0.06 to $0.13 per 1,000 requests while browser-rendered responses run $0.48 to $1.01. At "advanced" the split is $0.61 to $1.27 against $7.68 to $16.08. Roughly eight times, top to bottom, for the same pages.

ScrapingBee prices the same choice in credits: 5 per rendered request, 10 or 25 for premium proxies without and with rendering, 75 for stealth. On the $49 Freelance plan of 250,000 credits that is 50,000 rendered pages, about $0.98 per 1,000. On stealth it is 3,333 pages, about $14.70 per 1,000, a fifteenfold jump for the same content.

Both read from the vendors' own pages on 13 August 2026. Both point the same way: a browser is the expensive path whether you rent it or run it.

Memory is the other tax. A Chromium instance holds hundreds of megabytes resident. Thirty will outgrow the machine running your scheduler, and a browser.close() that a raised exception skips leaks the whole process. Use a context manager, cap concurrency, and reuse one browser with many contexts.

The target may be rate-adaptive rather than fingerprint-strict. In our PyPI runs, hammering the host produced challenge pages for every client we tried, full Chrome impersonation included. A ninety-second pause produced the real page for the plainest client we had. Before you build a proxy pool, find out whether the site wants you to slow down.

One wasted second per page across a ten-thousand-page crawl is close to three hours.

Where each approach stops working

The hidden API stops working when the endpoint is signed. Some sites compute a request signature in JavaScript from a timestamp, a nonce and a key buried in the bundle. You can reverse it, and it will change. When the signing logic moves every release, driving the page is the cheaper maintenance choice even though it is slower.

Cookie-and-requests hybrids stop working when the site binds the session to a TLS or browser fingerprint rather than to the cookie. You log in with Playwright, hand the cookies to requests, and get logged straight out. Keep the whole session inside the browser.

Headless browsers stop working when the target checks for automation rather than for rendering. A stock Playwright or Selenium Chromium is detectable in a dozen ways, and the popular patches age badly, as the undetected-chromedriver date above shows. Sites that fight back at this level are where a managed extraction service earns its keep, because the cost is ongoing maintenance rather than a one-off script.

Everything stops working when the content sits behind an account you are not entitled to hold. No tool on this page solves an authorisation problem, and treating it as a technical one is how people end up in front of lawyers.

Choosing the right method

The order matters more than the rows. Try the endpoint first, render only when you have to, and treat a browser as a cost centre rather than a default.

Parsing what you get back

Once you have HTML, from requests or from a browser's page.content(), parse it. On the numbers above, lxml is the default for volume and BeautifulSoup is the default for a script you will read again in six months.

# convenient
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")
titles = [h.get_text(strip=True) for h in soup.select("h2.title")]

# fast
import lxml.html
doc = lxml.html.fromstring(html)
titles = [e.text_content().strip() for e in doc.cssselect("h2.title")]

For JSON from an endpoint you need no parser at all. Index into the dict and move on. A fuller comparison of parsing and request tooling lives in the best Python web scraping libraries.

Scrapy does not render JavaScript on its own. Scrapy 2.17.0 shipped on 7 July 2026 and pairs with scrapy-playwright, at 0.0.48 on 10 July 2026, when rendering is unavoidable. For endpoint-driven targets its scheduling and retry machinery is worth more than the rendering you are avoiding. See Scrapy web scraping.

Staying unblocked

Rendering makes you look more like a visitor. At volume you still need hygiene.

• Realistic headers and a persistent session. A Session keeps cookies and connections alive, which is faster and less conspicuous than a fresh handshake per page.
• Back off before you scale out. Our measurements show a site that answered a slow client and challenged a fast one. Concurrency is the first thing to reduce and the last thing to raise.
• Rotating proxies to spread requests across addresses, once you have established that the address is what the target is counting.
• CAPTCHA solving when challenges are unavoidable, though a challenge usually means the rate was wrong before it means the solver was missing.
• Read robots.txt properly. It became a standard in September 2022 as RFC 9309, worth knowing for two clauses. Crawlers are "requested to honor" the rules, and "these rules are not a form of access authorization." A file that permits you says nothing about what the terms of service allow. If robots.txt is unreachable, the specification says a crawler "MUST assume complete disallow", the opposite of what most homegrown crawlers do.
• Terms of service and privacy law come before technique. GDPR applies to personal data whether or not a page is public, and no header changes that.

Bottom line

Scraping a JavaScript website with Python was never about picking the right HTTP library, and in 2026 the original comparison has no second term left: urllib2 died with Python 2 in 2020. What replaced the question is more interesting. The clients differ in ways you can see, like urllib.request refusing to ask for compression and requests decoding text/* as ISO-8859-1 under a specification withdrawn in 2014. They also differ in ways you cannot patch from Python, like the TLS handshake each one emits.

So diagnose before you install anything. A short 200 response is a challenge, not a rendering problem. Mojibake is a decode problem, not a rendering problem. Only data that appears in Inspect and not in View Source needs a browser, and even then the payload is usually one expect_response call away in structured form. Reach for Playwright when the endpoint genuinely does not exist, wait for elements rather than for network silence, and price a browser per thousand pages before committing to ten thousand. Where that arithmetic does not work in your favour, extraction and delivery can be handed off as a managed pipeline and consumed as a clean feed.