Scrape a JavaScript Website with Python: Full Guide

You write a few lines of Python, call requests.get(), and the HTML that comes back is missing the very data you can plainly see in your browser. The product list is empty, the reviews are gone, the table has no rows. This is the single most common wall people hit when they try to scrape a JavaScript website with Python — and once you understand why it happens, the fix is usually simple and often does not require a browser at all.

This guide walks through the diagnosis and the three ways to solve it, from fastest to heaviest.

Why requests returns an incomplete page

Start with a classic symptom. Years ago people noticed that requests and the old Python 2 urllib2 could return different-looking results for the same URL:

import requests
resp = requests.get("https://en.wikipedia.org/wiki/Talk:Land_value_tax")
html = resp.text  # looks partial compared to what the browser shows

The confusion back then was blamed on the library. It was not the library. Both requests and urllib do exactly one thing: they perform an HTTP request and hand you the raw response body the server sent. Neither one executes JavaScript.

Historical note: urllib2 was Python 2 only. Python 2 reached end of life in January 2020 — do not use it. In modern Python the equivalents are the third-party requests (or httpx) and the standard library's urllib.request. Any difference you see between them today comes down to default headers, redirects, or compression, not JavaScript.

Here is the real mechanism. A modern page arrives as a small HTML shell plus JavaScript. The browser then runs that JavaScript, which fires more network calls (XHR/fetch), receives JSON, and builds the visible DOM. An HTTP client is not a browser: it downloads the shell and stops. The JavaScript never runs, the follow-up calls never fire, and the content you wanted never materializes.

So the question is never "which HTTP library?" It is: "where does the data actually come from, and how do I get it without a full browser if I can?"

Step 1: Confirm the content is JavaScript-rendered

Two quick checks tell you what you are dealing with:

1. View source vs. Inspect. In the browser, use View Page Source (the raw HTML the server sent) and compare it to Inspect (the live DOM after JavaScript). If your data is in Inspect but not in View Source, it is rendered client-side.
2. Print what Python actually got.

import requests
html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text
print("target text present:", "some known value" in html)

If the value is missing from html, JavaScript is building it. Now pick the right tool.

Step 2 (best): Find the hidden API

Here is the part most tutorials skip. That JavaScript is fetching your data from somewhere — almost always a JSON endpoint. If you call that endpoint directly, you skip rendering entirely and get clean, structured data that is faster to parse and far less likely to break. This is the professional default; reach for a browser only when this fails.

How to find it:

1. Open DevTools (F12) and go to the Network tab.
2. Filter to Fetch/XHR.
3. Reload the page and interact with it (scroll, paginate, click).
4. Watch for requests that return JSON containing your data. Click one and read its Headers (URL, method, query params) and Response.

Then reproduce that request in Python:

import requests

# The endpoint you found in the Network tab:
api_url = "https://example.com/api/v2/products"
params = {"page": 1, "per_page": 50, "sort": "popular"}
headers = {
    "User-Agent": "Mozilla/5.0",
    "Accept": "application/json",
    # copy any required headers you saw in DevTools, e.g. Referer or an API key:
    # "Referer": "https://example.com/",
}

data = requests.get(api_url, params=params, headers=headers, timeout=20).json()
for item in data["products"]:
    print(item["name"], item["price"])

Advantages of the hidden-API approach:

• Fast and cheap — no browser, low memory, easy to parallelize.
• Structured — you get JSON, not HTML you have to scrape.
• Pagination is obvious — usually a page or offset parameter you can loop.

Watch out for auth: some endpoints need a token, a Referer, a cookie, or a CSRF header. Copy whatever DevTools shows as cURL (right-click the request → Copy as cURL) and translate it — the tool curlconverter turns a cURL command straight into Python requests code. For more on this technique, see scraping dynamic content and API scraping.

Step 3 (when you must render): headless browsers

If there is no clean endpoint — the data is assembled in the browser, protected, or hidden behind heavy client-side logic — render the page with a real browser engine. Note that some of the tools from older Python tutorials are dead: dryscrape and PyV8 are unmaintained, and requests-html is effectively abandoned. In 2026 the live options are:

Playwright (recommended)

Playwright is the modern default: fast, reliable auto-waiting, and one API across Chromium, Firefox, and WebKit.

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/products", wait_until="networkidle")

    # wait for the real content, not a fixed sleep:
    page.wait_for_selector(".product-card")

    names = page.locator(".product-card .name").all_inner_texts()
    prices = page.locator(".product-card .price").all_inner_texts()
    for n, pr in zip(names, prices):
        print(n, pr)

    browser.close()

A powerful hybrid: let Playwright load the page and log in, then read its cookies and switch to requests for the fast endpoints — or intercept the JSON responses the page itself fetches with page.on("response", ...).

Selenium

The long-standing option, with bindings in many languages and a huge community. Since Selenium 4.6 it auto-manages drivers, so setup is minimal.

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)

driver.get("https://example.com/products")
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-card"))
)
for card in driver.find_elements(By.CSS_SELECTOR, ".product-card"):
    print(card.text)
driver.quit()

For the full treatment — waits, locators, headless mode, and anti-bot notes — see Selenium web scraping.

A note on waiting

Never scrape immediately after goto(). Wait for the actual content to appear (wait_for_selector, WebDriverWait), not a fixed time.sleep(). Explicit waits are both faster and far more reliable, because they finish the instant the element exists instead of guessing a duration.

Choosing the right method

The order matters. Try the endpoint first, render only when you have to. A hidden-API scraper can be 10–50x faster than driving a browser and breaks far less often.

Parsing what you get back

Once you have HTML — from requests or from a browser's page.content() — parse it with BeautifulSoup or lxml:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")
titles = [h.get_text(strip=True) for h in soup.select("h2.title")]

For JSON from a hidden API you rarely need a parser at all — index into the dict. For a fuller comparison of parsing and request tooling, see the best Python web scraping libraries.

Staying unblocked

Rendering JavaScript makes you look more like a real user, but at scale you still need hygiene:

• Realistic headers, especially User-Agent, and a consistent session.
• Rotating proxies to spread requests across IPs.
• Rate limiting and delays — be gentle; concurrency plus a JS site is an easy way to get throttled.
• CAPTCHA solving if you hit challenges — though a CAPTCHA usually means you should slow down first.
• Respect robots.txt, Terms of Service, and privacy law (GDPR, CCPA/CPRA) before collecting anything personal.

FAQ

Can requests run JavaScript? No. requests, httpx, and urllib only perform HTTP requests. To execute JavaScript you need a browser engine (Playwright, Selenium) — or you avoid the problem by calling the site's underlying JSON API.

Playwright or Selenium in 2026? Playwright for new projects: faster, better auto-waiting, cleaner API, network interception built in. Selenium is still excellent and worth it if your team already uses it or you need its specific ecosystem.

Is Scrapy able to handle JavaScript? Scrapy by itself does not render JavaScript, but it pairs with a headless browser via scrapy-playwright. For static or hidden-API sites, plain Scrapy is a great fit — see Scrapy web scraping.

How do I find the hidden API? DevTools → Network → filter Fetch/XHR → reload and interact → look for JSON responses containing your data → copy the request as cURL and replay it in Python.

Bottom line

Scraping a JavaScript website with Python is not about picking a magic HTTP library — requests and urllib both stop before the JavaScript runs. Diagnose first: if the data lives in a JSON endpoint, call it directly for a fast, sturdy scraper. Only when there is genuinely no endpoint should you render the page with Playwright or Selenium, waiting for real content and dropping back to requests wherever you can.

If maintaining browsers, proxies, and anti-bot workarounds is not how you want to spend your time, scraping.pro offers this as a managed web scraping service and delivers the extracted data as a clean feed.