Convert a Website to JSON: A Simple Way

The ten fields you want off a product page are often in the HTML twice. Once as markup a browser can draw, once as a JSON blob put there for search engines. Nearly every guide to web scraping teaches you to rebuild the first one. On the test page we measured for this article, reading the second took 3 milliseconds and rebuilding the first with the BeautifulSoup recipe everyone prints took 560.

This piece follows the order you should try things in: what to look for before writing a parser, the selector code for when nothing is there, the headless-browser version for pages drawn in JavaScript, the failure modes that surface on the second thousand pages, and what the hosted services cost.

An older version of this post reviewed a tiny hosted service that took a URL and an XPath and returned JSON. It is gone. So is the practice site this article used to link to: testing-ground.scraping.pro no longer resolves, checked on 10 August 2026. The Python here was written against Python 3.11.15 with beautifulsoup4 4.14.3, lxml 6.1.0, parsel 1.11.0 and selectolax 0.4.11, and the measurements further down were taken on that setup. Current PyPI releases the same day: beautifulsoup4 4.15.0 (7 June 2026), lxml 6.1.1, parsel 1.11.0, selectolax 0.4.11, playwright 1.62.0 (31 July 2026).

First: the page may already contain JSON

Three places are worth ten seconds each before you write anything.

1. Structured data (JSON-LD)

Open the page source and search for this:

<script type="application/ld+json"> ... </script>

The contents are valid JSON, often with name, sku, price and ratingValue already typed for you.

How often is it there, honestly. The previous version of this article said most commerce, recipe, job and article pages carry it. The measured picture is narrower. The HTTP Archive Web Almanac's 2024 structured data chapter found JSON-LD on 41% of mobile pages, up from 34% in 2022, but the types are mostly site-level furniture: WebSite on 12.7% of pages, Organization on 7.2%, BreadcrumbList on 5.7%, LocalBusiness on 4.0%, ItemList on 2.4%. Product sits below 1%. So the block is usually present and usually not the block you wanted. Check, do not assume.

What is there tends to be truthful, because Google's structured data guidelines make it a ranking risk not to be. Their general policy page says, verbatim, "Don't mark up content that is not visible to readers of the page" and "Your structured data must be a true representation of the page content." That is a reason to trust the fields, not a guarantee. Compare the JSON-LD price against the rendered price on twenty pages before you build on it. Sale prices and per-variant prices are where the two drift apart.

Extracting it takes more care than the two-line version suggests:

import json
from bs4 import BeautifulSoup

def jsonld_nodes(html):
    """Yield every JSON-LD object on a page, unwrapped."""
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.select('script[type="application/ld+json"]'):
        raw = (tag.string or tag.get_text()).strip()
        raw = raw.removeprefix("//<![CDATA[").removesuffix("//]]>").strip()
        try:
            data = json.loads(raw)
        except json.JSONDecodeError:
            continue                      # one bad block must not kill the page
        for node in (data if isinstance(data, list) else [data]):
            if isinstance(node, dict) and "@graph" in node:
                yield from node["@graph"]
            else:
                yield node

def has_type(node, want):
    t = node.get("@type", "")
    return want in (t if isinstance(t, list) else [t])

products = [n for n in jsonld_nodes(html) if has_type(n, "Product")]

Four things in that function are there because of real pages. The top level can be an object or a list. WordPress SEO plugins wrap everything in @graph. Some CMSes still wrap the block in //<![CDATA[, which is not JSON and makes json.loads raise. And @type is sometimes a list, so node["@type"] == "Product" quietly misses ["Product", "Book"].

If you want microdata, RDFa and Open Graph as well, extruct reads all six formats into one dict. Its last release is 0.18.0, from November 2024.

2. Hidden JSON APIs

Open DevTools, go to Network, filter to Fetch/XHR, and reload. Sort by size. The response that holds the listing is usually the biggest JSON body on the list, and calling it directly skips HTML parsing entirely. These are the hidden APIs worth hunting for on any dynamic site.

Copy the request as cURL rather than retyping the URL. What breaks the replay is almost never the path. It is a Referer, an Accept: application/json, an anti-CSRF header or a short-lived bearer token you did not notice.

The trade-off is rarely stated: an internal API has no compatibility contract. A public API version is a promise, an internal one is an implementation detail that changes on a Tuesday deploy with no changelog. Fastest route to JSON, likeliest to break silently.

3. Embedded app state

Framework pages often ship their data inside the HTML as a JSON blob. window.__INITIAL_STATE__ and its cousins are still everywhere, and json.loads on the string between the assignment and the closing </script> still works.

The __NEXT_DATA__ advice has aged badly, and this article used to repeat it. That script tag belongs to the Next.js Pages Router. Pages built with the App Router, the default since Next.js 13, do not emit it at all. The server sends a React Server Component payload instead, split across a run of self.__next_f.push([1, "..."]) calls appended to the body. Inside those strings is not JSON but a line-oriented format of its own, rows like 1:HL["/_next/static/...","font",{...}] separated by \n, which you have to concatenate and then parse row by row. The Next.js maintainers explain the mechanism in discussion 42170. Next.js is on 16.3, released 3 August 2026, so a page you meet today is likelier to be App Router than not.

Find any of the three and you are done. The rest of this article is for when you are not.

What the shortcut is worth

The claim that reading embedded JSON beats parsing HTML is easy to make, so we measured it.

Method. A synthetic catalogue page, 1,183,894 bytes, 2,000 product cards, each with a name, a price, a link, a rating and a review count, plus a <script type="application/ld+json"> block carrying the same 2,000 items as an ItemList. Every approach pulls the same thing: name and price for all 2,000 rows. Median of seven runs, one machine, one core, Python 3.11.15, an Intel Xeon at 2.10 GHz.

Your numbers will differ. The ordering will not, and neither will the gaps, because they come from where the work happens. BeautifulSoup builds a Python object for every node in the tree; lxml and lexbor keep the tree in C and hand you objects only when asked. parsel sits in between: lxml underneath, a Python wrapper on top of every selector call.

The arithmetic that matters is not per page. Ten thousand pages at 560 ms of parsing is an hour and a half of CPU. The same ten thousand at 46 ms is under eight minutes. At 3.3 ms it is half a minute. None of that touches the network wait, which is the real bill on a small crawl. It decides the machine count on a large one.

The general method: selectors to JSON

When the data lives only in the visible markup, the recipe is fixed: fetch the HTML, select the elements, build a dictionary, serialize. The version below is the one from the earlier draft of this article with two bugs taken out.

import json
import requests
from bs4 import BeautifulSoup

def text_of(card, css):
    el = card.select_one(css)
    return el.get_text(strip=True) if el else None

url = "https://example.com/products"
resp = requests.get(url, headers={"User-Agent": "MyScraper/1.0"}, timeout=15)
resp.raise_for_status()

soup = BeautifulSoup(resp.content, "lxml")   # bytes, not resp.text

products = []
for card in soup.select(".product-card"):
    link = card.select_one("a")
    products.append({
        "name":  text_of(card, ".name"),
        "price": text_of(card, ".price"),
        "url":   link["href"] if link and link.has_attr("href") else None,
    })

print(json.dumps(products, indent=2, ensure_ascii=False))

resp.content, not resp.text. This is the fix worth the most. When a server sends Content-Type: text/html with no charset, requests falls back to ISO-8859-1, and resp.text decodes UTF-8 bytes through the wrong codec. Measured on requests 2.33.1: the same page yields Café £9,99 from resp.content and Café £9,99 from resp.text. Handing bytes to the parser lets it read the <meta charset> the page actually declares. The mojibake version parses without an error and ships broken strings into your JSON.

select_one returns None. The original snippet called .get_text() on the result directly, so one card missing a price crashed the run after 800 good rows. A helper that returns None is three lines and turns a crash into a null you can count.

Swap the CSS selectors for the fields you want and this converts almost any static page. ensure_ascii=False keeps £ and é as themselves instead of the escapes \u00a3 and \u00e9.

For XPath, or for speed on large pages, parsel (the selector library from Scrapy) and selectolax do the same job:

from parsel import Selector

sel = Selector(body=resp.content, encoding="utf-8")   # bytes here too
names = sel.xpath('//div[@class="product-card"]//span[@class="name"]/text()').getall()

In JavaScript

The Node equivalent uses cheerio, a server-side jQuery, currently on 1.2.0. Since 1.0 it is ESM-first and needs a namespace import:

import * as cheerio from "cheerio";

const html = await (await fetch(url)).text();
const $ = cheerio.load(html);

const products = $(".product-card").map((_, el) => ({
  name:  $(el).find(".name").text().trim(),
  price: $(el).find(".price").text().trim(),
})).get();

console.log(JSON.stringify(products, null, 2));

When the page is rendered by JavaScript

If requests returns an almost-empty shell, the content is drawn client-side. Call the hidden API if you found one. Otherwise render the page and extract from the live DOM:

from playwright.sync_api import sync_playwright
import json

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url, wait_until="domcontentloaded")
    page.wait_for_selector(".product-card")      # wait for the thing you need
    data = page.eval_on_selector_all(
        ".product-card",
        """els => els.map(e => ({
            name: e.querySelector('.name')?.innerText.trim(),
            price: e.querySelector('.price')?.innerText.trim()
        }))"""
    )
    browser.close()

print(json.dumps(data, indent=2, ensure_ascii=False))

wait_until="networkidle" is gone from that snippet on purpose. The Playwright documentation now marks the value DISCOURAGED and says, in its own words, "Don't use this method for testing, rely on web assertions to assess readiness instead." It waits for 500 ms of network silence, which a page with analytics beacons, a chat widget or a polling socket never reaches, so you pay the full navigation timeout and then fail. Waiting for the selector you are about to query is both faster and a real assertion: if it times out, the data genuinely is not there.

Where this breaks

Nothing above fails loudly. The list below is what actually goes wrong, roughly in order of how long it takes to notice.

• One malformed JSON-LD block takes down the page. A single json.loads outside a try and the whole record is lost, including the good blocks that came after it. This is the most common crash in scrapers built from tutorials.
• A CSS selector string is not portable across libraries. We ran script[type="application/ld+json"] against a page whose tag reads type="application/LD+JSON". BeautifulSoup matched it, selectolax matched it, lxml's cssselect returned nothing. Same string, three libraries, two answers.
• Duplicate keys. RFC 8259 says object names "SHOULD be unique" and nothing more, so a page emitting price twice is technically legal JSON. Python keeps the last one. Some validators keep the first.
• The layout drifts before the data does. A class renamed from .price to .product-price gives you nulls, not errors. Nulls look like out-of-stock items and can sit in a dataset for weeks.
• The JSON-LD is stale. It is generated by a plugin, on a schedule that is not the page's own. Ship a comparison check.

Making the JSON usable

A few habits separate throwaway output from data other people can consume.

Type your values. "$1,299.00" should become 1299.0 plus a currency field. A string forces every consumer to re-parse it, and they will each do it slightly differently.

Nest sensibly. A rating object holding value and count beats rating_value and rating_count sprawled across the top level.

Write JSON Lines for anything large. One JSON value per line, \n between them, no byte order mark, .jsonl on the end. The format page is about a screen long and worth the read. The practical win is that a crawl killed at hour six leaves you a valid file, while a single giant array leaves you an unclosed bracket.

Get the encoding right at the edges. RFC 8259 is blunt about it: "JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8." ensure_ascii=False is safe as long as you write the file as UTF-8, and it is smaller: the record {"name": "Café Ø 12 mm", "price": "£1,299.00"} is 50 bytes that way and 66 with escapes.

Validate before you trust. A schema check catches the day a field starts arriving as a list. For eyeballing output fast, see the roundup of JSON parsers and viewers.

What changes at ten thousand pages

A single page is an afternoon. A nightly run over ten thousand is a different piece of software.

The fetch becomes the whole problem. Anti-bot systems fingerprint the TLS handshake, so requests looks nothing like Chrome on the wire regardless of the User-Agent you set. That is what curl_cffi exists for: it impersonates browser TLS and HTTP/2 signatures, and it shipped 0.16.0 on 1 August 2026. Past that you are into residential proxies and, on some targets, CAPTCHA solving.

The default answer changed. Cloudflare announced on 1 July 2025 that it would block AI crawlers by default for new domains and let publishers charge for access. A large share of the web now sits behind a system whose default posture toward automated clients is no.

Failure has to be per record. One page that times out must not end the run. Write JSON Lines, keep a record of which URLs are done, and make the job resumable. Sites that fight back this hard are where a managed extraction service starts to pay for itself against engineer-hours.

Watch the shape, not just the errors. Track the null rate per field. A jump from 2% to 40% overnight is a layout change, and it is the only signal you get.

The hosted route, priced

If you would rather not run any of this, several categories of service take a URL and hand back JSON. Prices below were read from each vendor's own pricing page on 10 August 2026.

Read the middle column carefully. "Returns JSON" covers two different products. Diffbot and Zyte's extraction classify an arbitrary page and fill standard fields. ScraperAPI's structured endpoints cover a named list of sites: Amazon, eBay, Walmart, Google, Redfin. Both are useful. Only one of them works on your obscure regional retailer.

Letting a model do the extraction

The genuinely new option since this article was first written is handing the HTML to a language model with a JSON Schema and getting conforming output back. It works, and it costs per page forever, which selectors do not.

The arithmetic is worth doing before the experiment rather than after. HTML runs roughly four characters to the token, so a 200 KB page is about 50,000 input tokens. At $0.20 per million input tokens, the rate for OpenAI's cheapest flagship-line model on its pricing page on 10 August 2026, that is one cent a page, or $100 across ten thousand pages, repeated every run. Strip the page to the relevant fragment first and that drops by an order of magnitude. Feeding raw HTML is the expensive mistake.

Where it earns its cost is variety: fifty sites with fifty layouts, or a one-off extraction not worth writing selectors for. Where it does not is the same hundred pages every night, where a selector is deterministic, free and diffable, and a model can return a plausible price that no element on the page contains.

Where this approach stops working

Some pages cannot be turned into JSON by any of the above, and recognising them early saves a week. Content behind a login you do not own is out. So is anything gated by a payment step, or covered by terms you accepted when you opened an account. Data about identifiable people carries obligations that survive its being publicly visible.

The technical limit is different and softer. Once a target requires a real browser, a residential IP, a solved challenge and a session that ages like a human's, you are no longer converting a website to JSON. You are maintaining a browser fleet, and the JSON is a by-product.

To practise any of this, use a site that asks to be scraped. books.toscrape.com has 1,000 books across 50 pages and states plainly that its prices and ratings are random and meaningless, which makes it useless for analysis and ideal for testing a parser.

Converting one page is a few lines. Doing it for thousands of URLs, past blocking and layout drift, with someone on call when the null rate spikes, is a project with a maintenance budget. If you would rather receive the JSON than own that, scraping.pro does it as a web scraping service and as an ongoing data feed.