Convert a Website to JSON: A Simple Way

HTML is built for browsers; JSON is built for programs. So a recurring task in web scraping is turning a web page into clean, structured JSON your code can actually use — a list of products, a table of prices, an article's fields. This guide shows the simplest reliable ways to convert a website to JSON in 2026, from a two-line trick that works surprisingly often to a few lines of Python and, when you don't want to run anything yourself, the API route.

An older version of this post reviewed a tiny hosted service that took a URL and an XPath and returned JSON. That service is long gone — but the idea is more relevant than ever, and the modern ways to do it are both simpler and sturdier.

First: the page may already contain JSON

Before writing a scraper, check whether the site hands you JSON for free. Very often it does, and this is genuinely the simplest way.

1. Structured data (JSON-LD)

Most commerce, recipe, job, and article pages embed schema.org metadata as JSON-LD for search engines. Open the page source and look for:

<script type="application/ld+json"> ... </script>

That block is already valid JSON with fields like name, price, sku, and ratingValue. Extract the script's contents and you're done — no parsing of the visible HTML at all:

import json, requests
from bs4 import BeautifulSoup

html = requests.get(url, timeout=15).text
soup = BeautifulSoup(html, "lxml")
for tag in soup.select('script[type="application/ld+json"]'):
    data = json.loads(tag.string)
    print(json.dumps(data, indent=2))

2. Hidden JSON APIs

Modern sites render most content from background API calls. Open your browser's DevTools → Network → Fetch/XHR, reload, and watch for requests that return JSON. Hitting that endpoint directly gives you clean data with no HTML parsing — usually the fastest, most robust route of all. (These are the hidden APIs worth hunting for on any dynamic site.)

3. Embedded app state

Framework-driven pages often ship their data inside the HTML as a JSON blob — a <script id="__NEXT_DATA__"> tag on Next.js sites, or a window.__INITIAL_STATE__ assignment. Pull that string out and json.loads it.

If any of these exist, stop here — you already have your JSON.

The general method: selectors to JSON

When the data only lives in the visible markup, the recipe is always the same: fetch the HTML, select the elements you want, build a dictionary, serialize to JSON. A few lines cover it.

import json
import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
resp = requests.get(url, headers={"User-Agent": "MyScraper/1.0"}, timeout=15)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")

products = []
for card in soup.select(".product-card"):
    products.append({
        "name":  card.select_one(".name").get_text(strip=True),
        "price": card.select_one(".price").get_text(strip=True),
        "url":   card.select_one("a")["href"],
    })

print(json.dumps(products, indent=2, ensure_ascii=False))

That's the whole trick. Swap the CSS selectors for the fields you care about and you can convert almost any static page to JSON. ensure_ascii=False keeps characters like £ or é intact rather than escaping them.

Prefer XPath, or want speed on large pages? parsel (the selector library from Scrapy) or selectolax do the same job:

from parsel import Selector

sel = Selector(text=resp.text)
names = sel.xpath('//div[@class="product-card"]//span[@class="name"]/text()').getall()

In JavaScript

The Node equivalent uses cheerio (a server-side jQuery):

import * as cheerio from "cheerio";

const html = await (await fetch(url)).text();
const $ = cheerio.load(html);

const products = $(".product-card").map((_, el) => ({
  name:  $(el).find(".name").text().trim(),
  price: $(el).find(".price").text().trim(),
})).get();

console.log(JSON.stringify(products, null, 2));

When the page is rendered by JavaScript

If requests returns an almost-empty shell, the content is drawn client-side. Either use the hidden-API approach above, or render the page with a headless browser and then extract:

from playwright.sync_api import sync_playwright
import json

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    data = page.eval_on_selector_all(
        ".product-card",
        """els => els.map(e => ({
            name: e.querySelector('.name')?.innerText.trim(),
            price: e.querySelector('.price')?.innerText.trim()
        }))"""
    )
    browser.close()

print(json.dumps(data, indent=2, ensure_ascii=False))

More on driving a real browser is in the Playwright scraping guide.

The no-code / API route

Don't want to run or maintain any code? Several categories of hosted service will take a URL and give you JSON:

• Web scraping APIs (ScraperAPI, Zyte API, Bright Data, and others) fetch the page for you — handling proxies, CAPTCHAs, and JavaScript rendering — and many can return the page pre-parsed into JSON when you supply CSS/XPath rules or use their auto-extraction.
• Automatic extraction / AI APIs (such as Diffbot) go a step further: point them at a product or article URL and they return structured JSON without you defining any selectors, by classifying the page and pulling standard fields.
• Reader APIs can fetch a URL and return cleaned content, optionally wrapped in a JSON envelope — handy for feeding pages to downstream tools or LLMs.
• No-code scrapers (Octoparse, Apify actors, browser-extension tools) let you click the fields you want in a visual editor and export the result as JSON.

The trade-off is the classic one: hosted APIs cost money but absorb the anti-bot arms race; your own script is free but you maintain it.

Making the JSON usable

A few things separate throwaway output from data you can rely on:

• Type your values. Convert "$1,299.00" to a number (1299.0) and a currency field rather than leaving it as a string, so consumers don't have to re-parse it.
• Nest sensibly. Group related fields (e.g., a rating object with value and count) instead of a flat sprawl.
• Stream large results as JSON Lines (.jsonl, one object per line) rather than one giant array — it's far easier to process incrementally.
• Validate. Run the output through a schema or a linter; see the roundup of JSON parsers and viewers for tools that catch malformed output fast.

Want to experiment? scraping.pro runs a public testing ground with sample pages that are safe to practice these techniques on.

FAQ

What's the absolute simplest way to convert a website to JSON? Check whether the page already embeds JSON-LD or calls a JSON API in the background. If it does, you just read that — no parsing required.

Can I turn HTML into JSON without writing code? Yes. Auto-extraction APIs (like Diffbot) and no-code scrapers (Apify, Octoparse) take a URL and return JSON through a UI or a single API call.

Why not just regex the HTML into JSON? Regex breaks the moment the markup shifts. Parse the HTML with a real parser and query it with CSS or XPath — it's shorter and far more resilient.

How do I handle a site that loads data with JavaScript? Find and call its underlying JSON endpoint, or render the page with a headless browser like Playwright and extract from the live DOM.

Converting one page is a few lines; doing it for thousands of URLs, reliably, past blocking and layout changes, is a project. If you'd rather just receive the JSON, scraping.pro offers that as a web scraping service and as an ongoing data feed.