Web Scraping in the Browser with JavaScript

The question

Here is a question that sounds simple but turns out to be surprisingly deep: sitting in an ordinary web browser, can you write client-side JavaScript that loads an arbitrary HTML page from another website and pulls data out of it?

It is tempting to assume the answer is "yes, of course." The browser is already a powerful HTTP client — it fetches pages all day long. So why not point a script at example.com, grab the markup, and parse it?

The short answer is no, not in any practical way — and the reasons why are a tour through some of the most important security machinery on the web. This article walks through each approach you might reach for, explains exactly where it breaks, and then points you at the methods that do work (which all live on the server side).

Throughout, "scraping" means reading another site's raw HTML response so you can extract structured data from it. The wall you keep running into is built specifically to stop one origin from reading another origin's content.

You can skip to whichever section interests you:

1. The naive approach: just fetch it
2. The same-origin policy
3. "Can I just fake the headers?" — forbidden request headers
4. What you are allowed to load cross-origin
5. CORS: when the other server opts in
6. The no-Referer iframe trick
7. How WordPress shows previews of external pages
8. What actually works: server-side scraping
9. Takeaways

The naive approach: just fetch it

The most direct thing you can try is a single HTTP request from your page's JavaScript. The classic way to do this was XMLHttpRequest; the modern, promise-based equivalent is the Fetch API:

// Modern approach
try {
  const response = await fetch("https://example.com/data/json");
  const data = await response.json();
  console.log(data);
} catch (err) {
  console.error("Request failed:", err);
}

// The older XMLHttpRequest equivalent
const xhr = new XMLHttpRequest();
xhr.open("GET", "https://example.com/data/json");
xhr.onload = () => {
  const data = JSON.parse(xhr.responseText);
  console.log(data);
};
xhr.send();

If example.com happens to be the same origin as the page running the script, this works fine. The moment you point it at a different origin, the browser steps in. Whether you use fetch() or XMLHttpRequest, both obey the same rule, and that rule is the real subject of this article.

The same-origin policy

The same-origin policy (SOP) is the foundational security boundary of the web. In Mozilla's words, it is a critical security mechanism that restricts how a document or script loaded from one origin can interact with a resource from another origin. Its job is to keep a page you are visiting from quietly reading data out of other sites you happen to be logged into.

An origin is the combination of three things:

• scheme (e.g. http vs https),
• host (e.g. example.com), and
• port (e.g. 443).

Two URLs share an origin only if all three match. https://mysite.com and http://mysite.com are different origins (different scheme). https://mysite.com and https://api.mysite.com are different origins (different host). You can read the precise definition in the MDN reference.

The policy treats different kinds of cross-origin interaction differently:

• Cross-origin writes (links, redirects, form submissions) are generally allowed.
• Cross-origin embedding (loading an image, a stylesheet, a script) is generally allowed.
• Cross-origin reads — your script reading the content of another origin's response — are generally forbidden.

Scraping is squarely a cross-origin read, which is exactly the category the browser blocks. This is why the imaginary attack the policy is designed to stop is so serious: without it, any random page could silently issue requests to your bank (where your session cookie is still valid) and read back your transaction history. That is precisely the scenario the same-origin policy exists to prevent.

Can you bypass it? No. There is no client-side switch to turn it off, and that is by design.

"Can I just fake the headers?" — forbidden request headers

A common next idea: if the problem is that the request "looks" cross-origin, what if I lie about where it comes from? You might try to overwrite the Referer or Origin header so the destination server thinks the request is same-origin:

// This does NOT work the way you'd hope
xhr.setRequestHeader("Referer", "https://domain-to-scrape.com");

This fails, and it fails on purpose. The Fetch Standard defines a list of forbidden request headers that scripts are not allowed to set or overwrite. The browser controls these itself precisely so that pages cannot impersonate other origins. The list includes Origin, Cookie, Host, Connection, and many more — see the full enumeration on MDN. If you try to set one, the browser quietly ignores it.

There is one nuance worth being precise about. Referer is on the forbidden list, but the Fetch API does expose a referrer option that lets you adjust it within your own origin (or omit it entirely). What you cannot do is forge an arbitrary foreign Referer or touch the Origin header at all. In other words, the browser will gladly let you trim or blank your own referrer, but it will never let you pretend to be a site you are not. The spoofing path is closed.

What you are allowed to load cross-origin

It is worth being clear about what the same-origin policy does not block, because the exceptions are where most of the confusion comes from. Browsers deliberately allow a handful of HTML elements to embed resources from other origins. The crucial detail is that you can use these resources but you generally cannot read their raw contents with JavaScript.

Resources you can embed cross-origin include:

• Scripts via <script src="...">. The browser will run them, but for syntax errors it only exposes details for same-origin scripts.
• Stylesheets via <link rel="stylesheet" href="...">.
• Images via <img> (PNG, JPEG, GIF, WebP, SVG, and so on). A script can read the image's dimensions but not, in the general case, its pixels once it is "tainted" by a cross-origin source.
• Media via <video> and <audio>.
• Fonts via @font-face (subject to the font server's own CORS rules).
• Embedded documents via <iframe>. A site can refuse to be framed using the X-Frame-Options header or a Content-Security-Policy frame-ancestors directive.

A familiar real-world example is pulling a JavaScript library off a CDN — for years sites loaded jQuery from Google's Hosted Libraries at ajax.googleapis.com:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script>

That works because embedding and executing a cross-origin script is allowed. Reading its source as text through your own script would not be. The same logic applies to the iframe trick later in this article: you can put a foreign page on screen, but you cannot reach into it and read its HTML.

CORS: when the other server opts in

There is a sanctioned way to make a real cross-origin request and read the response — but it requires cooperation from the destination server, not a trick on the client. This mechanism is Cross-Origin Resource Sharing (CORS), standardized as part of the WHATWG Fetch Standard.

The core idea: a server can include response headers that explicitly tell the browser, "it's okay for these other origins to read my data." The key header is Access-Control-Allow-Origin. To allow any origin:

Access-Control-Allow-Origin: *

To allow only one specific origin:

Access-Control-Allow-Origin: https://mysite.com

For "non-simple" requests (custom headers, methods like PUT or DELETE, certain content types), the browser first sends a preflight OPTIONS request to ask the server what it permits, and only sends the real request if the answer allows it.

The decisive point for scraping is who is in control: CORS is granted by the owner of the data, on the response, on their own server. You cannot enable it from the outside. If a site does not return permissive CORS headers, your browser will refuse to hand you the response body, full stop. If you want to experiment, services like the CORS configuration guides on enable-cors.org show how a server operator turns it on — note that it is always a server-side change.

In practice, this means CORS is great for public APIs that deliberately want to be consumed in the browser, and almost useless for scraping a normal website that never intended to be read cross-origin. No site is going to expose its private HTML to arbitrary origins, so from a scraper's perspective the addressable surface is tiny.

The no-Referer iframe trick

Since the same-origin policy makes an exception for <iframe> embedding, you might try to exploit that. The idea is to submit a form with no Referer header into an iframe, on the theory that many sites will still serve a page when the referrer is absent (they don't want to lose the small slice of legitimate traffic that arrives with no referrer).

You can build a self-submitting virtual form inside a data: URL and drop it into a dynamically created iframe:

function noRefRequest(siteUrl) {
  const virtualForm =
    'data:text/html,' +
    '<form id="genform" action="https://www.' + siteUrl + '" method="GET">' +
    '<input type="submit"></form>' +
    '<script>genform.submit()<\/script>';

  const iframe = document.createElement('iframe');
  iframe.width = window.innerWidth;
  iframe.height = '350px';
  iframe.setAttribute('src', virtualForm);
  document.getElementById('response').appendChild(iframe);
}

When this runs, it really does add an iframe and load the foreign page into your browser. If you open your browser's developer tools and inspect the outgoing request, you will see the Origin header reported as null and no Referer present — so in a narrow sense the trick "works": the page loads.

But here is where it dead-ends. The same-origin policy still applies to the contents of that iframe. JavaScript on your page cannot read the HTML of a cross-origin document loaded inside an iframe. Browsers block this to prevent cross-site scripting (XSS) and data-theft attacks. The foreign page will display and even behave normally inside the frame, yet its DOM is sealed off from your code.

A telling detail: you can usually still see the loaded HTML by hand, using the browser's developer tools (open them with F12 and inspect the frame), and you could manually copy and paste it. You might even capture a screenshot image of the rendered frame. But "a human can copy-paste it" and "a script can read it programmatically" are completely different things — and only the second one counts as scraping. An image of a page is not extractable structured data.

So the iframe path gets you a visible foreign page and nothing your code can actually consume.

How WordPress shows previews of external pages

It is worth looking at a system that does successfully display previews of external pages, because it illustrates the only way out. If you have admin access to a WordPress site, visit the comments screen (/wp-admin/edit-comments.php) and hover over a commenter's linked website. The CMS pops up a preview of that external page.

How does it dodge everything described above? It doesn't — it sidesteps the browser entirely. The browser's JavaScript does not fetch the foreign page directly. Instead it asks the WordPress server to do it: it sends a request back to its own backend, passing along the external URL. The WordPress server (which is not bound by the browser's same-origin policy) fetches the target page server-to-server and returns something to the browser.

The catch that proves the point: what WordPress hands back is an image (Content-Type: image/jpeg), not raw HTML. That is the server rendering the page and returning a picture of it. You could just as easily program a server to return the page's HTML or extracted data instead — but notice that all of the real work happened on the server, not in the visitor's browser. The browser was only ever a messenger.

What actually works: server-side scraping

Every dead end in this article points to the same conclusion: real scraping happens outside the browser sandbox, on a server you control, where the same-origin policy and forbidden-header rules simply do not apply. From there you have several mature options.

1. Plain HTTP requests + an HTML parser. For static pages, fetch the markup with any HTTP client and parse it. In Node.js, cheerio gives you a jQuery-like API over the returned HTML; in Python, Requests plus Beautiful Soup is the classic pairing, and Scrapy is a full framework for larger crawls.

2. Check for a hidden API first. Before reaching for anything heavy, open the browser's developer tools, go to the Network tab, and watch the XHR/Fetch requests as the page loads. Many "dynamic" sites are just a front end calling a clean JSON endpoint. Calling that endpoint directly from your server is faster and far more reliable than parsing rendered HTML — and sometimes the data is also available via an official export, an RSS feed, or a documented public API, any of which is preferable to scraping at all.

3. Headless browsers for JavaScript-heavy pages. When content is rendered client-side (single-page apps, infinite scroll, content behind interaction), you need a real browser engine driven programmatically. The two standard tools are:

• Puppeteer — a Node.js library, originally from Google's Chrome team, that drives headless Chrome/Chromium over the DevTools Protocol.
• Playwright — built by Microsoft (by some of the same people who started Puppeteer), with one API across Chromium, Firefox, and WebKit, official clients in JavaScript, Python, Java, and .NET, and built-in auto-waiting that makes scripts less flaky.

A minimal Playwright example:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto("https://example.com");
const html = await page.content();   // fully rendered HTML, readable by your code
console.log(html);
await browser.close();

Because this runs server-side in a real engine, the same-origin restrictions that blocked you in the browser are irrelevant — you are the browser now, not a script trapped inside a page on someone else's origin.

4. A word on responsibility. Server-side power comes with obligations. Respect each site's robots.txt and terms of service, rate-limit your requests so you don't overload anyone, prefer official APIs and data feeds when they exist, and be mindful of the legal and copyright status of the data you collect. "Technically possible" is not the same as "permitted."

Takeaways

Pulling another site's HTML with pure client-side JavaScript is not practical, and the obstacles are features, not bugs:

1. The browser is a weaker engine for this job than a server — less memory, less control, and a sandbox built to constrain it.
2. The same-origin policy blocks cross-origin reads to protect users from XSS and cross-site data theft. There is no client-side override.
3. You cannot spoof your way around it. Headers like Origin are forbidden to scripts; you can only trim or omit your own Referer, never forge a foreign one.
4. CORS is opt-in by the data's owner and exists for public APIs, not for reading sites that never agreed to be read cross-origin.
5. The iframe / no-Referer trick loads a foreign page but seals its contents — your code still cannot read the DOM.
6. The only real path is indirect: a server fetches and extracts on your behalf — exactly the pattern WordPress uses for its link previews, and exactly what Puppeteer, Playwright, Scrapy, and friends are built for.

If you take one practical rule away: do the request from a server, not from the browser. Everything else follows from that.

References

• MDN — Same-origin policy
• MDN — Cross-Origin Resource Sharing (CORS)
• MDN — Using the Fetch API
• MDN — Forbidden request header
• WHATWG — Fetch Standard
• Puppeteer · Playwright · Scrapy · Beautiful Soup · Cheerio