JavaScript Web Scraping with Node.js: Complete Guide

A practical, end-to-end look at JavaScript web scraping on Node.js: how to fetch pages, pull data out of them, and scale the whole thing up to a production crawler — encodings, concurrency, proxies, TOR, SSL, cookies, headers, URL queues, and the gotchas nobody warns you about. With links to the libraries that actually matter in 2026.

If you already write JavaScript, you are closer to a working scraper than you think. The same language that runs in the browser runs your JavaScript scraper on the server, drives the best headless browsers, and handles thousands of concurrent requests without breaking a sweat. This guide to web scraping with JavaScript builds from the simplest possible request up to a resilient, distributed crawler, one layer at a time.

What this guide covers

1. What web scraping is and its three layers
2. Fetching the page: HTTP clients
3. Libraries for parsing content
4. Reading status codes and response headers
5. Handling non-UTF-8 encodings
6. Working with cookies
7. Working with HTTPS / SSL
8. Using proxies
9. Scraping through TOR
10. Concurrency and multithreading
11. Storing URLs and queues
12. Ready-made frameworks
13. Anti-bot, robots.txt, retries (the parts people forget)
14. The pros and cons of doing this in JavaScript

1. What web scraping is and its three layers

Almost every scraper breaks down into three independent layers, and it pays to design around them from the start:

1. Transport — how you get the bytes of a page (an HTTP client or a headless browser).
2. Extraction — how you pull the fields you want out of HTML/JSON (a DOM parser, selectors).
3. Orchestration — how you walk many URLs without getting banned: queues, concurrency, proxies, retries, and deduplication.

The whole guide is arranged by increasing difficulty: first "fetch a single page," and by the end "a resilient distributed crawler."

There is one fork in the road you should take early:

• Static site (the data is already in the HTML) → an HTTP client plus a DOM parser is enough. Fast, cheap, thousands of pages per minute.
• Dynamic site (the data is loaded by JavaScript) → you need either a headless browser (Playwright / Puppeteer) or a reverse-engineered internal API (the data often lives in a JSON endpoint, and no browser is needed).

Before you reach for a heavyweight browser, always check the Network tab in DevTools: if the page fetches its data from /api/... and gets JSON back, scrape that endpoint, not the rendered DOM.

2. Fetching the page: HTTP clients

2.1. Native fetch (Node 18+) — the default choice

Since Node.js 18, fetch is built in globally, stabilized in Node 21, and supported across the LTS lines (22 and 24). Under the hood it runs on undici, so separate packages like node-fetch are no longer needed for basic work.

const res = await fetch('https://example.com');
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const html = await res.text();

Two things trip up almost every beginner:

• fetch does not throw on 404/500 — you have to check res.ok yourself.
• fetch has no default timeout — a hung socket can hang forever. Set one with AbortSignal.timeout():

const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });

2.2. undici directly — when you need raw speed

undici is the engine behind native fetch, but its low-level API (request, connection pools, pipelining) beats fetch, axios, and got several times over in benchmarks. It makes sense once you are throughput-bound.

import { request } from 'undici';
const { statusCode, headers, body } = await request('https://example.com');
const html = await body.text();

2.3. got and got-scraping — convenience plus browser masking

got is a mature client with built-in retries, hooks, cookie-jar support, and HTTP/2.

For scraping, the more interesting option is the got-scraping fork from Apify: it automatically generates believable browser headers and header ordering, which lowers your ban rate. It is exactly what CheerioCrawler uses inside Crawlee.

import { gotScraping } from 'got-scraping';
const { body } = await gotScraping({ url: 'https://example.com' });

2.4. axios — if you want interceptors and a familiar API

axios (repo) is still the most popular client thanks to interceptors, convenient proxy handling, and automatic JSON parsing. It is not faster than fetch for scraping, but its ecosystem (for example, axios-retry) saves time.

2.5. Everything else

• ky — a thin wrapper over fetch with sensible defaults (retries, timeouts).
• node-fetch — legacy, only needed on very old Node; use the built-in fetch on anything modern.
• The built-in http/https modules — maximum control, but a lot of manual wiring; usually only needed under the hood of agents and proxies.

What to choose

3. Libraries for parsing content

Once you have the HTML string, you need to turn it into data. Do not parse HTML with regular expressions — it is brittle and breaks on the first nested tag. Use a real parser.

3.1. Cheerio — the standard for static pages

Cheerio (repo) is a fast server-side parser with a jQuery-like API. It does not execute JS or render anything — it just builds a tree and lets you walk it with selectors. Ideal paired with fetch/got.

import * as cheerio from 'cheerio';

const html = await (await fetch('https://example.com/products')).text();
const $ = cheerio.load(html);

const items = $('.product-card').map((_, el) => ({
  title: $(el).find('.title').text().trim(),
  price: $(el).find('.price').text().trim(),
  url: new URL($(el).find('a').attr('href'), 'https://example.com').href,
})).get();

3.2. jsdom — an almost-real DOM

jsdom implements a large share of the browser DOM and can even run scripts on the page. It is heavier than Cheerio but gives you the familiar querySelectorAll, document, and comes in handy when you need a more faithful DOM API.

3.3. Light and fast alternatives

• node-html-parser — very fast, with CSS selectors.
• htmlparser2 — a streaming low-level parser (Cheerio is built on it).
• parse5 — a spec-accurate HTML5 parser.
• linkedom — a lightweight jsdom alternative with a DOM API.

3.4. Recipe-style extraction

x-ray lets you describe extraction declaratively (selector → field) and follow pagination out of the box. Handy for prototypes.

3.5. Dynamic content: Playwright and Puppeteer

When JS draws the content, you need a headless browser:

• Playwright — the modern favorite: Chromium, Firefox, and WebKit behind one API, auto-waiting for elements, network interception, and isolated contexts for cookies.
• Puppeteer — the de facto standard for Chrome/Chromium, a bit simpler, with a huge ecosystem.

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle' });
const titles = await page.$$eval('h2', els => els.map(e => e.textContent.trim()));
await browser.close();

A browser is the most resource-hungry option: tens to hundreds of MB of RAM per tab. Use it only when there really is no static HTML and no internal API.

Hybrid tip: it is often optimal to open the page in a browser once, grab the rendered HTML with page.content(), and then parse it with fast Cheerio — combining JS rendering with convenient selectors.

4. Reading status codes and response headers

Status codes and headers are half of scraper diagnostics (a ban, a redirect, a rate limit, an encoding). With native fetch:

const res = await fetch(url, { redirect: 'follow' });

res.status;        // 200, 404, 429, 503 ...
res.statusText;    // 'OK', 'Too Many Requests'
res.ok;            // true on 2xx
res.redirected;    // whether any redirects happened
res.url;           // final URL after redirects

res.headers.get('content-type');   // text/html; charset=ISO-8859-1
res.headers.get('set-cookie');     // cookies
res.headers.get('retry-after');    // how long to wait on 429/503
[...res.headers];                  // every header as pairs

Good habits:

• 429 / 503 → read Retry-After and back off instead of hammering the server.
• 301/302/308 → decide whether to follow the redirect (redirect: 'manual' gives you manual control).
• Content-Type with charset= → the first and most important source of truth about encoding (see §5).
• Controlling your outgoing headers (User-Agent, Accept-Language, Referer) matters just as much — many sites drop requests without a believable User-Agent.

In got/axios the same data is available as response.statusCode and response.headers. In a browser, intercept the response: page.on('response', res => res.status()).

5. Handling non-UTF-8 encodings

A classic headache: a page is served in a legacy encoding — Windows-1251 (Cyrillic), ISO-8859-1/Latin-1 (Western European), Shift-JIS (Japanese), GBK/GB2312 (Chinese), EUC-KR (Korean) — and you get mojibake like café instead of café. The cause: res.text() always decodes bytes as UTF-8, but the site sent them in a different encoding.

The rule: for non-UTF-8 pages, do not use res.text(). Grab the raw bytes (arrayBuffer) and decode with the correct encoding via iconv-lite.

import iconv from 'iconv-lite';

const res = await fetch('https://legacy-site.example/');
const buf = Buffer.from(await res.arrayBuffer());

// 1) try to read the encoding from the Content-Type header
let charset = (res.headers.get('content-type') || '').match(/charset=([^;]+)/i)?.[1];

// 2) if the header has none, look in <meta> (decode a chunk as latin1 to read the tag)
if (!charset) {
  const head = iconv.decode(buf, 'latin1');
  charset = head.match(/<meta[^>]+charset=["']?([\w-]+)/i)?.[1]
         || head.match(/charset=([\w-]+)/i)?.[1];
}

charset = (charset || 'utf-8').toLowerCase().replace('windows-', 'win');

const html = iconv.decode(buf, charset); // correctly decoded text

If the encoding is declared nowhere, you can detect it heuristically:

• jschardet — a port of the Mozilla Universal Charset Detector.
• chardet — an alternative detector.

import jschardet from 'jschardet';
const guess = jschardet.detect(buf); // { encoding: 'windows-1251', confidence: 0.99 }

A few extras:

• Cheerio can decode for you if you pass the buffer and a hint: cheerio.load(buf, { decodeEntities: true }) — but an explicit iconv.decode is more reliable.
• In a headless browser there is usually no encoding problem: the browser decodes the page itself, and page.content() returns correct UTF-8.
• Don't forget HTML entities ( , é) — proper parsers (Cheerio, parse5) decode them for you.

6. Working with cookies

Cookies are needed for authenticated areas, sessions, carts, and getting past "first visit" walls. There are three levels.

6.1. Manually via headers

const res = await fetch(url, { headers: { cookie: 'sid=abc123; lang=en' } });
const setCookie = res.headers.get('set-cookie'); // parse it and send it onward

Fine for simple cases, but maintaining a cookie set across requests by hand is painful.

6.2. A cookie jar (recommended)

tough-cookie is the reference cookie store, honoring domain, path, expiry, and flags. Many clients integrate with it out of the box.

got accepts a jar directly and manages the session for you:

import got from 'got';
import { CookieJar } from 'tough-cookie';

const cookieJar = new CookieJar();
await got('https://site.example/login', { cookieJar, method: 'POST', form: { user, pass } });
const profile = await got('https://site.example/account', { cookieJar }); // cookies attach automatically

For axios there is the axios-cookiejar-support wrapper; for native fetch you will have to wire up tough-cookie by hand or use got/undici.

6.3. In a browser

In Playwright/Puppeteer, cookies live in the context and can be saved and restored — handy so you log in once and reuse the session:

// save state (cookies + localStorage)
await context.storageState({ path: 'state.json' });
// restore it on the next run
const context = await browser.newContext({ storageState: 'state.json' });

7. Working with HTTPS / SSL

A normal HTTPS site needs no effort — fetch/got/axios verify the certificate automatically. The special cases:

7.1. Self-signed / expired certificates

Sometimes you need to turn verification off (for example, when working through a MITM proxy or against a test rig). Do it deliberately — it removes protection against traffic tampering.

// undici / native fetch — via a dispatcher
import { Agent, setGlobalDispatcher } from 'undici';
setGlobalDispatcher(new Agent({ connect: { rejectUnauthorized: false } }));

// got / axios — via https.Agent
import https from 'node:https';
const httpsAgent = new https.Agent({ rejectUnauthorized: false });
// got:  got(url, { agent: { https: httpsAgent } })
// axios: axios.get(url, { httpsAgent })

The global sledgehammer NODE_TLS_REJECT_UNAUTHORIZED=0 disables verification for the whole process — better not to do that in production.

7.2. Custom root certs / client certificates (mTLS)

import https from 'node:https';
import fs from 'node:fs';

const agent = new https.Agent({
  ca:  fs.readFileSync('./ca.pem'),      // your certificate authority
  cert: fs.readFileSync('./client.pem'), // client certificate for mTLS
  key:  fs.readFileSync('./client.key'),
});

7.3. TLS fingerprinting (JA3) — advanced anti-bot

Modern defenses (Cloudflare, DataDome) can tell clients apart by their TLS handshake (JA3/JA4): a Node client's handshake looks different from real Chrome, and that gives away the bot even with perfect headers. Plain Node cannot "fix" this; what helps:

• got-scraping — partially masks the header layer;
• CycleTLS — spoofs the TLS fingerprint;
• an honest headless browser (Playwright) — gives you a real browser TLS handshake.

For heavily protected targets, this is often where rotating proxies and browser-based scraping become non-negotiable.

8. Using proxies

Proxies let you spread load across IPs, bypass geo-restrictions, and dodge IP bans. Types: HTTP, HTTPS, and SOCKS5 (the last is the most universal — it carries any traffic and DNS).

8.1. Native fetch (an important 2026 detail!)

Native fetch has no old { agent } option. The proxy is set through the undici dispatcher — ProxyAgent:

import { ProxyAgent, setGlobalDispatcher } from 'undici';

// globally: every fetch goes through the proxy
setGlobalDispatcher(new ProxyAgent('http://user:pass@proxy.host:8080'));
const res = await fetch('https://example.com');

// or per-request
const res2 = await fetch('https://example.com', {
  dispatcher: new ProxyAgent('http://user:pass@proxy.host:8080'),
});

In Node 24+ you can enable reading HTTP_PROXY/HTTPS_PROXY from the environment with NODE_USE_ENV_PROXY=1 (or --use-env-proxy), but an explicit ProxyAgent is more reliable.

8.2. got / axios via agents

Through the https-proxy-agent and socks-proxy-agent agents:

import got from 'got';
import { HttpsProxyAgent } from 'https-proxy-agent';
import { SocksProxyAgent } from 'socks-proxy-agent';

const httpsAgent  = new HttpsProxyAgent('http://user:pass@proxy:8080');
const socksAgent  = new SocksProxyAgent('socks5h://127.0.0.1:9050'); // h = DNS through the proxy

const r1 = await got('https://example.com', { agent: { https: httpsAgent } });
const r2 = await got('https://example.com', { agent: { http: socksAgent, https: socksAgent } });

8.3. Rotation and proxy pools

At scale you need a pool of proxies with rotation and dead-address culling. The simplest version is to pick a random / round-robin proxy per request. Ready-made tools:

• proxy-chain from Apify — spins up a local proxy that forwards to an upstream (including authenticated ones, which matters for Chromium, since it can't do login/password in --proxy-server).
• In Crawlee, proxy and session rotation is built in (ProxyConfiguration).

// a random proxy from the pool on every request
const pool = ['http://p1:8080', 'http://p2:8080', 'http://p3:8080'];
const pick = () => pool[Math.floor(Math.random() * pool.length)];
await fetch(url, { dispatcher: new ProxyAgent(pick()) });

Proxy types by quality: datacenter (cheap, easily flagged) → residential → mobile (expensive, almost never banned). The right choice depends on how aggressive your target's defenses are. If you would rather not run your own pool, our rotating proxy setup notes cover the trade-offs in more depth.

9. Scraping through TOR

TOR gives you free IP rotation: traffic runs through a chain of nodes, and you can change your exit IP on command. It is handy for learning and small jobs, but the approach has serious limits (see the end of this section).

9.1. Setup

TOR raises a SOCKS proxy on port 9050 and a control port 9051 for management. In the torrc config:

SocksPort 9050
ControlPort 9051
# get the password hash with: tor --hash-password "your_password"
HashedControlPassword 16:....
CookieAuthentication 1

9.2. Requests through TOR

Point the client at the local SOCKS5 (use socks5h so DNS also resolves through TOR — otherwise you leak your real IP):

import got from 'got';
import { SocksProxyAgent } from 'socks-proxy-agent';

const agent = new SocksProxyAgent('socks5h://127.0.0.1:9050');
const res = await got('https://httpbin.org/ip', {
  agent: { http: agent, https: agent },
});
console.log(JSON.parse(res.body).origin); // current TOR exit IP

9.3. Changing identity (a new IP)

To get a new exit IP, send a NEWNYM signal to the control port. You can use the ready-made tor-request, or do it by hand over a plain TCP socket with no dependencies:

import net from 'node:net';

function newTorIdentity(password = '') {
  return new Promise((resolve, reject) => {
    const socket = net.connect(9051, '127.0.0.1', () => {
      socket.write(`AUTHENTICATE "${password}"\r\nSIGNAL NEWNYM\r\nQUIT\r\n`);
    });
    socket.once('error', reject);
    socket.once('end', resolve);
    socket.resume();
  });
}

// between requests:
await newTorIdentity('your_password');

Note: TOR enforces a ~10-second cooldown between circuit changes — you can't rotate the IP faster than that.

9.4. Multiple instances for throughput

One TOR = one exit IP at a time plus a slow cooldown. For a pool of "free proxies," people run several TOR processes on different ports (9050/9051, 9052/9053, …) and round-robin across them. A ready-made Docker image for this is rotating-tor-http-proxy (several instances behind one HTTP endpoint via HAProxy).

9.5. Limitations (read this)

• There are only about 1,500 TOR exit nodes, their lists are public, and Cloudflare/DataDome/most anti-bot systems block them in advance — on protected targets TOR is nearly useless.
• Speed is low and unstable; a new IP is not guaranteed to be "clean" or working.
• It suits learning and small, unprotected targets; for production, use residential/mobile proxies.
• TOR is a privacy tool; use it within the law and within sites' rules.

10. Concurrency and multithreading

It is important to separate two different ideas here.

10.1. First — async concurrency (not threads)

Scraping is an I/O-bound task (you wait on the network). Thanks to the event loop, Node holds hundreds of simultaneous requests on a single thread — real threads are usually not needed. The danger is the opposite: firing Promise.all over 10,000 URLs at once and killing both your own network and the target server. So you cap concurrency.

p-limit limits how many tasks run at once:

import pLimit from 'p-limit';

const limit = pLimit(5); // at most 5 requests at a time
const results = await Promise.all(
  urls.map(url => limit(() => scrape(url)))
);

Relatives:

• p-queue — a queue with priorities, intervals, and rate limiting (e.g. "no more than 10 requests per second").
• p-map — map with a concurrency limit.
• bottleneck — an advanced rate limiter (including distributed via Redis).

10.2. worker_threads — for CPU-bound parsing

If the bottleneck isn't the network but heavy parsing (giant HTML/JSON, regexes, post-processing), it's worth moving it into worker_threads so you don't block the event loop. A convenient wrapper is a pool like piscina.

import { Worker } from 'node:worker_threads';
// each worker parses its own chunk of HTML in parallel, without blocking the main thread

10.3. cluster / multiple processes — to scale across cores

cluster and simply running N processes (often in Docker) spread load across CPU cores and add fault tolerance. In practice, for a crawler this usually means "several workers reading from a shared queue (Redis)" — see §11.

10.4. Auto-scaling out of the box

Crawlee tunes concurrency to the available CPU/RAM itself (AutoscaledPool): less chance of falling over in a small container, and maximum use of a big one.

Practical recipe: for most scrapers — fetch + p-limit/p-queue with a limit of 5–20 concurrent requests. Add threads/processes only once you hit a CPU wall or the limits of a single process.

11. Storing URLs and queues

As soon as a crawler visits more than one page, you get a frontier: a queue of URLs "to visit" plus a set of "already visited" ones.

Key tasks:

• Deduplication. You must not visit the same URL twice. In memory, a plain Set over the normalized URL; at scale, a Bloom filter (compact, at the cost of rare false positives), for example bloom-filters.
• URL normalization. Reduce to a canonical form (sort the query, drop #, trailing slash, UTM tags) or "duplicates" will multiply. normalize-url helps.
• Persistence. If the process crashes, the queue must not be lost. Memory isn't suitable for serious jobs.
• Priorities and traversal order — breadth-first (BFS) or depth-first (DFS), with priority for important sections.

Where to store it:

Crawlee provides a built-in persistent RequestQueue with deduplication and BFS/DFS traversal — if you'd rather not build the frontier by hand, that's the fastest path.

A typical "grown-up" architecture: Redis/BullMQ as the URL queue → a pool of workers takes tasks, parses, puts found links back on the queue (after dedup), and writes the result to a database/file.

12. Ready-made frameworks

If you don't want to assemble all of the above by hand:

Crawlee (repo) is the leading modern framework for Node.js/TS, from Apify. One interface for both HTTP and browser crawling (CheerioCrawler, PuppeteerCrawler, PlaywrightCrawler), a persistent URL queue, proxy and session rotation, auto-scaling, human-like browser fingerprints, and retries. Recent versions add an adaptive crawler (it decides for itself whether JS rendering is needed) and AI-oriented features. Requires Node 16+.

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  maxConcurrency: 10,
  async requestHandler({ $, request, enqueueLinks, pushData }) {
    await pushData({ url: request.url, title: $('title').text() });
    await enqueueLinks(); // finds links itself and enqueues them with dedup
  },
});
await crawler.run(['https://example.com']);

• node-crawler — a more classic crawler with a queue, limits, and built-in Cheerio.
• x-ray — declarative extraction plus pagination.

For most serious JS projects, the default answer today is Crawlee.

13. Anti-bot, robots.txt, retries (the parts people forget)

These topics rarely make it into a plan, but a production scraper doesn't survive without them.

13.1. Blending in as an ordinary client

• Set a believable User-Agent, Accept-Language, and Referer. A list of real UAs lives in user-agents.
• To generate consistent header sets and fingerprints, use got-scraping and Apify's fingerprint-suite.
• Against strong defenses (Cloudflare and the like), only an honest browser (Playwright) or a spoofed TLS fingerprint (see §7.3) will do. When you hit those walls, solving CAPTCHAs becomes part of the pipeline too.

13.2. Politeness and retries

• Respect robots.txt where it applies — robots-parser helps you parse it.
• Apply rate limiting and random delays between requests (p-queue/bottleneck).
• On 429/503, honor Retry-After, use exponential backoff with jitter, and cap the number of retries.
• Cache what you have already downloaded so a restart doesn't hit the site again.

Legally, scraping public data is generally defensible in the US (see hiQ v. LinkedIn and the narrowing of the CFAA), but personal data pulls in GDPR and CCPA obligations. Read the site's terms, and when in doubt, get counsel.

14. The pros and cons of doing this in JavaScript

Pros

• One language with the page. Sites are written in JS — keeping selectors, DOM logic, and even executing page scripts in the same environment is convenient.
• The best headless browsers are natively JS. Playwright and Puppeteer are first-class citizens in Node; for heavy dynamic content that's a strong advantage over other ecosystems.
• Async out of the box. The event loop fits I/O-bound scraping perfectly: high concurrency in a single process with no thread wrangling.
• A mature ecosystem. fetch/undici, Cheerio, Crawlee, BullMQ, ready-made proxy agents — it's all right there.
• Crawlee covers orchestration (queues, proxies, fingerprints, scaling) with almost no code.

Cons

• CPU-bound parsing (huge documents, heavy post-processing) is single-threaded Node's weak spot; you need worker_threads/multiple processes, whereas in Go/Rust that's simpler.
• Browser hunger. Playwright/Puppeteer eat a lot of RAM/CPU; at scale that's a real cost.
• Callback/promise soup during manual orchestration without a framework easily turns into spaghetti.
• TLS fingerprinting. Node clients get flagged by JA3/JA4; "fixing" this with pure Node is harder than it looks (you need CycleTLS or a browser).
• Selector brittleness. This is scraping's universal curse (layouts change), but the JS ecosystem doesn't spare you from hand-maintaining CSS/XPath selectors.
• Data science around the data. Python with pandas/numpy is stronger for downstream analytics — sometimes it's easier to "collect in JS, process in Python."

When JS is a good choice: dynamic sites, a need for a headless browser, a team already on Node, high I/O concurrency, and/or integration with JS web services. When to consider an alternative: purely CPU-bound processing of terabytes of HTML, or tight integration with Python analytics.

Don't want to build and babysit all this?

Getting a JavaScript scraper to work is easy; keeping it running past the first anti-bot update, proxy rotation, and layout change is the hard part. If you'd rather have the data than the maintenance, scraping.pro runs this as a done-for-you data extraction service — from one-off pulls to continuous data as a service feeds, with proxies, retries, and monitoring handled for you.