Web Scraping with Node.js: Puppeteer and Apify Tutorial

If you already write JavaScript, there is no reason to switch languages to pull data off the web. Node.js runs the same language your target pages run, ships a mature ecosystem of scraping libraries, and handles thousands of concurrent requests without breaking a sweat. This tutorial shows you how to scrape with Node.js end to end: a plain HTTP-and-Cheerio scraper for static pages, a Puppeteer script for JavaScript-rendered content, and a production-grade crawler built on Crawlee (the open-source successor to the classic Apify SDK) that scales to tens of thousands of URLs and runs in the cloud.

We will use a business directory as the running case study, because directories are the classic hard target: hundreds of thousands of records, search result caps, per-account limits, and pages that only fill in once JavaScript runs. Everything here generalizes to any dynamic site.

Why Node.js for web scraping

A few practical reasons Node.js is a strong pick for a node js screen scraper:

• One language, front to back. The DOM logic on the page and the extraction logic in your scraper are both JavaScript. You can copy a selector from DevTools and use it verbatim.
• Async by design. Node's event loop is built for I/O-bound work, and scraping is almost entirely I/O — waiting on the network. You can keep hundreds of requests in flight with a fraction of the memory a thread-per-request model would burn.
• First-class headless browser control. Puppeteer and Playwright are both Node libraries maintained by the teams behind the browsers themselves.
• A batteries-included crawler framework. Crawlee gives you request queues, autoscaling, proxy rotation, session management, and structured storage out of the box — the parts you would otherwise have to build yourself.

If you want the wider language comparison, see choosing a language for scraping. For Node specifically, read on.

The toolchain: Cheerio, Puppeteer, Playwright, Crawlee

A key decision on every project: do you need a browser at all? Rendering a full Chromium instance per page is powerful but expensive. If the data you want is present in the initial HTML (view source, or check the Network tab for a JSON API), skip the browser entirely and use raw HTTP plus Cheerio — it is roughly an order of magnitude faster and lighter. Reach for Puppeteer or Playwright only when the page genuinely needs JavaScript to render.

Project setup

You need Node.js 20 LTS or newer. Create a project:

mkdir node-scraper && cd node-scraper
npm init -y
npm pkg set type=module
npm install crawlee puppeteer cheerio

puppeteer downloads a compatible Chromium build automatically, so there is nothing else to install.

A first scraper: static pages with HTTP + Cheerio

When the markup already contains your data, this is all you need:

import * as cheerio from 'cheerio';

const res = await fetch('https://books.toscrape.com/');
const html = await res.text();
const $ = cheerio.load(html);

const books = $('.product_pod').map((_, el) => ({
  title: $(el).find('h3 a').attr('title'),
  price: $(el).find('.price_color').text().trim(),
  inStock: $(el).find('.availability').text().includes('In stock'),
})).get();

console.log(books);

No browser, no rendering — just a request and a parse. On a static catalog this scrapes hundreds of pages per minute. Cheerio understands CSS selectors, so anything you can target in DevTools you can target here.

Scraping JavaScript-rendered pages with Puppeteer

When the page loads its data after render — infinite scroll, "load more" buttons, client-side frameworks — you need a real browser. Puppeteer launches headless Chromium and lets you drive it:

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({ headless: 'new' });
const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'networkidle2' });
await page.waitForSelector('.quote');

const quotes = await page.$$eval('.quote', nodes =>
  nodes.map(n => ({
    text: n.querySelector('.text')?.innerText,
    author: n.querySelector('.author')?.innerText,
  }))
);

console.log(quotes);
await browser.close();

The pattern that matters: navigate, wait for the real signal, then extract. Waiting on a specific selector (waitForSelector) is far more reliable than a fixed sleep. Playwright works almost identically and adds automatic waiting plus Firefox/WebKit support, so if you are starting fresh it is a fine alternative — the concepts transfer directly. See Puppeteer and headless browser scraping for the deeper dive on stealth and resource blocking.

Scaling up with Crawlee (the modern Apify SDK)

A single script is fine for a page or two. Real projects need a queue of URLs, concurrency control, automatic retries, proxy rotation, and somewhere to put results. The old Apify SDK handled all of this; today that library lives on as Crawlee, an open-source framework maintained by Apify. Its PuppeteerCrawler is the most powerful crawler for JS-heavy sites; CheerioCrawler is the fast path for static pages; PlaywrightCrawler is the cross-browser option.

Here is the same quote-scraping job as a real crawler:

import { PuppeteerCrawler, Dataset } from 'crawlee';

const crawler = new PuppeteerCrawler({
  maxConcurrency: 8,
  maxRequestsPerCrawl: 12000,
  requestHandlerTimeoutSecs: 120,

  async requestHandler({ page, request, enqueueLinks, pushData, log }) {
    log.info(`Scraping ${request.url}`);
    await page.waitForSelector('.quote');

    const quotes = await page.$$eval('.quote', nodes =>
      nodes.map(n => ({
        text: n.querySelector('.text')?.innerText,
        author: n.querySelector('.author')?.innerText,
      }))
    );
    await pushData(quotes);

    // follow pagination automatically
    await enqueueLinks({ selector: '.next a' });
  },

  failedRequestHandler({ request, log }) {
    log.error(`Request ${request.url} failed too many times.`);
  },
});

await crawler.run(['https://quotes.toscrape.com/js/']);
await Dataset.exportToJSON('OUTPUT');

What you get for free here is exactly the machinery that used to take a week to build by hand:

• A managed request queue. enqueueLinks and crawler.run([...]) push URLs into a RequestQueue that automatically deduplicates — a URL already handled or already pending will not be added twice. That single feature is what makes crawling a directory tractable.
• Autoscaling. Crawlee watches CPU and memory and dials concurrency up or down to use available resources without thrashing.
• Automatic retries with backoff, plus a failedRequestHandler for URLs that exhaust their retries.
• Structured storage. pushData writes to a Dataset; Dataset.exportToJSON/exportToCSV gives you a consolidated file at the end.
• Built-in proxy rotation through ProxyConfiguration (more below).

Case study: crawling a large business directory

Directories such as company aggregators, Yellow Pages, or professional networks share a set of obstacles. Here is how to handle each, using the general techniques that a real Xing-style company scrape relied on.

1. Estimate the scope before you start

Directories hide their true totals — a search that would return more than ~10,000 hits often shows only a capped count, and each query returns a limited number of results (say 300). Before committing compute, estimate how many records exist so you can budget time and cost. The trick: for a category the site does disclose (for example, a small country, or a narrow company-size band), read the real count. Then extrapolate to the hidden categories using the visible ratios between them. You end up with a defensible approximation of total records — enough to plan the crawl and, if you are billing per record, to quote the job.

2. Beat the results cap by fanning out searches

If a single search only ever returns 300 results, you cannot enumerate a 60,000-company category directly. The answer is to split one broad search into many narrow ones whose individual result counts fall under the cap, then union the results (the request queue dedupes the overlap for you). Directories give you the filters to do this — location, industry, company size, and a free-text keyword. Combine them:

/search/companies?filter.location[]=2951839&filter.size[]=4&keywords=aH

When location plus size still overflows the cap, add a keywords dimension seeded with single letters (a, b, c…) and then letter pairs (aa, ab, ac…) until each query returns fewer results than the threshold. Generate these search URLs at startup and push them all into the queue:

const letters = [...'abcdefghijklmnopqrstuvwxyz'];
for (const loc of locations) {
  for (const size of sizeBands) {
    for (const kw of letters) {
      await crawler.addRequests([{
        url: `${base}?filter.location[]=${loc}&filter.size[]=${size}&keywords=${kw}`,
        label: 'SEARCH',
      }]);
    }
  }
}

Your requestHandler then branches on request.label: a SEARCH page yields company-profile URLs (enqueue them as DETAIL), and a DETAIL page yields the actual company record (push it to the dataset). This two-phase crawl — discover URLs, then scrape them — is the backbone of directory scraping and mirrors crawling versus scraping.

3. Respect per-account and per-IP limits

Many directories cap how many pages one account can view per month, and will throttle or ban an IP that requests too aggressively. Two mitigations:

• Rotate accounts. Keep a small pool of logins and switch when one approaches its cap. Free accounts are usually enough; you rarely need premium tiers just to read public listings.
• Rotate IPs. Route the crawler through rotating proxies so the target sees many unrelated visitors rather than one hammering address. Crawlee wires this in directly:

import { ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: ['http://user:pass@proxy1:8000', 'http://user:pass@proxy2:8000'],
});
const crawler = new PuppeteerCrawler({ proxyConfiguration, /* ... */ });

Add a small randomized delay between requests so you do not throttle the site, and be aware that heavy directories often layer on CAPTCHAs once they suspect automation.

4. Validate the data you collect

Directories sometimes substitute their own domain for a company's real website, or return partial records. After the crawl, run a validation pass: flag records whose website field points back at the directory's own domain, check counts per country and per size band against your earlier estimate, and drop duplicates. A short post-processing script beats trusting raw output. If you need the records cleaned and unified afterward, see data normalization.

Running it in the cloud (Apify Actors)

Local scraping is limited by your machine's CPU and memory, which cap concurrency and therefore speed. To run bigger jobs, package the crawler as an Apify Actor — a containerized microservice that takes a JSON input, runs your job, and stores the results. The workflow is straightforward:

npm install -g apify-cli
apify create my-directory-scraper   # scaffolds an Actor project
apify run                           # run locally against INPUT.json
apify push                          # deploy to the Apify platform

Your Actor reads its configuration from an input object — the letters to fan out, the locations, size bands, concurrency, account credentials, and so on — so you can re-run the same code with different parameters without editing it. On the platform, compute is metered in Compute Units (CU); how many pages you get per CU depends on how heavy the pages are and how tight your code is (logging on every request, for instance, measurably slows a run). Autoscaling, proxy pools, scheduling, and storage are handled by the platform. You can just as easily run the same Crawlee code on your own server or in any container host — the framework does not require Apify's cloud.

Exporting and using the data

At the end of a run, export the dataset:

await Dataset.exportToCSV('companies');   // or exportToJSON

From there it is a normal CSV/JSON pipeline — load it into a database, a spreadsheet, or a warehouse. If you would rather skip the infrastructure entirely, scraping.pro runs directory and aggregator extraction as a done-for-you web scraping service, delivering clean, validated records on a schedule.

FAQ

Is Node.js better than Python for scraping? Neither is universally better. Python has the larger data-science ecosystem; Node.js has a slight edge for JavaScript-rendered sites because you stay in the page's own language and its async model handles high-concurrency I/O naturally. Use whichever your team knows. Compare with Python web scraping.

Puppeteer or Playwright? Both drive headless Chromium well. Playwright adds cross-browser support (Firefox, WebKit) and stronger auto-waiting; Puppeteer is slightly lighter and Chrome-focused. For new projects, Playwright is the marginally safer default, but Crawlee supports both.

When should I avoid a headless browser? Whenever the data is in the raw HTML or reachable via a JSON endpoint. Rendering Chromium is 5–10x more expensive than a plain HTTP request. Check the Network tab first — you may be able to hit a private API directly.

How do I keep a large crawl from getting blocked? Rotate IPs with proxies, rotate user agents, throttle your request rate, reuse sessions, and respect the site's limits. Distributed, human-paced traffic is far harder to detect than a burst from one address.

Wrapping up

Scraping with Node.js scales cleanly from a ten-line Cheerio script to a Crawlee crawler chewing through tens of thousands of directory pages in the cloud. The mental model stays the same throughout: decide whether you truly need a browser, drive it with Puppeteer or Playwright when you do, and let Crawlee handle the queue, retries, proxies, and storage so you can focus on the extraction logic. Start small on a static page, add a headless browser only where the page forces you to, and reach for the framework the moment scale enters the picture.