Web Scraping with PHP: The Complete Guide

Move a working scraper onto PHP 8.4, switch it to the new Dom\HTMLDocument parser because the release notes promise correct HTML5 handling, and your XPath returns nothing. No exception, no warning, no libxml error in the buffer. Zero nodes. We hit this while rechecking this guide: on the same 1,075,976-byte page, the expression //tr[@class="row"]/td[@class="price"] matched 6,000 cells under the old DOMDocument and exactly 0 under the new parser. The reason is one line in a spec, and the fix is four characters long. It is in the parsing section below.

That is the shape of most PHP scraping trouble in 2026. Not "how do I download a page" — the language has shipped a good HTTP client and a good DOM parser for fifteen years — but the quiet mismatches: a parser that changed its rules, a header that gives you away before the first byte of HTML arrives, a queue that deadlocks at 3 a.m. because a worker died mid-lease.

Everything below was rerun or reread on 13 August 2026. Code was executed on PHP 8.4.21 with libxml 2.9.14 and curl 8.5.0; version numbers and prices come from Packagist, php.net, and each vendor's own page on that date. Where a claim could not be verified, it says so.

Three moving parts, not two

The old model of a scraper has two boxes: a fetcher that downloads HTML and a parser that pulls data out of it. Keep them separate, and you can change how you download — direct, through a proxy, through Tor — without touching extraction logic. That advice is still correct and it is still the single most useful architectural decision in a PHP crawler.

It is also no longer sufficient. There is a third box now, and it sits underneath the fetcher: how your request looks on the wire before any HTML exists. A modern bot-management product classifies you during the TLS handshake, before it has seen your User-Agent header. The curl-impersonate project, which exists solely to close that gap, states the problem plainly in its README: "The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser." Add HTTP/2, and there is a second handshake with its own settings frame and its own fingerprint.

So a PHP scraper in 2026 has three concerns, and they fail differently:

1. Transport. Does the target let you connect and answer with real HTML? Failures here are 403s, endless challenge pages, and empty 200s.
2. Parsing. Does your selector still match? Failures here are silent — zero rows, no error.
3. Volume. Does it survive 100,000 URLs? Failures here are memory, duplicated work, and workers stuck in processing forever.

What changed since this guide was first written. Guzzle is now on 8.0.2, published on 5 August 2026, eight days before this update. Symfony DomCrawler is on 8.1.1 from 5 June 2026. PHP 8.4 brought an entirely new, spec-compliant DOM implementation, and PHP 8.5 deprecated curl_close() outright. Meanwhile, three of the HTML parsers that PHP scraping roundups still recommend have not had a release since 2020, and one of them carries a warning from its own author. There is a section on that below, because a dead parser is the most expensive kind of recommendation: the code installs, runs, and quietly rots.

Which PHP you are actually writing for

Support windows decide what you may use. Read from php.net's supported versions page on 13 August 2026:

PHP 8.1 is not on that page at all any more. PHP 8.2 has four and a half months of security fixes left. If you are starting a crawler today, target 8.4, and treat 8.5 as the version you will be on before this crawler is retired.

Two consequences for scraping code specifically.

curl_close() is dead weight. The manual's changelog is blunt: as of PHP 8.0.0 "this function is a NOP now", and as of PHP 8.5.0 "this function has been deprecated". Since PHP 8.0 a cURL handle is a CurlHandle object, freed by refcount like anything else. Every curl_close($ch) in a PHP 8 codebase does nothing except produce a deprecation notice the moment you upgrade. Delete them, and use unset($ch) where you genuinely need the handle gone at a specific moment. There is one case where that timing matters, and it involves cookies — see below. Details and the changelog rows are on the curl_close manual page.

The DOM you learned is now the legacy DOM. PHP 8.4 added a Dom namespace whose classes are HTML5-aware and WHATWG-compliant, alongside the old DOMDocument. The old classes are not going away, and they are not equivalent. Mixing them up costs you 6,000 rows, as above.

Before the first request: robots.txt, RFC 9309, and the law

Since September 2022 the Robots Exclusion Protocol is an actual standard, RFC 9309, on the Standards Track. That matters because it turns "good manners" into a written set of rules you can be measured against, and the rules are not the ones most hand-rolled parsers implement.

Four of them catch people out:

• Group matching. Rules are grouped by user-agent. A Disallow under User-agent: GPTBot says nothing about your crawler. If several groups match your token, "the matching groups' rules MUST be combined into one group."
• Longest match wins. Allow does not simply lose to Disallow. "The most specific match found MUST be used. The most specific match is the match that has the most octets." On an exact tie, allow wins.
• Wildcards are part of the standard. * means zero or more of any character, $ anchors the end.
• An unreachable robots.txt is not permission. On 5xx, crawlers "MUST assume complete disallow." On 4xx, they "MAY access any resources on the server." Those are opposite defaults, and the difference is one status code.

The earlier version of this guide printed a twenty-line isAllowed() function and told you to use a real parser for anything serious. That was the wrong way round. The snippet ignored user-agent groups, ignored Allow entirely, and — because it used @file_get_contents() and returned true on failure — treated a 503 from robots.txt as a green light, which is precisely the case RFC 9309 marks MUST-disallow. It has been removed rather than patched. Use the library:

composer require spatie/robots-txt

use Spatie\Robots\Robots;

$robots = Robots::create('MyCrawlerBot/1.0');

if (! $robots->mayIndex('https://example.com/catalogue/page/2')) {
    return;   // respect it, and log it, so you know why the queue shrank
}

spatie/robots-txt is on 2.5.4, released 25 February 2026, requires PHP 8.1+, and has been installed 19.7 million times. It also reads robots meta tags and X-Robots-Tag headers, which the hand-rolled version never looked at.

Load is the part nobody standardised. No RFC tells you how fast is too fast. A practical floor: one request at a time per host with a randomised pause, scaled up only when you have watched response times stay flat. Crawl-delay is not in RFC 9309 and not honoured by everyone, but if a site publishes one, ignoring it is a deliberate choice you should be able to defend.

The legal part, stated accurately. Scraping cases get quoted third-hand and usually get the ending wrong. The most-cited one is hiQ Labs v. LinkedIn, and the popular summary — "hiQ won, scraping public data is legal" — is not what happened. hiQ won its early rounds on the Computer Fraud and Abuse Act, where the Ninth Circuit held that scraping a public page is not "access without authorization." It then lost on contract. In November 2022 the district court found hiQ had breached LinkedIn's User Agreement, and in December 2022 the parties entered a judgment against hiQ for $500,000 with a permanent injunction barring it from scraping LinkedIn. The company was already effectively out of business. So the honest statement of the law in the US is narrower than the slogan: the CFAA is a poor fit for public pages, while terms of service, copyright, and trespass theories are alive and are where defendants lose. Our longer treatment is in what is legal to scrape.

GDPR does not care that data is public. Under the EU and UK regimes, a name and an email address scraped from a public profile are personal data, and you need a lawful basis to process them plus, in most cases, to tell the person you did. CCPA/CPRA in California works differently but also does not exempt you because the page was open. Product prices are one thing. People are another, and the second one is where fines live.

Fetching the page

file_get_contents, and its exact failure mode

$html = file_get_contents('https://example.com');

One line, works when allow_url_fopen is on, and it is the right tool for a throwaway script. What it costs you is visibility. On a 404 or 500 it emits a warning and returns false; the status code is not in the return value, it is hidden in a magic local variable:

$context = stream_context_create([
    'http' => [
        'method'        => 'GET',
        'header'        => "User-Agent: Mozilla/5.0\r\n",
        'timeout'       => 10,
        'ignore_errors' => true,   // otherwise a 404 body is discarded
    ],
]);
$html = file_get_contents('https://example.com', false, $context);

// $http_response_header appears in the local scope only after the call
$status = (int) explode(' ', $http_response_header[0] ?? 'HTTP/1.1 0')[1];

No cookie jar, no proxy support, no connection reuse, and a fingerprint that says "PHP" to anything looking. Fine for a one-off. Not a scraper.

cURL, the workhorse

cURL ships with practically every PHP install and gives you the whole request. Here is the version of fetch() we would write today — note what is missing from it:

function fetch(string $url): string
{
    $ch = curl_init();
    curl_setopt_array($ch, [
        CURLOPT_URL             => $url,
        CURLOPT_RETURNTRANSFER  => true,   // return the body instead of printing it
        CURLOPT_FOLLOWLOCATION  => true,   // follow redirects
        CURLOPT_MAXREDIRS       => 5,
        CURLOPT_TIMEOUT         => 30,     // whole-transfer budget
        CURLOPT_CONNECTTIMEOUT  => 10,     // connection phase only
        CURLOPT_USERAGENT       => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                                 . 'AppleWebKit/537.36 (KHTML, like Gecko) '
                                 . 'Chrome/146.0.0.0 Safari/537.36',
        CURLOPT_ACCEPT_ENCODING => '',     // accept every encoding curl was built with
        CURLOPT_HTTP_VERSION    => CURL_HTTP_VERSION_2TLS,
    ]);

    $body = curl_exec($ch);
    $code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
    $err  = curl_errno($ch) ? curl_error($ch) : null;
    unset($ch);          // this releases the handle on PHP 8, not curl_close()

    if ($err !== null) {
        throw new RuntimeException("cURL: $err");
    }
    if ($code !== 200) {
        throw new RuntimeException("HTTP $code for $url");
    }

    return $body;
}

CURLOPT_ACCEPT_ENCODING is the current name of what older tutorials call CURLOPT_ENCODING; passing an empty string advertises every algorithm your libcurl supports and decompresses transparently. On a text-heavy catalogue page that is roughly a 4:1 saving on bandwidth, which is money once you are paying for residential traffic by the gigabyte.

The options worth knowing, in rough order of how often they save you:

PHP 8.5 adds CURLFOLLOW_OBEYCODE and CURLFOLLOW_FIRSTONLY as values for CURLOPT_FOLLOWLOCATION, plus CURLINFO_QUEUE_TIME_T for time spent waiting on a connection — useful when you are trying to work out whether your proxy or the target is the slow one. The full list is on the PHP 8.5 new features page.

Guzzle 8, and when the wrapper earns its place

If Composer is already in the project, Guzzle gives you a friendlier API over the same libcurl: promises, middleware, a real cookie jar, retries. The current release is 8.0.2, published 5 August 2026, from a package with over a billion installs.

use GuzzleHttp\Client;

$client = new Client([
    'timeout'         => 30,
    'connect_timeout' => 10,
    'http_errors'     => false,          // handle 4xx/5xx yourself instead of catching
    'headers'         => ['User-Agent' => 'Mozilla/5.0 ...'],
]);

$response = $client->get('https://example.com');
$status   = $response->getStatusCode();
$html     = (string) $response->getBody();

Guzzle 8's release notes list HTTP/3 support on PHP 8.4 with a suitable libcurl, stricter host and header validation, and enforcement of the __Secure- and __Host- cookie prefixes on responses. That last one is the kind of change that turns a working scraper into a broken one on upgrade, because a target with sloppy cookie naming will now have its cookies rejected. Read the upgrade notes before bumping the constraint. Symfony's HttpClient is the other credible choice, and it is what the ex-Goutte path leads to.

Bare cURL is used in the examples below so the mechanics stay visible. In production, the wrapper is usually worth it.

Parsing: what each approach actually costs

Why not regex, with the failure PHP gives you

HTML is not a regular language, so a regex that works on today's markup breaks on tomorrow's nesting. That argument is old and true. Here is the PHP-specific version, which is worse.

preg_match() returns false on failure and 0 on "no match", and the standard idiom cannot tell them apart:

$subject = str_repeat('word ', 20000) . '.';
$r = preg_match('/^(\s*\w+)+$/', $subject);

var_dump($r);                     // bool(false)
echo preg_last_error_msg();       // JIT stack limit exhausted

if (preg_match('/^(\s*\w+)+$/', $subject)) {
    // never reached
} else {
    // reached: your code has just concluded "no match" from an engine failure
}

That output is from PHP 8.4.21 with stock settings (pcre.jit=1, pcre.backtrack_limit=1000000). Note the error: most write-ups say you will hit the backtrack limit, and with JIT enabled — the default for a decade — you hit the JIT stack limit first, which is error code 6, not 2. Either way preg_match returns false, if (...) reads it as "not found", the field silently comes back empty, and nothing is logged. On an unusually large page. Which is exactly the page you care about.

If you use a regex on markup at all, check the return value against false explicitly and call preg_last_error_msg() on failure. Better: use a regex for what it is good at, which is pulling a number or a date out of a string you have already extracted from the DOM.

DOMDocument and DOMXPath, still the default

Built in, no dependencies, and fast enough for almost everything:

$dom = new DOMDocument();
libxml_use_internal_errors(true);            // real-world HTML produces hundreds of warnings
$dom->loadHTML($html);
libxml_clear_errors();

$xpath = new DOMXPath($dom);

foreach ($xpath->query('//div[@class="article"]//h2') as $node) {
    echo trim($node->textContent), PHP_EOL;
}

foreach ($xpath->query('//a/@href') as $attr) {
    echo $attr->value, PHP_EOL;
}

XPath expressions that cover most extraction work:

That last row is the one to try first on any e-commerce page. A JSON-LD block gives you price, currency, availability, and SKU as typed data, and it changes far less often than the surrounding markup.

The measurement

Three approaches on the same file, run on PHP 8.4.21, libxml 2.9.14, on one machine. The page is a synthetic catalogue: 6,000 table rows, four cells each, 1,075,976 bytes. Each run extracts all 6,000 prices; timings are the median of seven runs in a fresh process; memory is peak RSS from /proc/self/status, minus a ~35.7 MB idle baseline for the CLI process itself.

Measured on a single machine, on a 1,075,976-byte page of regular markup. Your numbers will differ; the ratios will not by much.

Three things fall out of that table.

The DOM costs about 25x the page size in RAM. A one-megabyte page becomes a twenty-five-megabyte tree. Ten workers parsing concurrently is 250 MB before you have stored a single result. That is the number that decides your concurrency, not CPU.

The new HTML5 parser is not slower. It is fatter. It came in marginally ahead on time and used roughly 1.8x the memory of libxml on the same input. Correct HTML5 parsing is worth paying for; know that you are paying in RSS.

memory_get_peak_usage() lies to you here. It reported 2.11 MB for the DOMDocument run while the process actually grew by 25 MB, because libxml allocates outside PHP's memory manager. Your memory_limit does not constrain the DOM tree, and neither does your instrumentation. A long-running worker can look healthy in your metrics right up to the moment the kernel's OOM killer takes it. Measure RSS, not memory_get_usage().

The PHP 8.4 parser, and the trap that empties your XPath

Dom\HTMLDocument, added in PHP 8.4, parses to the WHATWG spec rather than to libxml's HTML4 rules. Two differences change what your selectors match, and we measured both.

Difference one: implied <tbody>. Browsers insert a tbody element into every table that lacks one. libxml never has. So when you copy an XPath out of Chrome DevTools and it reads /html/body/table/tbody/tr[1], it returns nothing under DOMDocument and something under the new parser. On the same one-row test table:

legacy DOMDocument      //table/tbody/tr  -> 0     //table/tr  -> 1
Dom\HTMLDocument        //table/tbody/tr  -> 1     //table/tr  -> 0

The two parsers are exactly inverted. If you migrate, every table XPath in your codebase flips.

Difference two, the expensive one: namespaces. A Dom\HTMLDocument puts elements in the XHTML namespace, http://www.w3.org/1999/xhtml, the way a browser does. Legacy DOMDocument leaves them in no namespace at all. XPath name tests without a prefix only match nodes in no namespace, so every unprefixed expression you own returns an empty list, silently:

$doc   = \Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);
$xpath = new \Dom\XPath($doc);

$xpath->query('//tr[@class="row"]')->length;      // 0 — and no error anywhere

$xpath->registerNamespace('h', 'http://www.w3.org/1999/xhtml');
$xpath->query('//h:tr[@class="row"]')->length;    // 6000

Four characters, h:, on every element name test. That is the whole fix, and nothing in the migration notes shouts about it.

The alternative is to stop writing XPath and use the CSS selectors the new DOM ships with. querySelector() and querySelectorAll() arrived in PHP 8.4, they are namespace-agnostic, and they need no Composer package:

$doc = \Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR);

foreach ($doc->querySelectorAll('tr.row td.price') as $cell) {
    echo trim($cell->textContent), PHP_EOL;
}

$title = $doc->querySelector('meta[property="og:title"]')?->getAttribute('content');

That worked first time, returned all 6,000 rows, and is the reason to move to 8.4 for scraping work. PHP 8.5 adds Dom\Element::$outerHTML and a $children property on top.

Symfony DomCrawler, when you want both

DomCrawler wraps the DOM with a jQuery-ish traversal API and accepts CSS selectors or XPath. Version 8.1.1, released 5 June 2026, 410 million installs:

composer require symfony/dom-crawler symfony/css-selector

use Symfony\Component\DomCrawler\Crawler;

$crawler = new Crawler($html);

$crawler->filter('div.article h2')->each(function (Crawler $node) {
    echo $node->text(), PHP_EOL;
});

$title = $crawler->filter('meta[property="og:title"]')->attr('content');

$rows = $crawler->filterXPath('//table//tr')->each(
    fn (Crawler $tr) => $tr->filter('td')->each(fn (Crawler $td) => trim($td->text()))
);

The css-selector package is what translates div.article h2 into XPath; without it, filter() throws. DomCrawler still builds on the legacy DOMDocument internally, so it inherits the HTML4 table behaviour described above — write //table/tr, not //table/tbody/tr.

Which to pick. Built-in DOM plus querySelectorAll on PHP 8.4 covers most jobs with zero dependencies. DomCrawler is worth its weight when you are also submitting forms and following links, because it pairs with BrowserKit. Regex belongs on strings, not on documents.

Parsers that other roundups still recommend

Every PHP scraping list on the web recommends the same handful of libraries. We checked each on 13 August 2026, and the state of the field is not what the lists say.

Simple HTML DOM. The SourceForge project page still offers simplehtmldom_1_9_1.zip, released 20 October 2019, and the newest thing in the file area is dated 9 November 2019. The previous version of this guide called it "barely maintained" and suggested it was "fine for small jobs". That was too generous on both counts. It parses with string manipulation rather than a real tokenizer, it is the reason "PHP DOM parsing eats memory" became folklore, and nothing has been shipped for it since before the pandemic.

voku/simple_html_dom is the fork that stayed alive. Version 5.0.0 landed on 21 April 2026, 9.3 million installs, and the important detail is that it is not the old code: it is built on DOMDocument and Symfony's CssSelector underneath, with the familiar find('div.item') API on top. If you have legacy code full of str_get_html(), this is the migration that costs you the least.

phpQuery. The most-installed build, electrolinux/phpquery, last released 0.9.7 on 2 May 2023. Its README carries the maintainer's own assessment: he has not used the package in years, recently looked at the code, and calls it "scary, buggy and unfinished", followed by a plea not to run it on a production server. That is the author, not a competitor. The earlier version of this guide described phpQuery as merely "dated". It is not dated. It is disowned.

QueryPath. Last release 3.0.5 on 1 August 2016. The README now redirects users: "The version of QueryPath in this repository is no longer maintained. You are encouraged to use GravityPDF's version." The fork is real and does get attention; the package most tutorials name does not.

paquettg/php-html-parser. 3.1.1, 1 November 2020, 8.4 million installs, 154 dependent packages, 77 open issues. Not abandoned in the formal Packagist sense, not moving either.

Goutte deserves a line of its own because it is still the top answer in a lot of "PHP web scraping" search results. The repository was archived on 1 April 2023 and the README is explicit: "This library is deprecated. As of v4, Goutte became a simple proxy to the HttpBrowser class from the Symfony BrowserKit component. To migrate, replace Goutte\Client by Symfony\Component\BrowserKit\HttpBrowser in your code." The replacement is genuinely better, because it is Symfony HttpClient underneath and it hands you a DomCrawler:

use Symfony\Component\BrowserKit\HttpBrowser;
use Symfony\Component\HttpClient\HttpClient;

$browser = new HttpBrowser(HttpClient::create());

$crawler = $browser->request('GET', 'https://example.com/search?q=widgets');
$crawler = $browser->clickLink('Next');

$titles = $crawler->filter('h2.result')->each(fn ($n) => $n->text());

The BrowserKit documentation covers form submission and JSON responses too, and $browser->getResponse()->toArray() decodes a JSON body without a separate step. That is the honest replacement for Goutte, and it is where the maintainers point you.

One tool that is not on the list of the dead: roach-php/core, v3.2.1 from 21 March 2025, an unashamed port of Scrapy's ideas to PHP with spiders, middleware, item pipelines, and a run loop. It is a small project by download count. If you keep rewriting the same queue-and-worker scaffolding, look at it before you write another one.

Encoding: mojibake, measured

Garbled text is the most common single complaint about PHP scraping, and the cause is always the same shape: the bytes are in one encoding and something downstream assumed another. Three cases, all run on PHP 8.4.21 with libxml 2.9.14.

Case one: UTF-8 bytes, no charset declaration. The page contains Café Ünicode — 20 € and no <meta charset>:

DOMDocument::loadHTML($html)                          -> Café Ãnicode â 20 â¬
DOMDocument::loadHTML('<?xml encoding="UTF-8">'.$html) -> Café Ünicode — 20 €
Dom\HTMLDocument::createFromString($html)              -> Café Ünicode — 20 €

The legacy parser falls back to Latin-1 and mangles everything. The <?xml encoding> prefix hack fixes it and has been the standard workaround for a decade. The PHP 8.4 parser got it right with no help at all.

Case two: the hack leaves a trace. That XML declaration does not vanish; it becomes a node in the tree, and saveHTML() emits a document that starts with an HTML 4.0 Transitional doctype it invented. If you round-trip HTML through a parse and reserialize, the output is not the input. Prefer Dom\HTMLDocument::createFromString($html, LIBXML_NOERROR, 'UTF-8') on 8.4+, where the third parameter is a documented override encoding rather than a trick.

Case three: detection is a coin flip. The old fallback of mb_detect_encoding($html, ['UTF-8','Windows-1252','ISO-8859-1'], true) has a failure mode you can reproduce in one line. The byte string caf\xE9 is valid in both Windows-1252 and ISO-8859-1, so mb_detect_encoding returns whichever appears first in your candidate list:

$bytes = "caf\xE9";
mb_detect_encoding($bytes, ['UTF-8','Windows-1252','ISO-8859-1'], true);  // "Windows-1252"
mb_detect_encoding($bytes, ['UTF-8','ISO-8859-1','Windows-1252'], true);  // "ISO-8859-1"

Same bytes, different answer, decided by array order. Those two encodings differ in the range 0x80-0x9F, which is where curly quotes, the em dash and the euro sign live. Guess ISO-8859-1 on a Windows-1252 page and every smart quote becomes an invisible control character that survives all the way into your database.

The order to trust sources in: the HTTP Content-Type header, then the <meta charset> tag, then detection, and treat the third as a guess you log. cURL hands you the header for free:

$charset = null;
$ct = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);            // "text/html; charset=windows-1251"
if ($ct && preg_match('/charset=["\']?([\w-]+)/i', $ct, $m)) {
    $charset = strtoupper($m[1]);
}
if ($charset === null && preg_match('/<meta[^>]+charset=["\']?([\w-]+)/i', $html, $m)) {
    $charset = strtoupper($m[1]);
}
$charset ??= mb_detect_encoding($html, ['UTF-8', 'Windows-1252', 'ISO-8859-1'], true) ?: 'UTF-8';

if ($charset !== 'UTF-8') {
    $html = mb_convert_encoding($html, 'UTF-8', $charset);
    $html = preg_replace('/charset=[\w-]+/i', 'charset=UTF-8', $html, 1);
}

Rule of thumb. Normalise to UTF-8 once, in the fetcher, before anything else sees the bytes. Rewrite the meta declaration while you are there, so the parser cannot disagree with you afterwards. Do that and encoding stops being a category of bug.

Concurrency with curl_multi

PHP has no threads in the usual sense, and it does not need them here: libcurl multiplexes sockets by itself. The pattern is a sliding window — keep $concurrency transfers in flight, start a new one as each finishes.

The version below fixes two things the earlier one in this guide got wrong:

function fetchMany(array $urls, int $concurrency = 10, ?callable $onPage = null): void
{
    $multi    = curl_multi_init();
    $queue    = array_values($urls);
    $inFlight = [];

    $add = function (string $url) use ($multi, &$inFlight): void {
        $ch = curl_init();
        curl_setopt_array($ch, [
            CURLOPT_URL             => $url,
            CURLOPT_RETURNTRANSFER  => true,
            CURLOPT_FOLLOWLOCATION  => true,
            CURLOPT_TIMEOUT         => 30,
            CURLOPT_ACCEPT_ENCODING => '',
        ]);
        curl_multi_add_handle($multi, $ch);
        $inFlight[(int) $ch] = $url;
    };

    for ($i = 0; $i < $concurrency && $queue; $i++) {
        $add(array_shift($queue));
    }

    do {
        curl_multi_exec($multi, $running);

        // -1 means the underlying select() failed or had nothing to wait on.
        // Without the sleep, this loop pins a CPU core at 100%.
        if (curl_multi_select($multi, 1.0) === -1) {
            usleep(100_000);
        }

        while ($done = curl_multi_info_read($multi)) {
            $ch   = $done['handle'];
            $url  = $inFlight[(int) $ch];
            $code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);

            if ($done['result'] !== CURLE_OK) {
                error_log("$url failed: " . curl_strerror($done['result']));
            } elseif ($onPage !== null) {
                // parse and persist HERE; do not accumulate bodies in an array
                $onPage($url, curl_multi_getcontent($ch), $code);
            }

            curl_multi_remove_handle($multi, $ch);
            unset($inFlight[(int) $ch], $ch);

            if ($queue) {
                $add(array_shift($queue));
            }
        }
    } while ($running || $queue || $inFlight);

    curl_multi_close($multi);
}

The -1 check is not paranoia. The manual for curl_multi_select says it returns "-1 on a select failure (from the underlying select() system call)", and the user notes are full of platforms where it returns -1 constantly. Call it in a loop without handling that, and instead of sleeping until a socket is ready you spin. The symptom is a crawler that is not measurably faster than the single-threaded version while one core sits at 100%.

Returning an array of bodies is the other bug. The original fetchMany() collected every response into $results and returned it. Ten thousand pages at 300 KB each is 3 GB of strings, and PHP will hit memory_limit long before the crawl finishes. Passing a callback and handling each page as it lands keeps memory flat regardless of queue length.

Alternatives, in the order most people need them:

• Guzzle Pool. Same sliding window, declarative, with per-request fulfilled and rejected callbacks. If you already have Guzzle, start here.
• Fibers, AMPHP, ReactPHP. True async in PHP 8.1+, worth it when a scraper also has to talk to a database, a queue and an API concurrently. The rewrite cost is real: a fiber-based crawler is a different program, not a modified one.
• Swoole or a process pool. Multiple OS processes, each pulling from a shared queue. Boring, easy to reason about, and the only model that uses more than one core for parsing. Since parsing is where your CPU goes — 51 ms per page in the measurement above, against a few milliseconds of PHP overhead per request — this matters more than it looks.

Concurrency is not a number you tune once. Ten in flight against one host is aggressive; ten across a hundred hosts is polite. Rate-limit per host, not globally.

Proxies, with prices

Proxies buy you three things: escape from IP-based blocking, geography, and the ability to spread load so no single address looks like a crawler.

curl_setopt_array($ch, [
    CURLOPT_PROXY     => '123.45.67.89:8080',
    CURLOPT_PROXYTYPE => CURLPROXY_HTTP,
]);

// with authentication
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'login:password');

What it costs, read from vendor pricing pages on 13 August 2026. Webshare publishes per-unit numbers, which makes it a usable yardstick: 10 shared datacenter proxies free, 100 for $2.99/month, 1,000 for $26.91/month. Rotating residential is priced by traffic, from $3.50/GB at 1 GB down to $2.25/GB at 100 GB and $1.40/GB at 3,000 GB. Static residential runs $0.30 per proxy per month at small volumes.

Turn that into a crawl budget. A million pages at 300 KB of compressed HTML each is roughly 300 GB. Through rotating residential at $2.25/GB, that is about $675 in bandwidth for one pass. Through datacenter proxies it is a rounding error — and datacenter ranges are the first thing an anti-bot vendor blocks. That gap, not the monthly price, is the actual decision.

Rotation needs failure tracking, not just round-robin. A pool that keeps handing out a dead address is worse than a smaller pool:

final class ProxyPool
{
    /** @var array<string,int> proxy => unix time it becomes usable again */
    private array $penalty = [];
    private int $i = 0;

    public function __construct(private array $proxies)
    {
        $this->proxies = array_values($proxies);
    }

    public function next(): ?string
    {
        $n = count($this->proxies);
        for ($tries = 0; $tries < $n; $tries++) {
            $proxy = $this->proxies[$this->i++ % $n];
            if (($this->penalty[$proxy] ?? 0) <= time()) {
                return $proxy;
            }
        }
        return null;   // every proxy is cooling down; back off instead of hammering
    }

    public function penalise(string $proxy, int $seconds = 300): void
    {
        $this->penalty[$proxy] = time() + $seconds;
    }
}

Call penalise() on a connection failure, on a 403, and on a 429. Five minutes is a reasonable default; a proxy that earns three penalties in an hour should leave the pool until a human looks at it.

Sticky versus rotating matters more than the marketing suggests. Anything with a session — a login, a multi-step form, a cart — needs the same exit IP for the whole sequence, or the site invalidates the session between requests. Anything stateless wants a fresh IP per request. Mixing the two up produces the most confusing class of scraping bug there is, where a flow works when you test it by hand and fails at volume. The distinction between datacenter, residential and mobile, and how to pick, is covered in our guide to rotating proxies for scraping.

Scraping through Tor

Tor gives you a free SOCKS5 proxy on 127.0.0.1:9050 when you run the daemon, or 127.0.0.1:9150 under Tor Browser. It rotates IPs at no cost, and that is the whole of its appeal.

curl_setopt_array($ch, [
    CURLOPT_PROXY     => '127.0.0.1:9050',
    CURLOPT_PROXYTYPE => CURLPROXY_SOCKS5_HOSTNAME,   // DNS through Tor, not locally
]);

To change exit IP, ask the control port for a new circuit. Enable it in torrc:

ControlPort 9051
CookieAuthentication 0
HashedControlPassword 16:...   # generate with: tor --hash-password "your_password"

Then from PHP:

function torNewIdentity(string $password, string $host = '127.0.0.1', int $port = 9051): bool
{
    $fp = @fsockopen($host, $port, $errno, $errstr, 10);
    if (!$fp) {
        return false;
    }
    fwrite($fp, "AUTHENTICATE \"$password\"\r\n");
    $auth = fgets($fp);                 // expect 250 OK
    fwrite($fp, "SIGNAL NEWNYM\r\n");
    $signal = fgets($fp);               // expect 250 OK
    fclose($fp);

    sleep(10);                          // a 250 is an acknowledgement, not a new circuit
    return str_starts_with($auth, '250') && str_starts_with($signal, '250');
}

A 250 does not mean you have a new IP. The Tor control specification defines NEWNYM as "Switch to clean circuits, so new application requests don't share any circuits with old ones" and adds, in parentheses, "(Tor MAY rate-limit its response to this signal.)" Ask twice in quick succession and the second request can be absorbed silently while your code believes it rotated. The fix is not a longer sleep() — it is verifying. Fetch an IP-echo service through the proxy after signalling, and only proceed when the address actually changed.

The honest verdict on Tor for scraping. Exit node lists are public and widely blocked, throughput is a fraction of a datacenter proxy, and many targets serve a challenge page to Tor traffic on sight, at which point you are into solving CAPTCHAs for no good reason. Compare the failure rate against $2.25/GB for residential traffic and Tor stops looking free. It is a legitimate tool for a few hundred requests against a target that does not care, and a waste of a week at any real volume. There is also a fairness argument: the network is run by volunteers for people who need it, and a price crawler is not that.

HTTPS, certificates, and the option you must not disable

cURL verifies certificates by default, which is correct. Verification failures almost always mean a stale CA bundle on your server, not a problem with the target.

curl_setopt_array($ch, [
    CURLOPT_SSL_VERIFYPEER => true,   // verify the chain
    CURLOPT_SSL_VERIFYHOST => 2,      // verify the hostname matches the certificate
    CURLOPT_CAINFO         => '/path/to/cacert.pem',
]);

The curl project republishes Mozilla's root store as a PEM file at curl.se/docs/caextract.html. The service checks daily and regenerates when the source changes; the bundle available while writing this was stamped 13 August 2026, 03:12:01 GMT, roughly 200 KB. Point php.ini at it once and forget about it:

curl.cainfo = "/path/to/cacert.pem"
openssl.cafile = "/path/to/cacert.pem"

Do not do this:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Beyond the obvious exposure to interception, there is a scraping-specific reason to care. If you run through a proxy vendor and disable verification, you have no way of knowing whether the HTML you received came from the target or from something in the middle. Silent content modification by an intermediary is a real failure mode, and verification is what rules it out. Update the bundle instead.

Forcing a TLS version is available and is a trap:

curl_setopt($ch, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2);

It works, and it makes you more distinctive rather than less. Every browser on the planet offers TLS 1.3 first. Pinning to 1.2 is a signal, and the next section is about signals.

Cookies, and when the jar is actually written

A cookie jar is a file cURL reads before a request and writes after it:

$cookieFile = __DIR__ . '/cookies.txt';

curl_setopt_array($ch, [
    CURLOPT_COOKIEJAR  => $cookieFile,   // where received cookies are SAVED
    CURLOPT_COOKIEFILE => $cookieFile,   // where cookies are READ from
]);

A login followed by a request to a protected page works as long as both share the file:

$cookieFile = tempnam(sys_get_temp_dir(), 'ck');

$ch = curl_init('https://example.com/login');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => http_build_query(['user' => 'me', 'pass' => 'secret']),
    CURLOPT_COOKIEJAR      => $cookieFile,
    CURLOPT_COOKIEFILE     => $cookieFile,
    CURLOPT_FOLLOWLOCATION => true,
]);
curl_exec($ch);
unset($ch);                    // <- the jar is flushed here, not before

$ch = curl_init('https://example.com/account');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEFILE     => $cookieFile,
    CURLOPT_COOKIEJAR      => $cookieFile,
]);
$account = curl_exec($ch);
unset($ch);

That unset() is the point. cURL writes the jar when the easy handle is destroyed. On PHP 5 and 7, curl_close() did that. On PHP 8 curl_close() is a documented no-op, so the file is written when the CurlHandle object's last reference goes away. Keep the handle in a variable, open a second handle, read the jar file — and it is empty, or holds the state from the previous run. The login "silently fails" and the second request comes back as an anonymous visitor. Destroy the handle, or set CURLOPT_COOKIEJAR and let the script end.

You can also pass cookies inline, which is what you want when the values come from a database rather than a file:

curl_setopt($ch, CURLOPT_COOKIE, 'sessionid=abc123; lang=en');

In parallel scraping give every worker and every proxy its own jar file. Two processes writing one cookie file will interleave and corrupt it, and the resulting bug looks like random logouts rather than like a file problem.

Status codes and the headers that matter

A scraper that ignores the response code produces a database full of error pages. The minimum: 200 processes, 404 drops, 403 and 429 back off, 5xx retries later.

$body = curl_exec($ch);
$code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);

match (true) {
    $code === 200            => $this->process($body),
    $code === 404            => $this->drop($url),
    $code === 429            => $this->backOff($url, $retryAfter),
    $code === 403            => $this->rotateProxyAndRequeue($url),
    $code >= 500             => $this->requeueWithDelay($url, 60),
    default                  => $this->log($url, $code),
};

curl_getinfo() without a second argument returns the whole picture, and four of its keys earn their keep:

$info = curl_getinfo($ch);
// $info['http_code']     — response code
// $info['content_type']  — the charset lives here, and it beats the meta tag
// $info['primary_ip']    — which server answered; proves whether the proxy was used
// $info['total_time']    — per-request timing, the input to your concurrency decision

Retry-After is not always a number. This is the header the 429 branch depends on, and RFC 9110 allows two forms: a delay in seconds, or an HTTP-date. Code that does (int) $header turns Wed, 13 Aug 2026 09:00:00 GMT into 0 and retries immediately, which is how a soft rate limit becomes a hard ban:

function retryAfterSeconds(?string $header): ?int
{
    $header = trim((string) $header);
    if ($header === '') {
        return null;
    }
    if (ctype_digit($header)) {
        return (int) $header;
    }
    $ts = strtotime($header);                  // HTTP-date form
    return $ts === false ? null : max(0, $ts - time());
}

Collecting headers as they arrive is cleaner than slicing the response by CURLINFO_HEADER_SIZE:

$headers = [];
curl_setopt($ch, CURLOPT_HEADERFUNCTION, function ($ch, string $line) use (&$headers) {
    $parts = explode(':', $line, 2);
    if (count($parts) === 2) {
        $name = strtolower(trim($parts[0]));
        $headers[$name][] = trim($parts[1]);   // arrays: Set-Cookie repeats
    }
    return strlen($line);                      // returning anything else aborts the transfer
});

Two details in eight lines. The callback must return the byte length of the line, or cURL treats it as an error and kills the transfer. And it fires for every response in a redirect chain, so with CURLOPT_FOLLOWLOCATION on you get the 301's headers and the 200's headers in the same array. Store values as lists and take the last one, or you will read a Content-Type that belongs to a redirect.

Looking like a browser is not about the User-Agent

Here is where the guides written before 2023 quietly stop being true, this one's earlier version included. It advised a realistic User-Agent, an Accept-Language, a Referer and a random pause. All of that is still worth doing and none of it addresses how you are actually identified.

The handshake happens first. Before a single header is sent, your client offers a TLS Client Hello: a specific list of cipher suites, extensions, elliptic curves, and signature algorithms, in a specific order. Hash that and you have a JA3 or JA4 fingerprint. PHP's bundled curl — 8.5.0 on the box used for this article — produces a Client Hello that matches no browser on earth. The curl-impersonate README puts it this way: most clients differ "drastically" from a real browser, and if HTTP/2 is in play "there is also an HTTP/2 handshake where various settings are exchanged", which fingerprints separately.

That is why a request with a perfect Chrome 146 User-Agent still gets a 403 on the first hit. The header says Chrome and the handshake says curl.

What you can do from stock PHP, honestly ranked:

1. Get the cheap signals right anyway. Header order, Accept, Accept-Language, Accept-Encoding, and sec-ch-ua client hints that agree with the User-Agent you claim. This clears the naive filters, which are still the majority of filters.
2. Speak HTTP/2. CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_2TLS. A client claiming to be Chrome while negotiating HTTP/1.1 is contradicting itself.
3. Shape the TLS layer as far as libcurl allows. CURLOPT_SSL_CIPHER_LIST and CURLOPT_TLS13_CIPHERS move the fingerprint. They do not reproduce a browser's, because extension order and the GREASE values are not exposed.
4. Shell out to curl-impersonate. A patched curl built against browser TLS stacks; the maintained fork is at 2.0.0 on curl 8.21.0, with profiles for Chrome 99-146, Firefox 133-147, Safari 15.3-26.0.1, Edge and Tor. From PHP you drive it with proc_open() and lose connection reuse, which is a real cost.
5. Use a real browser, or buy the problem away. Headless Chrome produces a genuine handshake. So does an unblocking API, at a price — see the next section.

Delays and retries still matter. Randomise, and back off on the codes that tell you to:

usleep(random_int(800_000, 2_500_000));   // 0.8-2.5 s, jittered

function fetchWithRetry(string $url, int $maxAttempts = 3): ?string
{
    for ($attempt = 1; $attempt <= $maxAttempts; $attempt++) {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 30,
            CURLOPT_FOLLOWLOCATION => true,
        ]);
        $html = curl_exec($ch);
        $code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
        $net  = curl_errno($ch);
        unset($ch);

        if ($net === 0 && $code === 200) {
            return $html;
        }
        if ($code === 404 || $code === 410) {
            return null;                       // retrying will not help
        }
        sleep(min(2 ** $attempt, 60) + random_int(0, 3));
    }
    return null;
}

The random_int() on the backoff is not decoration. Without jitter, a batch of workers that all hit a 429 at the same second retries at the same second, three times, and the site sees three synchronised spikes.

Cost of getting this wrong, in units you can count. One wasted second per page across a ten-thousand-page crawl is just under three hours. A ban that forces a proxy switch mid-crawl costs the crawl, not the request.

JavaScript pages and headless browsers

cURL fetches source HTML. It does not run JavaScript, so a React or Vue application hands you an empty shell and a bundle reference. Three ways out, in the order you should try them.

First: find the endpoint the page itself calls. Open DevTools, Network, XHR/Fetch, reload. In most single-page applications the data arrives as JSON from an endpoint you can request directly:

$data = json_decode($json, true, 512, JSON_THROW_ON_ERROR);

This is not a shortcut, it is the better implementation. A JSON contract is typed, it usually paginates cleanly, and it changes far less often than markup. The response often carries fields the rendered page never displays. Check for a __NEXT_DATA__ script tag or a JSON-LD block first — plenty of "JavaScript-rendered" pages ship their entire dataset in the HTML and render it client-side.

Second: drive a real browser from PHP. Two live options, both checked on 13 August 2026:

• chrome-php/chrome, v1.16.1 from 6 July 2026, 5.5 million installs. Talks the Chrome DevTools Protocol directly, no WebDriver in the middle. The lighter of the two for pure scraping.
• Symfony Panther, v2.4.0 from 8 January 2026, PHP 8.1+. Built on php-webdriver 1.16.0 (28 December 2025) and the DomCrawler API, so the same filter() calls work against a live browser.

use Symfony\Component\Panther\Client;

$client  = Client::createChromeClient();
$crawler = $client->request('GET', 'https://spa-example.com');

$client->waitFor('.product', 10);         // wait for the selector, never sleep() blindly
$titles = $crawler->filter('.product .title')->each(fn ($n) => $n->text());

$client->quit();                          // leaked Chrome processes are the classic bug

Third: put rendering behind a service. A headless browser is a heavyweight dependency: it needs the binary present, it starts slowly, it holds far more memory than a DOM tree, and it leaks processes when a run dies. At volume, a dedicated rendering service — Playwright driven from another runtime, or a hosted API — is easier to scale and to restart than a fleet of PHP workers each spawning Chrome. Our walkthrough of running headless Firefox on Linux covers the self-hosted version.

What the hosted version costs. ScrapingBee publishes plans at $49/month for 250,000 API credits, $99 for 1,000,000, and $249 for 3,000,000, with 1,000 credits free to trial and no card. At the entry plan that is about $0.20 per thousand credits. The pricing page does not break down how many credits a JavaScript render or a premium proxy consumes, so treat the per-request figure as a floor rather than an estimate. The reason to consider it is not the rendering, which you can do yourself. It is that the fingerprint problem from the previous section becomes somebody else's maintenance burden.

Queues, dedup, and what changes at 100,000 URLs

A script that walks a list of URLs and a crawler are different programs. The crawler needs a frontier of URLs to visit, a record of what it has seen, and the ability to lose a worker without losing work.

What has to be stored:

• the frontier: URLs discovered but not yet fetched;
• the seen set, for deduplication, keyed by a hash rather than the URL itself;
• per-URL state: pending, processing, done, failed, plus an attempt count and the time of the next allowed try;
• the extracted data, which should live somewhere other than the queue table.

A database queue that survives a dead worker

CREATE TABLE crawl_queue (
    id          BIGINT AUTO_INCREMENT PRIMARY KEY,
    url         VARCHAR(2048) NOT NULL,
    url_hash    BINARY(20) NOT NULL,             -- sha1($url, true): 20 bytes, not 40 chars
    status      ENUM('pending','processing','done','failed') NOT NULL DEFAULT 'pending',
    attempts    TINYINT UNSIGNED NOT NULL DEFAULT 0,
    leased_at   TIMESTAMP NULL DEFAULT NULL,
    next_try_at TIMESTAMP NULL DEFAULT NULL,
    created_at  TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uniq_hash (url_hash),
    KEY idx_claim (status, next_try_at, id)
);

sha1($url, true) returns 20 raw bytes. The same digest as a 40-character hex string in a utf8mb4 column can reserve up to 160 bytes per row, and this is the table with the most rows in the system. On a hundred million URLs that difference is measured in gigabytes of index.

Claiming work atomically is what stops two workers fetching the same page:

$pdo->beginTransaction();

$row = $pdo->query(
    "SELECT id, url FROM crawl_queue
      WHERE status = 'pending'
        AND (next_try_at IS NULL OR next_try_at <= NOW())
      ORDER BY id
      LIMIT 1
      FOR UPDATE SKIP LOCKED"
)->fetch();

if ($row) {
    $pdo->prepare(
        "UPDATE crawl_queue
            SET status = 'processing', attempts = attempts + 1, leased_at = NOW()
          WHERE id = ?"
    )->execute([$row['id']]);
}

$pdo->commit();

MySQL's manual describes SKIP LOCKED precisely: "A locking read that uses SKIP LOCKED never waits to acquire a row lock. The query executes immediately, removing locked rows from the result set." It also carries the warning that matters, which is that such queries "return an inconsistent view of the data" and the feature is "not suitable for general transactional work" — while being explicitly recommended "to avoid lock contention when multiple sessions access the same queue-like table." That is our case exactly. PostgreSQL has had the same clause since 9.5.

leased_at is the column people leave out. A worker that dies mid-page leaves its row in processing forever, and the crawl quietly shrinks by one URL per crash. One sweeper query fixes it:

UPDATE crawl_queue
   SET status = 'pending', next_try_at = NOW()
 WHERE status = 'processing'
   AND leased_at < NOW() - INTERVAL 15 MINUTE;

Deduplication that fits in memory

A SELECT per candidate URL against a hundred-million-row table is a round trip you cannot afford in an inner loop. Redis is the usual answer: SADD returns 1 for a new member and 0 for a known one, which is a decision in a single command.

At real scale the set itself becomes the problem, and a Bloom filter replaces it. Redis's own documentation publishes the numbers: 9.585 bits per item at a 1% false-positive rate, 14.378 bits at 0.1%, 19.170 bits at 0.01%, against capacity * (192 bits + value) for a plain set.

Run those figures for 100 million URLs. A Bloom filter at 0.01% costs about 1.92 billion bits, roughly 240 MB. The equivalent set of 40-byte hashes costs 512 bits per entry, about 6.4 GB. Same job, 27x the memory.

BF.RESERVE crawl:seen 0.0001 100000000
BF.ADD     crawl:seen https://example.com/p/12345
BF.EXISTS  crawl:seen https://example.com/p/12345

The trade is one-sided in a specific way: a Bloom filter never says "new" about a URL it has seen, but it will occasionally say "seen" about a URL it has not. At 0.01% you silently skip roughly one page in ten thousand. For a price crawler that is invisible. For a legal archive it is unacceptable. Choose the error rate against the job, not against the memory chart.

Normalise before you hash

https://example.com/p/7, https://example.com/p/7?utm_source=newsletter, and https://EXAMPLE.com/p/7#reviews are one page and three hashes. Left alone, tracking parameters alone can multiply a frontier several times over:

function canonicalise(string $url): string
{
    $p = parse_url($url);
    $host = strtolower($p['host'] ?? '');
    $path = $p['path'] ?? '/';

    parse_str($p['query'] ?? '', $q);
    foreach (array_keys($q) as $k) {
        if (str_starts_with($k, 'utm_') || in_array($k, ['gclid', 'fbclid', 'ref'], true)) {
            unset($q[$k]);
        }
    }
    ksort($q);                                   // parameter order is not meaning

    $scheme = $p['scheme'] ?? 'https';
    $port   = isset($p['port']) && !in_array($p['port'], [80, 443], true) ? ':' . $p['port'] : '';

    return $scheme . '://' . $host . $port . rtrim($path, '/')
         . ($q ? '?' . http_build_query($q) : '');   // fragment deliberately dropped
}

PHP 8.5 ships a built-in uri extension with Uri\Rfc3986\Uri and Uri\WhatWg\Url, offering spec-defined normalisation, resolve() for relative links, and an equals() comparison — the manual has the full API. It is the right replacement for the function above, and it is the first standard-library URL handling PHP has had that agrees with what browsers do. We could not exercise it for this article, because the machine used for the measurements runs 8.4.

Producers and workers

Split the roles. A producer discovers links and writes them to the frontier; workers claim, fetch, parse, and write results. Nothing else needs to be shared. That is what lets you add a machine without changing code, and it is also what lets you throttle politely: rate limits belong on the worker side, keyed by host, so ten workers on ten domains do not become ten workers on one.

Where PHP stops being the right answer

PHP is a good scraping language and a bad choice for some scraping jobs. The line is worth drawing explicitly.

PHP is the obvious answer when the data lands in a PHP application anyway, when the crawl is thousands to low millions of pages, when the target serves server-rendered HTML, and when the team already reads PHP. cURL is excellent, the DOM is built in, and hosting is cheap. Nothing about the language limits you here.

PHP starts costing you when:

• Rendering dominates. If most targets need a real browser, the runtime running the browser matters more than the runtime issuing requests, and Playwright's own ecosystem is in Node and Python.
• Parsing dominates. One core per process, ~50 ms per megabyte-sized page. Ten million pages of that is 140 CPU-hours of parsing alone, and you will be managing a process pool rather than writing extraction code. Scrapy on Python or Colly on Go start ahead here, mostly because of the surrounding tooling rather than raw speed.
• The fight is the product. When targets run serious bot management, the work stops being extraction and becomes fingerprint maintenance: TLS profiles, proxy pools, challenge handling. That is a full-time job in any language. Teams that decide it is not their product either buy the unblocking layer or hand the pipeline to a managed extraction service and consume the output.
• Long-lived processes leak. PHP was built to start, serve, and exit. Workers that run for days need a restart policy — after N pages or M megabytes of RSS — because libxml allocations, as measured above, are invisible to memory_limit.

Where PHP is genuinely strong and gets no credit: the queue-and-worker part. A PDO queue with SKIP LOCKED, a supervisor, and twenty processes is boring, debuggable infrastructure that a junior can operate. The exotic runtimes buy you concurrency you may not need and cost you a team that can maintain it.

What a production PHP scraper actually contains

The parts, in the order you will need them:

• fetching through cURL with both timeouts set, bounded redirects, CURLOPT_ACCEPT_ENCODING => '', and no curl_close();
• a status-code branch: 200 processes, 404 drops, 403 rotates, 429 honours Retry-After in both its formats, 5xx requeues;
• encoding normalised to UTF-8 in the fetcher, from the Content-Type header first and the meta tag second;
• parsing through Dom\HTMLDocument with querySelectorAll on PHP 8.4+, or DOMXPath on older branches, never through a regex on markup;
• a cookie jar per worker, with the handle destroyed so the jar is written;
• curl_multi with the -1 case handled and results streamed to a callback rather than accumulated;
• proxies with penalty tracking, sticky sessions where state exists, rotating elsewhere;
• a queue with atomic claiming, a lease timeout, an attempt counter, and jittered backoff;
• deduplication on a normalised, hashed URL;
• restart policy and RSS monitoring for every long-lived worker.

Four rules hold the whole thing together. Separate fetching from parsing, so transport changes cost nothing. Fix encoding once, before parsing, so mojibake stops being a category. Look for JSON before you look for a selector, because the endpoint is more stable than the markup. Respect the other server — one host at a time, a real delay, robots.txt read by a library that implements RFC 9309 rather than by twenty lines of your own.

The upkeep is the part that surprises people. Proxies rot, layouts change, and anti-bot systems ship updates you find out about from a monitoring alert. A crawler is not a script you finish; it is a system with an operations cost, and the decision worth making early is whether that cost belongs to your team, whether it is handed to a managed web scraping service, or whether the cleaner answer is to consume the result as data as a service and spend the engineering time on what you do with the data instead.