Web Scraping with PHP: The Complete Guide

This is a practical, example-driven guide to web scraping using PHP — from a one-line page download to concurrency, proxies, Tor, and job queues. Every snippet is working code you can copy and adapt to your own task. If PHP is already your stack, there is no reason to reach for another language just to pull data off the web: the language ships with an excellent HTTP client (cURL) and a solid DOM parser out of the box.

Table of contents

1. What web scraping is and when you need it
2. Ethics and the legal side (robots.txt, load)
3. Fetching the page — file_get_contents, cURL, Guzzle
4. Libraries for parsing content — why not regex, DOMDocument + DOMXPath, Symfony DomCrawler, Simple HTML DOM / phpQuery, when to parse JSON / a hidden API
5. Fixing character-encoding problems
6. multi_curl and concurrency
7. Using proxies
8. Scraping through Tor
9. Working with HTTPS / SSL
10. Working with cookies
11. Response status and other headers
12. Looking like a browser: delays and retries
13. JavaScript pages and headless browsers
14. Storing URLs and queues
15. Pros and cons of a PHP implementation
16. Conclusion

1. What web scraping is and when you need it {#what-scraping-is}

Web scraping is the automated retrieval of web pages and the extraction of structured data from them: prices, product descriptions, contact details, news. The process almost always breaks down into two steps:

1. Download the HTML page (an HTTP request).
2. Parse it and pull out the pieces you need (HTML/DOM parsing).

Keep those two steps separate: a "fetcher" and a "parser". That way you can change how you download (cURL → proxy → Tor) independently of your extraction logic. This separation is the single most important architectural decision in a PHP web scraper, and it pays off the moment a target site starts blocking you or you need to swap in a different transport.

Before you write a scraper, always check one thing first: does the site expose an open API or a JSON endpoint? Parsing ready-made JSON is ten times simpler and more reliable than clawing data out of markup that changes every week.

2. Ethics and the legal side {#ethics}

Before you start hammering someone else's server, keep a few things in mind:

• robots.txt — the file where a site declares what may be crawled. It does not legally block access, but respecting it is good manners and is sometimes referenced in the terms of service.
• Server load. Do not fire hundreds of requests per second — that looks like a denial-of-service attack. Put pauses between requests (see section 12).
• Copyright and personal data. Collecting and republishing content can violate the law. Be especially careful with personal data — under GDPR (EU/UK) and CCPA/CPRA (California) the bar is high.
• Terms of service (ToS). Many sites explicitly forbid automated collection. In the US, the hiQ v. LinkedIn line of cases and the Computer Fraud and Abuse Act (CFAA) shaped how courts think about scraping public data — the short version is "scraping public pages is generally not a crime, but breaching ToS or bypassing access controls can still get you sued or blocked."

A minimal robots.txt reader:

function isAllowed(string $url, string $userAgent = '*'): bool
{
    $parts = parse_url($url);
    $robotsUrl = $parts['scheme'] . '://' . $parts['host'] . '/robots.txt';
    $robots = @file_get_contents($robotsUrl);
    if ($robots === false) {
        return true; // no robots.txt — nothing formally disallowed
    }
    // Simplified check: look for a Disallow that matches our path.
    $path = $parts['path'] ?? '/';
    foreach (preg_split('/\R/', $robots) as $line) {
        if (preg_match('/^\s*Disallow:\s*(\S+)/i', $line, $m)) {
            if ($m[1] !== '' && str_starts_with($path, $m[1])) {
                return false;
            }
        }
    }
    return true;
}

For anything serious, use a proper robots.txt parser (for example spatie/robots-txt) rather than a hand-rolled one — real robots files have wildcards, Allow rules, and user-agent groups that the snippet above ignores.

3. Fetching the page {#fetching}

3.1. file_get_contents — the simplest way

$html = file_get_contents('https://example.com');

This works if allow_url_fopen is enabled in php.ini. You can pass a context with headers:

$context = stream_context_create([
    'http' => [
        'method'  => 'GET',
        'header'  => "User-Agent: Mozilla/5.0\r\n",
        'timeout' => 10,
    ],
]);
$html = file_get_contents('https://example.com', false, $context);

Downsides: no real cookie support, no proxy support, no response codes out of the box, poor error handling. Fine for a throwaway script, not for a production scraper.

3.2. cURL — the workhorse

cURL is an extension that ships almost everywhere and gives you full control over the request. It is the main tool for PHP web scraping.

function fetch(string $url): string
{
    $ch = curl_init();
    curl_setopt_array($ch, [
        CURLOPT_URL            => $url,
        CURLOPT_RETURNTRANSFER => true,   // return the result as a string instead of printing it
        CURLOPT_FOLLOWLOCATION => true,   // follow redirects
        CURLOPT_MAXREDIRS      => 5,
        CURLOPT_TIMEOUT        => 30,     // overall timeout
        CURLOPT_CONNECTTIMEOUT => 10,     // connection timeout
        CURLOPT_USERAGENT      => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                                . 'AppleWebKit/537.36 (KHTML, like Gecko) '
                                . 'Chrome/124.0 Safari/537.36',
        CURLOPT_ENCODING       => '',     // accept gzip/deflate/br and decompress automatically
    ]);

    $html = curl_exec($ch);

    if ($html === false) {
        $error = curl_error($ch);
        curl_close($ch);
        throw new RuntimeException("cURL error: $error");
    }

    curl_close($ch);
    return $html;
}

Key options:

3.3. Guzzle — a modern HTTP client

If your project uses Composer, Guzzle is more comfortable. It is a wrapper over cURL with a human-friendly API, async support, middleware, a cookie jar, and so on.

use GuzzleHttp\Client;

$client = new Client([
    'timeout' => 30,
    'headers' => ['User-Agent' => 'Mozilla/5.0 ...'],
]);

$response = $client->get('https://example.com');
$html = (string) $response->getBody();
$status = $response->getStatusCode();

The examples below use "bare" cURL so you can see the mechanics, but in a real project Guzzle often saves time. A modern alternative is Symfony's HttpClient, which also supports HTTP/2 and native async.

4. Libraries for parsing content {#parsing-libraries}

4.1. Why not regex

The temptation to parse HTML with a regular expression is strong, but HTML is not a regular language. Any nesting, unclosed tag, or line break will break your regex. Regular expressions are only appropriate for very simple, flat fragments (pulling a number out of a string, say), not for parsing a document tree.

4.2. DOMDocument + DOMXPath (built into PHP)

The most reliable built-in approach. Load HTML into a DOM and walk it with XPath.

$dom = new DOMDocument();
libxml_use_internal_errors(true);           // silence warnings about "broken" HTML
$dom->loadHTML($html);
libxml_clear_errors();

$xpath = new DOMXPath($dom);

// Every h2 inside a block with class "article"
$nodes = $xpath->query('//div[@class="article"]//h2');
foreach ($nodes as $node) {
    echo trim($node->textContent), PHP_EOL;
}

// Get href from links
$links = $xpath->query('//a/@href');
foreach ($links as $link) {
    echo $link->value, PHP_EOL;
}

Useful XPath expressions:

On PHP 8.4+ you can also use the new Dom\HTMLDocument class, a spec-compliant HTML5 parser that handles modern markup far better than the old libxml-based DOMDocument. If you are on 8.4, prefer it.

4.3. Symfony DomCrawler (via Composer) — recommended

A convenient wrapper over the DOM with support for both CSS selectors and XPath.

composer require symfony/dom-crawler symfony/css-selector

use Symfony\Component\DomCrawler\Crawler;

$crawler = new Crawler($html);

// CSS selectors (requires css-selector)
$crawler->filter('div.article h2')->each(function (Crawler $node) {
    echo $node->text(), PHP_EOL;
});

// Attributes
$title = $crawler->filter('meta[property="og:title"]')->attr('content');

// XPath is available too
$crawler->filterXPath('//a')->each(fn(Crawler $a) => print($a->attr('href') . "\n"));

4.4. Simple HTML DOM and phpQuery

• Simple HTML DOM (simple_html_dom) — an old, very simple library with jQuery-like syntax. Convenient, but memory-hungry and barely maintained. Fine for small jobs.
• phpQuery — a port of jQuery to PHP. Also dated, but the familiar pq('div.item')->find('a') syntax appeals to some.

For new projects prefer DomCrawler or DOMXPath — they are faster and actively supported.

4.5. A hidden JSON / API — the cleanest path

Open DevTools → the Network tab. Data is often loaded by a separate XHR/fetch request that returns ready-made JSON. Parsing it is a single line:

$data = json_decode($jsonString, true);

This is more reliable than any HTML parsing: a JSON structure changes far less often than markup. Finding and calling a site's hidden API should be your first move on any modern, JavaScript-heavy site.

5. Fixing character-encoding problems {#encoding}

The single most common headache in web scraping with PHP is garbled text (mojibake) instead of readable characters. Most of the web is UTF-8 today, but you will still hit legacy pages served as Windows-1252, ISO-8859-1 (Latin-1), Shift-JIS (Japanese), or GB2312 (Chinese). The two root causes are an encoding mismatch (the site is in Windows-1252, you expect UTF-8) and the fact that DOMDocument assumes UTF-8 input by default.

5.1. Detecting the page encoding

The site announces its encoding in the HTTP Content-Type header or in a <meta charset> tag.

function detectCharset(string $html, ?string $contentTypeHeader = null): string
{
    if ($contentTypeHeader && preg_match('/charset=([\w-]+)/i', $contentTypeHeader, $m)) {
        return strtoupper($m[1]);
    }
    if (preg_match('/<meta[^>]+charset=["\']?([\w-]+)/i', $html, $m)) {
        return strtoupper($m[1]);
    }
    // Heuristic as a fallback
    return mb_detect_encoding($html, ['UTF-8', 'Windows-1252', 'ISO-8859-1'], true) ?: 'UTF-8';
}

5.2. Converting to UTF-8

$charset = detectCharset($html, $contentType);
if ($charset !== 'UTF-8') {
    $html = mb_convert_encoding($html, 'UTF-8', $charset);
    // and rewrite the meta declaration so the DOM parser is not confused
    $html = preg_replace('/charset=[\w-]+/i', 'charset=UTF-8', $html, 1);
}

5.3. The key trick for DOMDocument

DOMDocument::loadHTML guesses the encoding from the content and often gets it wrong. The most reliable fix is to add a "hint" before loading:

$dom = new DOMDocument();
libxml_use_internal_errors(true);
// force the parser to treat the input as UTF-8
$dom->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();

or the flag-based variant (PHP 8.1+ adds nothing extra):

$dom->loadHTML($html, LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD);

Rule of thumb: convert the whole HTML to UTF-8 during the download stage, and only then hand it to the parser. Do the encoding fix once, up front, and non-UTF-8 pages stop breaking your extraction.

6. multi_curl and concurrency {#multi-curl}

PHP is single-threaded by nature, but cURL can run several requests in parallel through curl_multi_*. This is a huge speed win: while one server is "thinking", others are downloading.

function fetchMany(array $urls, int $concurrency = 10): array
{
    $multi = curl_multi_init();
    $results = [];
    $queue = array_values($urls);
    $active = [];

    // helper to add one request
    $addHandle = function (string $url) use ($multi, &$active) {
        $ch = curl_init();
        curl_setopt_array($ch, [
            CURLOPT_URL            => $url,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_TIMEOUT        => 30,
            CURLOPT_ENCODING       => '',
        ]);
        curl_multi_add_handle($multi, $ch);
        $active[(int) $ch] = $url;
        return $ch;
    };

    // first batch
    for ($i = 0; $i < $concurrency && $queue; $i++) {
        $addHandle(array_shift($queue));
    }

    do {
        curl_multi_exec($multi, $running);
        curl_multi_select($multi);   // wait for events instead of spinning the CPU

        // collect finished handles
        while ($done = curl_multi_info_read($multi)) {
            $ch = $done['handle'];
            $url = $active[(int) $ch];
            $results[$url] = curl_multi_getcontent($ch);

            curl_multi_remove_handle($multi, $ch);
            curl_close($ch);
            unset($active[(int) $ch]);

            // feed the next URL from the queue
            if ($queue) {
                $addHandle(array_shift($queue));
            }
        }
    } while ($running || $queue || $active);

    curl_multi_close($multi);
    return $results;
}

$pages = fetchMany([
    'https://example.com/1',
    'https://example.com/2',
    'https://example.com/3',
], concurrency: 5);

The core idea is a sliding window: keep no more than concurrency requests in flight at once, and add a new one as each finishes. This keeps you fast without opening a thousand connections at once.

Alternatives for "real" parallelism:

• Guzzle Pool / Promises — async requests with a concurrency limit, higher-level than curl_multi.
• ReactPHP / Amp / Swoole / Fibers (PHP 8.1+) — async/coroutine runtimes when you need serious scale.
• pcntl_fork / parallel workers — several processes, each pulling its own batch of URLs from a queue (see section 14).

7. Using proxies {#proxies}

Proxies are needed to:

• get around IP-based blocks (a site bans you for frequent requests),
• collect data from different regions,
• distribute load across addresses.

curl_setopt_array($ch, [
    CURLOPT_PROXY     => '123.45.67.89:8080',
    CURLOPT_PROXYTYPE => CURLPROXY_HTTP,     // or CURLPROXY_SOCKS5
]);

// proxy with authentication
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'login:password');

Proxy types:

Proxy rotation. Keep a pool of addresses and hand them out round-robin; mark "dead" ones and exclude them temporarily.

class ProxyPool
{
    private array $proxies;
    private int $i = 0;

    public function __construct(array $proxies)
    {
        $this->proxies = array_values($proxies);
    }

    public function next(): string
    {
        $proxy = $this->proxies[$this->i % count($this->proxies)];
        $this->i++;
        return $proxy;
    }
}

There is a difference between datacenter proxies (cheap, easily detected) and residential/mobile ones (pricier, but they look like real users). The right choice depends on how aggressively the site defends itself. For a deeper dive, see our guide to rotating proxies for scraping.

8. Scraping through Tor {#tor}

Tor is a free network that gives you an anonymous SOCKS5 proxy at 127.0.0.1:9050. It is handy for free IP rotation, but it is slow and many sites block Tor exit nodes.

Connecting

curl_setopt_array($ch, [
    CURLOPT_PROXY     => '127.0.0.1:9050',
    CURLOPT_PROXYTYPE => CURLPROXY_SOCKS5_HOSTNAME, // DNS through Tor — important for anonymity
]);

Changing IP (a new circuit)

Tor has a control port (9051) through which you can request a new circuit with the NEWNYM signal. First enable it in torrc:

ControlPort 9051
CookieAuthentication 0
HashedControlPassword 16:...   # generate with: tor --hash-password "your_password"

Then from PHP:

function torNewIdentity(string $password, string $host = '127.0.0.1', int $port = 9051): bool
{
    $fp = @fsockopen($host, $port, $errno, $errstr, 10);
    if (!$fp) {
        return false;
    }
    fwrite($fp, "AUTHENTICATE \"$password\"\r\n");
    $auth = fgets($fp);                 // wait for 250 OK
    fwrite($fp, "SIGNAL NEWNYM\r\n");
    $signal = fgets($fp);               // 250 OK
    fclose($fp);

    sleep(5); // Tor does not build a new circuit instantly
    return str_starts_with($auth, '250') && str_starts_with($signal, '250');
}

A typical loop: make N requests → torNewIdentity() → continue with the new IP.

Downsides of Tor: low speed, some sites serve a CAPTCHA or a 403 immediately, and there is a limited number of exit nodes. For serious volume, paid proxies win. And if you do hit challenge pages, you will need a strategy for solving CAPTCHAs.

9. Working with HTTPS / SSL {#ssl}

By default cURL verifies the SSL certificate — and that is correct. Problems arise when the server has an outdated set of root certificates (CA bundle).

curl_setopt_array($ch, [
    CURLOPT_SSL_VERIFYPEER => true,   // verify the certificate (do NOT disable without reason)
    CURLOPT_SSL_VERIFYHOST => 2,      // verify the host matches the certificate
    CURLOPT_CAINFO         => '/path/to/cacert.pem', // a fresh CA bundle
]);

Get a fresh cacert.pem from curl.se/docs/caextract.html and register it in php.ini:

curl.cainfo = "/path/to/cacert.pem"
openssl.cafile = "/path/to/cacert.pem"

Do not do this in production:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // disables protection against MITM

This removes certificate verification. It is only acceptable temporarily, for local debugging. The correct fix for a "certificate error" is to update the CA bundle, not to disable verification.

You can also force a TLS version:

curl_setopt($ch, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2);

10. Working with cookies {#cookies}

Cookies are needed for sessions, authentication, and getting past "checks" that set a cookie and expect it back on the next request. cURL can store and re-send them automatically through a cookie jar — a file.

$cookieFile = __DIR__ . '/cookies.txt';

curl_setopt_array($ch, [
    CURLOPT_COOKIEJAR  => $cookieFile,  // where to SAVE received cookies
    CURLOPT_COOKIEFILE => $cookieFile,  // where to READ them from on request
]);

If both requests (the login and the follow-up) use the same $cookieFile, the session persists between them.

An authentication example

$cookieFile = tempnam(sys_get_temp_dir(), 'ck');

// 1) POST with login/password — the server returns a session cookie
$ch = curl_init('https://example.com/login');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => http_build_query(['user' => 'me', 'pass' => 'secret']),
    CURLOPT_COOKIEJAR      => $cookieFile,
    CURLOPT_COOKIEFILE     => $cookieFile,
    CURLOPT_FOLLOWLOCATION => true,
]);
curl_exec($ch);
curl_close($ch);

// 2) request to a protected page — the cookie is attached automatically
$ch = curl_init('https://example.com/account');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEFILE     => $cookieFile,
    CURLOPT_COOKIEJAR      => $cookieFile,
]);
$account = curl_exec($ch);
curl_close($ch);

You can also pass a cookie manually (without a file):

curl_setopt($ch, CURLOPT_COOKIE, 'sessionid=abc123; lang=en');

In multi-threaded scraping, give each worker/proxy its own cookie file, or the sessions will get mixed up.

11. Response status and other headers {#status}

A scraper must react to the response code: 200 — OK, 404 — page not found, 403/429 — banned or asked to slow down, 5xx — server error.

The response code

$html = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if ($httpCode === 200) {
    // process
} elseif ($httpCode === 429) {
    // too frequent — wait and retry
} elseif ($httpCode >= 500) {
    // server error — retry later
}

Useful info from curl_getinfo

$info = curl_getinfo($ch);
// $info['http_code']       — response code
// $info['content_type']    — Content-Type (charset lives here too!)
// $info['redirect_url']    — where it redirected
// $info['total_time']      — how long it took
// $info['primary_ip']      — the server's IP (useful when checking a proxy)
// $info['size_download']   — response size

Getting response headers separately

curl_setopt($ch, CURLOPT_HEADER, true); // include headers in the output
$response = curl_exec($ch);

$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$rawHeaders = substr($response, 0, $headerSize);
$body       = substr($response, $headerSize);

More cleanly — via a callback that collects headers into an array:

$headers = [];
curl_setopt($ch, CURLOPT_HEADERFUNCTION, function ($ch, $line) use (&$headers) {
    $parts = explode(':', $line, 2);
    if (count($parts) === 2) {
        $headers[strtolower(trim($parts[0]))] = trim($parts[1]);
    }
    return strlen($line); // you MUST return the length
});
curl_exec($ch);
// now $headers['content-type'], $headers['set-cookie'], etc.

The ones that matter most: Content-Type (encoding), Set-Cookie, Location (redirect), Retry-After (how long to wait on a 429), Content-Length.

12. Looking like a browser: delays and retries {#browser-mask}

To keep your scraper from being banned on the second request, it must behave like a human.

Realistic headers

curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
        . '(KHTML, like Gecko) Chrome/124.0 Safari/537.36',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.9',
    'Referer: https://example.com/',
    'Connection: keep-alive',
]);

Delays between requests

usleep(random_int(800_000, 2_500_000)); // random pause of 0.8–2.5 seconds

Random pauses look more natural than fixed ones. This is both polite to the server and lowers your ban risk.

Retries with exponential backoff

function fetchWithRetry(string $url, int $maxAttempts = 3): ?string
{
    for ($attempt = 1; $attempt <= $maxAttempts; $attempt++) {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 30,
            CURLOPT_FOLLOWLOCATION => true,
        ]);
        $html = curl_exec($ch);
        $code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if ($html !== false && $code === 200) {
            return $html;
        }
        if ($code === 404) {
            return null; // no point retrying
        }
        sleep(2 ** $attempt); // 2, 4, 8 seconds...
    }
    return null;
}

What else helps you avoid a ban

• Rotate user agents and proxies.
• Persist cookies between requests (like a real browser).
• Respect Retry-After on a 429.
• Parallelize moderately (not hundreds of threads on one domain).

13. JavaScript pages and headless browsers {#javascript}

cURL retrieves the source HTML but does not execute JavaScript. If the content is rendered on the client (a React/Vue SPA), it will not be in the HTML — you will see empty blocks.

Options:

1. Find the hidden API (section 4.5) — almost always the best way out: an SPA pulls its data from a JSON endpoint you can hit directly.
2. A headless browser — spin up a real engine that runs the JS: - Symfony Panther — a PHP wrapper over ChromeDriver/Selenium. - php-webdriver + Selenium/Chrome. - Pairing with Puppeteer/Playwright (Node.js) — sometimes it is simpler to move rendering into a separate microservice.

// Example with Symfony Panther
use Symfony\Component\Panther\Client;

$client = Client::createChromeClient();
$crawler = $client->request('GET', 'https://spa-example.com');
$client->waitFor('.product');         // wait until the JS renders
$titles = $crawler->filter('.product .title')->each(fn($n) => $n->text());

Headless browsers are heavy and slow — use them only when there is no way around JS. If you need to scale JS rendering, running Playwright as a dedicated rendering service tends to scale better than driving Chrome from PHP.

14. Storing URLs and queues {#queues}

When a scraper crawls hundreds of thousands of pages, you need a URL queue and a record of what has already been processed. The main approaches, briefly:

What to store

• the queue of URLs to process (the frontier);
• the set of already-visited URLs (so you do not fetch twice) — for deduplication it is convenient to store a hash of the URL;
• the status of each URL: pending / processing / done / failed / attempt count;
• the results themselves (the parsed data).

The simple option — a database

CREATE TABLE crawl_queue (
    id          BIGINT AUTO_INCREMENT PRIMARY KEY,
    url         VARCHAR(2048) NOT NULL,
    url_hash    CHAR(40) NOT NULL,           -- sha1(url), for uniqueness
    status      ENUM('pending','processing','done','failed') DEFAULT 'pending',
    attempts    INT DEFAULT 0,
    created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uniq_hash (url_hash),
    KEY idx_status (status)
);

A worker "claims" a task atomically so two processes do not grab the same URL:

$pdo->beginTransaction();
$row = $pdo->query(
    "SELECT id, url FROM crawl_queue
     WHERE status='pending' ORDER BY id LIMIT 1 FOR UPDATE SKIP LOCKED"
)->fetch();

if ($row) {
    $pdo->prepare("UPDATE crawl_queue SET status='processing', attempts=attempts+1 WHERE id=?")
        ->execute([$row['id']]);
}
$pdo->commit();

FOR UPDATE SKIP LOCKED (MySQL 8+/PostgreSQL) is the key to safely handing out tasks to multiple workers.

When you need more scale

• Redis (LPUSH/BRPOP lists, SADD sets for deduplication) — a very fast queue and a popular choice.
• RabbitMQ / Kafka / Beanstalkd — full-blown message brokers when you have many workers and need reliable delivery.
• A Bloom filter — a compact "have we seen this URL?" check across billions of addresses without storing every string.

The architectural principle

Separate the roles: a producer finds new links and puts them in the queue, and workers consume the queue in parallel and write the results. That way the system scales horizontally — you just add more workers.

15. Pros and cons of a PHP implementation {#pros-cons}

Pros

• Low barrier to entry — cURL and DOM are built in, and the runtime is available almost everywhere.
• Excellent cURL — flexible handling of proxies, cookies, SSL, and headers.
• Mature libraries — Guzzle, Symfony DomCrawler/Panther, ready-made queues.
• Easy to embed in an existing PHP project (a CMS or admin panel) — the scraper writes straight into the same database.
• Cheap to deploy — PHP hosting is ubiquitous and inexpensive.

Cons

• No true multithreading out of the box. Parallelism means curl_multi, multiple processes, or async runtimes (ReactPHP/Amp/Swoole/Fibers). It is fiddlier than threads in Go or async in Python.
• JS is not executed — SPAs need a headless browser, which is heavy and slow.
• Memory. Old libraries (Simple HTML DOM) are wasteful; on large volumes, watch for leaks in long-lived workers.
• Speed. For extreme scale, specialized stacks (Scrapy on Python, Colly on Go) are often more efficient and have more ready-made tooling.
• Fragility. Like any scraper, it breaks when the site's markup changes — not PHP-specific, but worth remembering.

Bottom line: PHP is a great choice for most small- to medium-scale scraping tasks, especially when the data needs to land directly in a PHP project. For very large crawlers and heavy JS rendering, look at specialized stacks or offload rendering to a separate service.

16. Conclusion {#conclusion}

A minimal production-grade web scraper in PHP includes:

• fetching via cURL with timeouts, redirects, and CURLOPT_ENCODING => '';
• a realistic User-Agent and headers;
• normalizing the encoding to UTF-8 before parsing (the mojibake cure);
• parsing via DOMXPath or Symfony DomCrawler (not regex);
• checking the HTTP code and handling 404/403/429/5xx;
• pauses between requests and retries with backoff;
• a cookie jar when you need authentication/sessions;
• proxies / IP rotation when you hit blocks (or Tor for a free option);
• curl_multi/workers for speed on large volumes;
• a URL queue with deduplication for serious crawling.

The main rules: separate downloading from parsing, always fix the encoding before parsing, respect the other server (pauses, robots.txt, no DDoS), and look for ready-made JSON/API first before wrestling with markup.

Would rather skip building and maintaining the pipeline? Scraping is easy to start and surprisingly hard to keep running at scale — proxies rot, layouts change, and anti-bot systems evolve. scraping.pro runs this as a done-for-you data extraction service, delivering clean, structured data via API or data as a service so your team consumes results instead of babysitting a crawler.