PHP Web Scraping with cURL: From Basics to Anti-Bot Realities

Web scraping is the art of pulling data out of pages that were never designed to hand it to you cleanly. In PHP, the most reliable workhorse for this is cURL — the Client URL library. This guide walks from a bare-bones request up to the point where simple scrapers stop working, so you understand not only how to scrape but why certain techniques exist.

We'll go step by step:

1. A basic scraper
2. Parsing the content — choosing a library
3. Faking a browser with a User-Agent and headers
4. Storing and sending cookies
5. Running requests in parallel with multi cURL
6. Reading the response code and other response metadata
7. Routing through a proxy (and rotating proxies)
8. Scraping through TOR
9. Ignoring SSL verification to scrape HTTPS
10. Calling the API directly with POST requests
11. Why fast cURL scrapers get banned — and the advanced tools that don't

Why cURL Instead of file_get_contents()?

cURL is part of libcurl, a library that speaks many protocols — HTTP, HTTPS, FTP and more. Compared with a quick file_get_contents(), cURL gives you real control over headers, cookies, proxies, timeouts, redirects, and error handling. That control is exactly what separates a toy script from something that survives contact with a real website.

If the cURL extension isn't installed, enable php_curl in your php.ini on Windows, or install php-curl via your package manager on Linux (apt install php-curl, then restart your web server).

1. A Basic Scraper

Setting Up cURL

First, initialize a cURL handle pointed at the target URL:

$curl = curl_init('https://testwebsite.com/textlist');

By default cURL prints the response straight to the output buffer. We almost never want that — we want the page as a string we can process. Set CURLOPT_RETURNTRANSFER to true:

curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

Executing the Request and Checking for Errors

Now run the request and check for transport-level errors (DNS failure, timeout, refused connection, etc.):

$page = curl_exec($curl);

if (curl_errno($curl)) {
    echo 'Scraper error: ' . curl_error($curl);
    exit;
}

curl_errno() catches problems reaching the server. It does not catch HTTP errors like 404 or 403 — for those you need the response code, covered in section 6.

Closing the Connection

curl_close($curl);

The Whole Basic Scraper

<?php
$curl = curl_init('https://testwebsite.com/textlist');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

$page = curl_exec($curl);

if (curl_errno($curl)) {
    echo 'Scraper error: ' . curl_error($curl);
    exit;
}

curl_close($curl);

// $page now holds the raw HTML, ready to be parsed (see next section)

This already works on simple, unprotected pages. Now we need to extract the data from the HTML — and how you do that matters.

2. Parsing the Content: Choosing a Library

A quick preg_match() can pull out one simple, predictable snippet, but HTML is not a regular language. Nested tags, attributes in any order, optional closing tags, and stray whitespace will break regex patterns fast. For anything beyond the most trivial extraction, use a real DOM/HTML parser. There are two families: parsers built into PHP (documented on php.net) and third-party libraries (installed via Composer, hosted on GitHub/their own sites — not on php.net).

Built-in PHP parsers (documented on php.net)

• DOM extension — DOMDocument + DOMXPath — the classic, always-available option. XML-centric and strict about malformed HTML, so older HTML often needs @ to suppress warnings.
• https://www.php.net/manual/en/book.dom.php
• https://www.php.net/manual/en/class.domdocument.php
• https://www.php.net/manual/en/class.domxpath.php
• New HTML5 DOM API — Dom\HTMLDocument (PHP 8.4+) — a standards-compliant HTML5 parser (built on Lexbor) with browser-style querySelector() / querySelectorAll(), so you usually don't need XPath anymore. This is the modern recommendation if you're on PHP 8.4+.
• https://www.php.net/manual/en/book.dom.php
• https://www.php.net/releases/8.4/en.php
• SimpleXML — easiest API for well-formed XML (RSS/Atom feeds, sitemaps, APIs returning XML).
• https://www.php.net/manual/en/book.simplexml.php
• XMLReader — a streaming (pull) parser for very large documents you don't want fully in memory.
• https://www.php.net/manual/en/book.xmlreader.php
• Tidy — not a scraper itself, but cleans up broken HTML before you parse it.
• https://www.php.net/manual/en/book.tidy.php
• libxml — the underlying library; useful for controlling parse error reporting.
• https://www.php.net/manual/en/book.libxml.php

Third-party libraries (Composer / GitHub — not on php.net)

• Simple HTML DOM Parser (simplehtmldom) — pure-PHP, jQuery-like find() selectors; very beginner-friendly. This is the well-known simple_html_dom library.
• Official site & docs: https://simplehtmldom.sourceforge.io/
• Source mirror: https://github.com/simplehtmldom/simplehtmldom
• Composer-friendly forks: voku/simple_html_dom (https://github.com/voku/simple_html_dom), sunra/php-simple-html-dom-parser (https://github.com/sunra/php-simple-html-dom-parser)
• Symfony DomCrawler — robust, well-maintained, supports both CSS selectors and XPath; great for serious projects.
• https://symfony.com/doc/current/components/dom_crawler.html
• paquettg/php-html-parser — modern, CSS-selector based, installs via Composer.
• https://github.com/paquettg/php-html-parser
• Masterminds HTML5 (html5-php) — a full HTML5 parser that produces a standard DOM tree.
• https://github.com/Masterminds/html5-php
• QueryPath — jQuery-style chained API over the DOM.
• https://github.com/technosophos/querypath

Example: new HTML5 DOM API (PHP 8.4+)

$dom = \Dom\HTMLDocument::createFromString($page, LIBXML_NOERROR);

// CSS selectors, just like in the browser
$node = $dom->querySelector('#case_textlist');

echo $node ? $node->textContent : 'Not found';

Example: Simple HTML DOM Parser (simple_html_dom)

require_once 'simple_html_dom.php';

$html = str_get_html($page);

// jQuery-like selectors
foreach ($html->find('#case_textlist a') as $link) {
    echo $link->plaintext, ' -> ', $link->href, PHP_EOL;
}

$html->clear();
unset($html);

Pick built-in Dom\HTMLDocument on modern PHP, DOMDocument+DOMXPath for wide compatibility, or simple_html_dom/Symfony DomCrawler if you prefer a higher-level, jQuery-style API.

3. Look Like a Browser: User-Agent and Headers

Out of the box, cURL sends almost no identifying headers, and the ones it does send scream "I am a script." Many sites inspect the User-Agent and other headers and quietly serve you a block page, a captcha, or empty data.

The first and cheapest disguise is a realistic User-Agent:

curl_setopt($curl, CURLOPT_USERAGENT,
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' .
    '(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36');

But a real browser sends a whole bundle of headers together. Sending a Chrome User-Agent while omitting the headers Chrome always sends is itself a tell. Add a believable, consistent set:

curl_setopt($curl, CURLOPT_HTTPHEADER, [
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.9',
    'Accept-Encoding: gzip, deflate, br',
    'Connection: keep-alive',
    'Upgrade-Insecure-Requests: 1',
]);

// Let cURL transparently decode gzip/deflate/br responses
curl_setopt($curl, CURLOPT_ENCODING, '');

// Follow 301/302 redirects automatically
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_MAXREDIRS, 5);

// Always set sane timeouts so a stalled request can't hang your scraper
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($curl, CURLOPT_TIMEOUT, 30);

Two important details:

• If you advertise Accept-Encoding: gzip, you must let cURL decode it. Setting CURLOPT_ENCODING to an empty string tells cURL to accept every encoding it supports and unpack the response for you. Forget this and you'll be parsing binary garbage.
• A consistent header set matters more than any single header. The goal isn't one perfect value; it's a request that, taken as a whole, looks like it came from a browser.

4. Storing and Sending Cookies

Most sites use cookies to track sessions: login state, language, a "you've passed our check" flag, a CSRF token, and so on. A scraper that throws away cookies after every request will keep landing on the same gate.

cURL can persist cookies to a file (a "cookie jar") and replay them on later requests:

$cookieFile = __DIR__ . '/cookies.txt';

curl_setopt($curl, CURLOPT_COOKIEJAR,  $cookieFile); // WRITE cookies received from the server
curl_setopt($curl, CURLOPT_COOKIEFILE, $cookieFile); // READ cookies to send with the request

• CURLOPT_COOKIEJAR writes any cookies the server sets into the file when the handle is closed.
• CURLOPT_COOKIEFILE reads cookies from that file and attaches them to outgoing requests.

Set both, pointing at the same file, and cURL maintains a session across multiple requests — exactly what you need for flows like open the login page → submit credentials → hit a page that requires being logged in.

If you'd rather hold cookies in memory across requests inside one script run, reuse the same cURL handle; cURL keeps the cookies internally between curl_exec() calls on that handle. To send specific cookies manually:

curl_setopt($curl, CURLOPT_COOKIE, 'session=abc123; lang=en');

Cookies are the first place where scraping stops being "fetch a URL" and becomes "behave like a returning user."

5. Going Faster: Parallel Requests with multi cURL

Fetching hundreds of pages one after another is slow — most of the time is just waiting for the network. cURL's multi interface lets you run many requests concurrently, dramatically cutting total time.

$urls = [
    'https://testwebsite.com/textlist',
    'https://testwebsite.com/blocks',
    'https://testwebsite.com/whitespace',
    // ...more URLs
];

$mh = curl_multi_init();
$handles = [];

// 1. Build and register one handle per URL
foreach ($urls as $i => $url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 30);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
}

// 2. Drive all the transfers until they're done
$running = null;
do {
    curl_multi_exec($mh, $running);
    curl_multi_select($mh); // block efficiently until there's activity
} while ($running > 0);

// 3. Collect results and clean up
$results = [];
foreach ($handles as $i => $ch) {
    $results[$i] = curl_multi_getcontent($ch);
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}
curl_multi_close($mh);

A few practical notes:

• curl_multi_select() keeps the loop from burning CPU in a busy-wait — it sleeps until one of the transfers has progress to report.
• Don't fire off thousands of requests at once. Beyond exhausting your own resources, hammering a server in parallel is the fastest way to get rate-limited or banned. Process URLs in batches of, say, 5–20, with a short pause between batches.
• For higher-level concurrency with a friendlier API, the Guzzle HTTP client wraps all of this in promises and pools.

6. Reading the Response Code and Other Metadata

curl_exec() gives you the body. To make a robust scraper, you also need to know what kind of response you got. curl_getinfo() exposes that:

$page = curl_exec($curl);

$httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
$info     = curl_getinfo($curl); // full array of metadata

curl_close($curl);

The full $info array includes useful fields such as http_code, total_time, redirect_count / redirect_url, content_type, size_download, and primary_ip.

Why This Matters: Error Handling and Retries

The status code lets you branch your logic instead of blindly parsing whatever came back:

switch (true) {
    case $httpCode === 200:
        // Success — go ahead and parse $page
        break;

    case $httpCode === 403:
        // Forbidden — often an anti-bot block. Rotate IP/headers and back off.
        break;

    case $httpCode === 404:
        // Page gone — log it and skip, don't retry forever
        break;

    case $httpCode === 429:
        // Too Many Requests — you're being rate-limited. Slow down / wait.
        break;

    case $httpCode >= 500:
        // Server error — usually worth retrying with backoff
        break;

    default:
        // Anything unexpected
        break;
}

This is the foundation for retry logic (e.g. exponential backoff on 429/5xx), for logging failures, and for telling a real "no such page" (404) apart from a temporary hiccup worth retrying. Without inspecting the response code, your scraper can't tell the difference between "got the data" and "got a block page that happens to be valid HTML."

7. Routing Through a Proxy

If a site bans your IP or rate-limits it (429), routing requests through proxies lets you spread traffic across many addresses. cURL has direct options for this:

curl_setopt($curl, CURLOPT_PROXY,     '203.0.113.10');     // proxy host
curl_setopt($curl, CURLOPT_PROXYPORT, 8080);               // proxy port
curl_setopt($curl, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);     // or CURLPROXY_SOCKS5

// If the proxy requires authentication:
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'user:password');

You can also pass everything in one compact string:

curl_setopt($curl, CURLOPT_PROXY, 'http://user:password@203.0.113.10:8080');

Rotating proxies

Keep a pool and pick a different proxy per request (or per batch) so no single IP draws attention:

$proxies = [
    '203.0.113.10:8080',
    '203.0.113.11:8080',
    '203.0.113.12:8080',
];

$proxy = $proxies[array_rand($proxies)];
curl_setopt($curl, CURLOPT_PROXY, $proxy);

Combine rotation with sensible delays and a check on the response code (section 6): when a proxy starts returning 403/429, drop it from the pool for a while.

8. Scraping Through TOR

TOR is essentially a free, anonymizing proxy network. When the Tor service runs locally, it exposes a SOCKS5 proxy (by default on 127.0.0.1:9050), and you can point cURL straight at it. Using CURLPROXY_SOCKS5_HOSTNAME makes Tor resolve DNS too, which prevents DNS leaks:

$curl = curl_init('https://testwebsite.com/');
curl_setopt_array($curl, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => '127.0.0.1:9050',
    CURLOPT_PROXYTYPE      => CURLPROXY_SOCKS5_HOSTNAME, // route DNS through Tor
    CURLOPT_USERAGENT      => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' .
                              'AppleWebKit/537.36 (KHTML, like Gecko) ' .
                              'Chrome/124.0.0.0 Safari/537.36',
]);

$page = curl_exec($curl);

if (curl_errno($curl)) {
    echo 'Tor scraper error: ' . curl_error($curl);
}
curl_close($curl);

To confirm you're actually exiting through Tor, request Tor's check endpoint and look at the IP it reports:

$curl = curl_init('https://check.torproject.org/api/ip');
curl_setopt_array($curl, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_PROXY          => '127.0.0.1:9050',
    CURLOPT_PROXYTYPE      => CURLPROXY_SOCKS5_HOSTNAME,
]);
echo curl_exec($curl); // {"IsTor":true,"IP":"..."}
curl_close($curl);

For a fresh exit IP, you can send a NEWNYM signal to Tor's control port (default 127.0.0.1:9051) to build a new circuit — an advanced step that requires enabling and authenticating the control port in your torrc.

Caveats: Tor is slow, many sites actively block known Tor exit nodes, and you should always stay within a site's terms and the law.

9. Scraping HTTPS When SSL Verification Gets in the Way

Sometimes a request to an HTTPS URL fails with an SSL error — a self-signed certificate, an outdated CA bundle on your server, or a site behind a corporate proxy. You can tell cURL to skip certificate checks:

curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false); // don't verify the cert chain
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);     // don't verify the hostname

⚠️ Use this only when you have to, and never in production against untrusted hosts. Disabling verification removes the protection TLS gives you and opens the door to man-in-the-middle attacks. It's acceptable for quick debugging, internal endpoints you control, or known self-signed certs — not as a default.

The correct fix for most "certificate problem" errors is to give cURL an up-to-date CA bundle instead of turning verification off:

curl_setopt($curl, CURLOPT_CAINFO, '/path/to/cacert.pem'); // from https://curl.se/docs/caextract.html

10. Calling the API Directly with POST Requests

Here's a trick that often beats HTML scraping entirely. Many "JavaScript-heavy" sites don't embed data in the HTML at all — their front-end JS fetches it from a JSON/XHR API and renders it in the browser. If you open your browser's DevTools → Network → Fetch/XHR, you can often see that exact request: its URL, method, headers, and body. You can then replicate it directly with cURL and get clean, structured JSON — no HTML parsing and no JS engine required.

Many of these endpoints expect a POST request with a JSON body:

$payload = json_encode([
    'query'    => 'laptops',
    'page'     => 1,
    'pageSize' => 50,
]);

$curl = curl_init('https://testwebsite.com/api/v2/search');
curl_setopt_array($curl, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => $payload,
    CURLOPT_HTTPHEADER     => [
        'Content-Type: application/json',
        'Accept: application/json',
        'X-Requested-With: XMLHttpRequest',
        // Copy any token the site's own JS sends with the call, e.g.:
        // 'Authorization: Bearer eyJhbGci...',
        // 'X-CSRF-Token: ...',
    ],
]);

$response = curl_exec($curl);
$httpCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
curl_close($curl);

if ($httpCode === 200) {
    $data = json_decode($response, true);
    foreach ($data['items'] ?? [] as $item) {
        echo $item['title'], ' — ', $item['price'], PHP_EOL;
    }
}

This is frequently the most efficient and most stable way to scrape: APIs change their shape far less often than page markup, and the response is already structured. The catch is authentication — some endpoints require a token or CSRF value that the site's JavaScript generates or that gets set by JS, which leads directly into the next section.

11. The Hard Wall: Why Fast cURL Scrapers Get Banned

Everything above is fast, lightweight, and effective — on sites that serve their content directly. On modern, protected sites it often isn't enough, and the reason is fundamental:

cURL fetches HTML. It does not execute JavaScript.

That single fact is the basis of most anti-bot defenses you'll meet:

• JS-rendered content. Many sites ship an almost-empty HTML shell and build the real page in the browser with JavaScript (React, Vue, and other SPAs). cURL receives the shell and finds nothing to scrape, because the data only appears after JS runs.
• JavaScript-set cookies. A very common protection (Cloudflare, DataDome, PerimeterX/HUMAN, and similar) serves an interstitial whose JavaScript performs a small computation and then sets a cookie proving "a real browser ran this." A genuine browser runs the script, gets the cookie, and is let through. cURL never runs the script, never gets the cookie, and is shown the challenge page forever — or simply blocked.
• Behavioral and fingerprint checks. Beyond cookies, defenses inspect TLS fingerprints (JA3), HTTP/2 frame ordering, header order, timing, and mouse/scroll behavior. A cURL request looks nothing like Chrome at these layers, no matter how perfect your User-Agent string is.

So the tradeoff is real: cURL scrapers are fast and cheap, but easy to detect and ban the moment a site decides to defend itself with JavaScript.

Where to go next: real browsers

Everything in this guide is a relatively simple, raw-HTTP implementation. When you genuinely need JavaScript execution, you step up to tools that drive a real browser — they run the JS, receive the JS-set cookies, and render SPA content for you, at the cost of speed and resources:

• Puppeteer / Playwright — Node-based, control headless Chromium/Firefox/WebKit.
• Selenium — language-agnostic browser automation, works with PHP too.
• PHP bridges — symfony/panther and chrome-php/chrome drive Chromium directly from PHP.

A natural next chapter is to take the same example site and render it with one of these, since that's the direct answer to the wall described above.

Going Further: Topics to Expand This Guide

Each of these deserves its own section:

1. Use Guzzle instead of raw cURL. A modern PHP HTTP client with a clean API, middleware, promises, and easy concurrency pools.
2. Rate limiting and politeness. Delays, jitter, batching, honoring Retry-After, reading robots.txt, and not melting the target server.
3. Retry logic and backoff. Exponential backoff with a cap; distinguishing retryable (5xx, 429) from permanent (404) failures.
4. Sessions, logins, and CSRF tokens. POSTing forms with cURL, extracting hidden form tokens, and maintaining an authenticated session.
5. Captchas and advanced anti-bot. What reCAPTCHA / hCaptcha / Cloudflare Turnstile and services like DataDome do, and the realistic (and limited) options for dealing with them.
6. TLS / HTTP-2 fingerprinting. Why JA3 and header-order fingerprints give cURL away, and what tools like curl-impersonate try to do about it.
7. Storing and structuring scraped data. Writing to CSV/JSON, inserting into a database, deduplication, and incremental scraping.
8. Legal and ethical boundaries. Terms of Service, robots.txt, copyright, personal data/GDPR, and scraping responsibly.