A scraper that works on the happy path is easy. A scraper that keeps working at 3 a.m. when a target site times out, returns a 503, throttles you, or quietly changes its markup — that takes real error handling. This PHP scraper example is built around exactly that: how to use try/catch, custom exceptions, retries with backoff, and logging so your crawler survives the internet instead of dying on the first hiccup.
We'll use modern PHP 8.x with Guzzle for HTTP and Symfony's DomCrawler for parsing — the current stack for web scraping using PHP — and finish with a global handler as a last line of defense.
Why exception handling matters more in scrapers than almost anywhere else
Most code talks to systems you control. A scraper talks to the open web, which fails constantly and creatively:
- Network errors — DNS failures, connection timeouts, dropped connections.
- HTTP errors —
404(gone),429(rate-limited),500/503(server down or overloaded). - Content errors — the page loaded fine but the markup changed and your selector matches nothing.
- Parsing errors — malformed HTML, unexpected encodings, empty responses.
Without handling, any one of these throws and kills the whole run — losing every item you hadn't saved yet. The goal isn't to prevent errors (you can't); it's to expect them, recover where possible, log the rest, and keep going.
The tools: Guzzle, DomCrawler, Monolog
Install the modern stack:
composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector monolog/monolog
- Guzzle — HTTP client. By default it throws on
4xx/5xxresponses (http_errorsis on), which is exactly what we want to catch. - Symfony DomCrawler + CssSelector — parse HTML with CSS selectors or XPath. (Note: the old Goutte library is now part of Symfony as
HttpBrowser, so this is the modern Goutte.) - Monolog — structured logging so failures leave a trail.
Guzzle's exception hierarchy (know what you're catching)
Guzzle throws specific exceptions, and catching the right one lets you react correctly. From most to least specific:
GuzzleHttp\Exception\GuzzleException (interface — catch-all)
└─ TransferException
├─ ConnectException network/DNS/timeout — no response at all
└─ RequestException
├─ BadResponseException
│ ├─ ClientException 4xx (404, 429, 403 …)
│ └─ ServerException 5xx (500, 502, 503 …)
└─ TooManyRedirectsException
The distinction that matters most: a ConnectException or ServerException is usually transient — worth retrying. A ClientException like 404 is usually permanent — don't retry, just log and skip. A 429 is special: back off and retry, honoring any Retry-After header.
A robust PHP scraper example
Here's a compact but production-minded scraper that ties it all together: custom exceptions, a retry loop with exponential backoff, targeted try/catch, parse-time validation, and logging.
<?php
declare(strict_types=1);
require 'vendor/autoload.php';
use GuzzleHttp\Client;
use GuzzleHttp\Exception\ConnectException;
use GuzzleHttp\Exception\ClientException;
use GuzzleHttp\Exception\ServerException;
use GuzzleHttp\Exception\GuzzleException;
use Symfony\Component\DomCrawler\Crawler;
use Monolog\Logger;
use Monolog\Handler\StreamHandler;
/* ---- Custom exceptions: give failures meaningful types ---- */
class ScraperException extends \RuntimeException {}
class FetchException extends ScraperException {} // couldn't get the page
class ParseException extends ScraperException {} // got the page, markup broke
final class ProductScraper
{
private Client $http;
public function __construct(private Logger $log)
{
$this->http = new Client([
'timeout' => 20,
'connect_timeout' => 10,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (compatible; MyScraper/1.0)',
'Accept' => 'text/html,application/xhtml+xml',
],
]);
}
/** Fetch with retries + exponential backoff on transient failures. */
private function fetch(string $url, int $maxRetries = 3): string
{
$attempt = 0;
while (true) {
$attempt++;
try {
$res = $this->http->get($url);
return (string) $res->getBody();
} catch (ConnectException $e) {
// Network problem — transient, retry.
$this->backoffOrFail($url, $attempt, $maxRetries, $e);
} catch (ServerException $e) {
// 5xx — server-side, transient, retry.
$this->backoffOrFail($url, $attempt, $maxRetries, $e);
} catch (ClientException $e) {
$code = $e->getResponse()->getStatusCode();
if ($code === 429) {
// Rate-limited: honor Retry-After if present, then retry.
$wait = (int) ($e->getResponse()->getHeaderLine('Retry-After') ?: 2 ** $attempt);
$this->log->warning("429 rate-limited, sleeping {$wait}s", ['url' => $url]);
sleep($wait);
if ($attempt > $maxRetries) {
throw new FetchException("Rate limit not clearing for $url", 0, $e);
}
continue;
}
// Other 4xx (404, 403, …) — permanent, don't retry.
throw new FetchException("Client error $code for $url", 0, $e);
} catch (GuzzleException $e) {
// Anything else Guzzle can throw.
throw new FetchException("Request failed for $url: {$e->getMessage()}", 0, $e);
}
}
}
private function backoffOrFail(string $url, int $attempt, int $max, \Throwable $e): void
{
if ($attempt > $max) {
throw new FetchException("Gave up on $url after $max retries", 0, $e);
}
$delay = (2 ** $attempt) + (random_int(0, 1000) / 1000); // jitter
$this->log->warning("Retry {$attempt}/{$max} for {$url} in {$delay}s: {$e->getMessage()}");
usleep((int) ($delay * 1_000_000));
}
/** Parse one product page; fail loudly if the markup changed. */
public function scrape(string $url): array
{
$html = $this->fetch($url);
$dom = new Crawler($html);
try {
$title = trim($dom->filter('h1.product-title')->text());
$price = trim($dom->filter('.price')->text());
} catch (\InvalidArgumentException $e) {
// DomCrawler throws this when filter() matches nothing.
throw new ParseException("Selector missed on $url — markup may have changed", 0, $e);
}
if ($title === '') {
throw new ParseException("Empty title on $url");
}
return ['url' => $url, 'title' => $title, 'price' => $price];
}
}
/* ---------------- Driver: keep going despite per-URL failures ---------------- */
$log = new Logger('scraper');
$log->pushHandler(new StreamHandler('scraper.log', Logger::DEBUG));
$scraper = new ProductScraper($log);
$urls = [
'https://example.com/product/1',
'https://example.com/product/2',
'https://example.com/does-not-exist',
];
$results = [];
foreach ($urls as $url) {
try {
$results[] = $scraper->scrape($url);
$log->info("Scraped OK", ['url' => $url]);
} catch (FetchException $e) {
$log->error("Fetch failed", ['url' => $url, 'error' => $e->getMessage()]);
continue; // skip this URL, keep the crawl alive
} catch (ParseException $e) {
$log->error("Parse failed", ['url' => $url, 'error' => $e->getMessage()]);
continue;
}
usleep(500_000); // be polite: throttle between requests
}
echo "Scraped " . count($results) . " of " . count($urls) . " URLs\n";
The important design choices here:
- The
try/catchlives in the driver loop, so one broken URL is skipped, not fatal. Everything you've scraped so far is safe. - Retries only happen for transient errors (network,
5xx,429) with exponential backoff and jitter, so you don't hammer a struggling server or synchronize retries into a thundering herd. 404/403fail fast — retrying a permanent error just wastes time and looks more like abuse.- Custom exception types (
FetchExceptionvsParseException) let the caller react differently — e.g., alerting only when parsing breaks, which usually means the site's layout changed and your selectors need updating.
Retries the built-in way: Guzzle middleware
Rolling your own loop is clear and explicit, but Guzzle also ships a retry middleware if you'd rather centralize it:
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use GuzzleHttp\Psr7\Request;
use Psr\Http\Message\ResponseInterface;
$stack = HandlerStack::create();
$stack->push(Middleware::retry(
function (int $retries, Request $req, ?ResponseInterface $res = null, ?\Throwable $e = null) {
if ($retries >= 3) return false;
if ($e instanceof \GuzzleHttp\Exception\ConnectException) return true;
if ($res && $res->getStatusCode() >= 500) return true;
if ($res && $res->getStatusCode() === 429) return true;
return false;
},
fn (int $retries) => (int) (1000 * (2 ** $retries)) // ms delay, doubles each time
));
$client = new Client(['handler' => $stack, 'timeout' => 20]);
Either approach works; the middleware keeps retry policy out of your business logic, while the explicit loop is easier to read and debug when you're learning.
A global safety net: set_exception_handler
Structured try/catch should handle the errors you anticipate. For the ones you don't, register a global handler so an uncaught exception is logged (and, for a long-running scraper, doesn't just vanish). This is the modernized, CLI-appropriate version of the old pattern:
set_exception_handler(function (\Throwable $e) use ($log): void {
$log->critical('Uncaught exception', [
'type' => $e::class,
'message' => $e->getMessage(),
'file' => $e->getFile() . ':' . $e->getLine(),
]);
fwrite(STDERR, "Fatal: {$e->getMessage()}\n");
exit(1);
});
// Catch fatal errors (out of memory, etc.) that bypass the handler above:
register_shutdown_function(function () use ($log): void {
$err = error_get_last();
if ($err && in_array($err['type'], [E_ERROR, E_PARSE, E_CORE_ERROR], true)) {
$log->critical('Fatal error', $err);
}
});
Two notes for anyone porting the classic web-based version of this: only send an HTTP/1.1 500 header if your scraper actually runs behind a web server and before any output — for a CLI/cron scraper (the normal case) you want a non-zero exit code and a log entry instead. And in PHP 7+/8 catch \Throwable, not just \Exception, so Errors (like type errors) are caught too.
Beyond exceptions: making a scraper truly robust
Error handling is necessary but not sufficient. Round it out with:
- Validation, not just try/catch. After parsing, assert that fields look right (non-empty title, a price that parses as a number). A silently-empty field is worse than an exception.
- Timeouts everywhere. Always set
connect_timeoutandtimeout; a hung connection with no timeout can freeze a whole run. - Polite throttling and realistic headers to avoid triggering anti-scraping defenses in the first place.
- Rotating proxies when a single IP gets blocked or rate-limited at scale.
- CAPTCHA handling for challenges — though a CAPTCHA usually means you should slow down first.
- Idempotent saves. Persist each item as you get it (or in batches) so a crash never loses completed work.
For JavaScript-rendered pages that Guzzle can't see, step up to a headless browser in PHP (Symfony Panther / ChromeDriver). And for the broader toolkit, see our web scraping with PHP guide.
FAQ
How do I handle a 429 (rate limit) in a PHP scraper?
Catch Guzzle's ClientException, check for status 429, read the Retry-After header if present (fall back to exponential backoff otherwise), sleep, then retry. Persistent 429s mean you should slow your overall request rate or add proxies.
Should I retry every failed request?
No. Retry transient failures — network errors (ConnectException), 5xx, and 429. Don't retry permanent ones like 404 or 403; log and skip them.
try/catch or a global exception handler?
Both. Use targeted try/catch around fetching and parsing for expected, recoverable errors; register set_exception_handler (plus register_shutdown_function) as a last-resort net for the unexpected.
Why does DomCrawler throw when my selector finds nothing?
filter()->text() throws InvalidArgumentException on an empty match. Catch it (or check ->count() first) and treat it as a signal that the site's markup changed — that's your cue to update selectors.
Bottom line
A robust PHP scraper isn't the one that never errors — it's the one that expects to. Wrap fetching and parsing in targeted try/catch, give failures meaningful custom types, retry only what's transient (with backoff and jitter), validate what you parsed, log everything, and keep a global handler as backstop. Build it that way and a single flaky URL costs you one skipped item and a log line, not the whole run.
If maintaining scrapers through every site change, block, and outage isn't where you want to spend your time, scraping.pro runs extraction as a managed web scraping service and delivers clean, structured data on a schedule.