Authentication, dynamic pages, headless browsers, and anti-bot bypass.
XPath for web scraping: the cheat sheets and library references worth keeping open, five expressions that run clean and return the wrong nodes, and where XPath stops working.
How to pull data from web APIs instead of HTML: finding the endpoint a page already calls, auth, cursor and offset pagination, 429 handling and retries. With measured byte and parse costs against HTML, client versions checked in August 2026, and the rate-limit header that no standard actually defines.
PHP cURL behind a login, rerun on 10 August 2026 against PHP 8.4.21 and libcurl 8.5.0: cookie jars, CSRF tokens, the curl_close() trap that silently drops the session, HTML5 parsing, and the point where a headless browser becomes the only option.
How cookies travel between browser and server, and where scraping clients disagree about them. Measured in August 2026 on curl, requests, httpx and aiohttp, with the rules only browsers enforce.
Crawl a sitemap before you scrape: find it through robots.txt, parse it past the namespace trap, survive gzip and index loops, filter by lastmod. Measured on a 50,000-URL file, checked August 2026.
CSS selectors for web scraping, rechecked in August 2026: combinators, attribute matching, nth-child, :has() and text matching, with a support matrix run against BeautifulSoup 4.15.0, parsel 1.11.0 and cssselect 1.5.0, plus timings from a 2.4 MB page.
How distributed crawling actually works, with every figure read from primary sources in August 2026: Common Crawl at 2.14 billion pages and 84.69 TiB a month, a measured Bloom filter at 1.8 bytes per URL against 82 for an exact set, and why Spark's shuffle is a disk workload.
Edit and resend an HTTP POST from DevTools: the Firefox inline editor, the Edge Network Console, why Chrome Replay XHR skips fetch requests, and the 25 headers Copy as fetch drops.
Email validation regex patterns run against the same address list on 10 August 2026: what each one accepts and rejects, the SMTP length limits none of them encode, and measured backtracking costs.