Core concepts and how web scraping works.
Web scraping best practices rechecked in August 2026: robots.txt under RFC 9309, a Python stdlib parser that got three rules of four wrong until May 2026, Retry-After, Decimal money, and measured storage and parsing costs for a ten-thousand-page run.
A web scraping reading list rechecked in August 2026: core Python titles with dates, page counts and prices, the regex canon, storage books, and the parts that have quietly rotted.
Eleven languages compared for web scraping, rechecked in August 2026: measured parse times, current library versions and release dates, dead projects flagged, and what the choice actually costs.
Sixteen web scraping projects from beginner to advanced, each with a stack, a failure mode and a price per thousand pages. Includes a measured parser benchmark, proxy and scraping-API rates read from vendor pricing pages on 10 August 2026, and the three ideas on this list that stopped working since it was written.
Web crawling finds pages, scraping fetches and extracts them, parsing turns markup into structure. Where the line falls, why robots.txt covers only one of the three, measured fetch-versus-parse timings from August 2026, and three parsers returning three different answers for the same tag.
Crawling finds product URLs, scraping lifts price and stock. What four large retailers allow in robots.txt, which JSON endpoints stores publish on purpose, what a pass over 200,000 pages costs at list prices, and where the pipeline breaks. Sources read 10 August 2026.
How a DOM parser turns markup into a node tree, and why two parsers build two different trees from the same bytes: the tbody rule that breaks copied selectors, measured parse time and memory for lxml, BeautifulSoup, cheerio, jsdom and lexbor, and library versions read from each vendor in August 2026.
HTTP for scrapers, rechecked against the current specifications on 13 August 2026: RFC 9110-9114 instead of the obsolete 7230 series, exact section numbers, measured default headers from four clients, cookie limits, caching, and version fingerprints.
HTTP vs HTTPS rechecked on 13 August 2026: the RFC 9110-9114 series that retired RFC 2818, the 200-day certificate cap in force since 15 March 2026, post-quantum ClientHellos on over 60% of Cloudflare HTTPS traffic, and why TLS fingerprinting blocks a client whose headers are perfect.