Best Python Web Scraping Libraries (Plus Other Languages)

Parsing a 1.5 MB HTML page with lxml takes 15 milliseconds on a slow cloud core. Parsing the same page with BeautifulSoup's default backend takes 190. Neither number matters on page one. Across ten thousand pages the gap is close to half an hour of pure CPU, and that is the smaller of the two decisions a library choice makes for you. The larger one is whether you start a browser at all.

We rechecked every package here against its own release feed, repository or package registry on 10 August 2026. The dates are printed for a reason. Four tools the earlier version of this article recommended without qualification have shipped nothing in over two years, and one was archived by its owner in January 2026. A catalog without dates cannot tell you that.

In a separate article we looked at which language to write a scraper in. This is the next level of detail.

The four roles a library plays

Almost any scraper is assembled from tools of four types, and most working ones use two of them glued together in about six lines.

• HTTP clients. Send a request, receive bytes. No JavaScript, so they see what the server sent and nothing more. They are also where anti-bot systems meet you first, because a client's TLS handshake and HTTP/2 settings frame are a fingerprint before a single header is read.
• HTML/XML parsers. Turn a string into a tree you query with CSS selectors or XPath. They fetch nothing.
• Browser automation. Launch a real browser, run its JavaScript, click and scroll and wait. Indispensable for single-page apps, and two orders of magnitude more expensive per page than the pair above.
• Full frameworks. Take over the queue, the concurrency, the retries, the deduplication and the output.

The split that costs money is between the first two and the third. Everything else is taste.

What the parser choice costs, measured

Every roundup asserts that lxml is fast and BeautifulSoup is slow. Almost none put a number on it, and the number is more interesting than the ranking.

We ran two documents through five parser configurations. The small one is a PyPI project page, 182,494 bytes. The large one is the PyPI simple index for boto3, 1,555,175 bytes and 4,180 anchors, which behaves like the category listings you actually crawl. The task: parse, then pull every href.

Measured on 10 August 2026 on one 2-core Intel Xeon at 2.10 GHz, 8 GB RAM, Python 3.11.15, lxml 6.1.0, beautifulsoup4 4.14.3, selectolax 0.4.11. Best of 9 to 15 runs, garbage collector run between iterations. Your absolute numbers will differ by a factor of two or three. The ratios travel.

import time, gc
from bs4 import BeautifulSoup
import lxml.html
from selectolax.lexbor import LexborHTMLParser

html = open("big.html", encoding="utf-8", errors="replace").read()

def best_of(fn, reps=9):
    ts = []
    for _ in range(reps):
        gc.collect()
        t0 = time.perf_counter(); fn(html); ts.append(time.perf_counter() - t0)
    return min(ts) * 1000

print(best_of(lambda h: BeautifulSoup(h, "html.parser").find_all("a", href=True)))
print(best_of(lambda h: lxml.html.fromstring(h).xpath("//a/@href")))
print(best_of(lambda h: LexborHTMLParser(h).css("a[href]")))

Three results are worth arguing with.

The first is the size of the gap. On the large page lxml with XPath is about 17 times faster than BeautifulSoup on its default backend. Across ten thousand pages that is 267 milliseconds each, or 45 minutes of one core spent building objects you throw away.

The second is that switching BeautifulSoup's backend to lxml buys only 1.4x. That is the correction that matters, because "just pass features='lxml'" is the standard advice and it fixes the smaller half. BeautifulSoup's cost is not the parse. It is the Python object tree built on top: 133 ms of the 190 ms survived handing the parsing to C.

The third contradicts the earlier version of this article, which called selectolax the fastest at large volumes. On raw parsing it is, at 10.5 ms against lxml's 14.9. Add the selector query and lxml takes it back, 17.1 ms against 23.7, because selectolax returns Python objects node by node while lxml's XPath engine stays in C for the whole traversal. Which one wins depends on how much you pull out, not on how big the page is.

Use lxml or parsel by default. Reach for selectolax when you parse far more than you select. Keep BeautifulSoup for code a colleague will read in a year.

Python

HTTP clients

requests 2.34.2, 14 May 2026. Still the one to reach for first, and it moved this year after a quiet stretch. The 2.34.0 release shipped inline type hints, retiring the separate stub package most typed codebases carried, and the docs now put the floor at Python 3.10. Synchronous, no HTTP/2, and neither has ever been why a scraper failed.

httpx 0.28.1, 6 December 2024. Check this one before repeating it: httpx is usually described as the modern replacement for requests, and it has not published a release in twenty months. It is still on a 0.x number, with an UNRELEASED heading sitting on master. None of that says abandoned, and the library works. It does mean you should pin it and expect no fixes on a schedule.

aiohttp 3.14.3, 23 July 2026. Client and server in one package, async only, shipping regularly. For thousands of concurrent connections against a static target it is the fastest ordinary thing in Python.

curl_cffi 0.16.0, 1 August 2026. Bindings to a fork of curl-impersonate, so your TLS and HTTP/2 fingerprints match a real browser without running one. This is the package whose shape changed most since this article was written, and the change is a boundary. Development moved to the lexiforest fork, HTTP/3 fingerprints arrived in 0.15.0, and since 0.15.1 curl-cffi update pulls new ones without a version bump. The open-source build ships 37 preset fingerprints with free updates for Chrome, Safari and Firefox. The full per-platform database is the vendor's commercial plan at impersonate.pro, listed at $99 per month on 10 August 2026.

Its own documentation draws the important line for you: "curl_cffi is a python binding to a C library, with no browser or JavaScript runtime under the hood." It cannot repair a JavaScript fingerprint, and against systems that read one, it will not help. Cloudflare, Akamai and the older Distil-class defenses all read one.

Parsers

lxml 6.1.1, 18 May 2026. The default answer. C speed, full XPath 1.0, and what parsel and half the ecosystem sit on. A 7.0.0a1 alpha appeared on 29 May 2026, so a major version is in flight. If XPath is new to you, our lxml walkthrough covers the query side.

BeautifulSoup 4.15.0, 7 June 2026. Not a parser at all: it is a wrapper that delegates to html.parser, lxml or html5lib and then builds its own object tree. Small detail with a lesson in it. PyPI has served 4.15.0 since June, while the project's own home page still advertises 4.14.3 from 30 November 2025. When a version number matters, read the registry rather than the home page.

parsel 1.11.0, 29 January 2026. Scrapy's selector layer, usable standalone: CSS, XPath, and chaining between them in one object. If you like .css('a::attr(href)') but not Scrapy, this is that piece alone.

selectolax 0.4.11, 15 July 2026. Two C backends, Lexbor and Modest. Fastest raw parse in the table above, lowest memory, no XPath.

pyquery 2.1.0, 27 July 2026. jQuery syntax over lxml, for people arriving from the front end.

Browser automation

Playwright 1.62.0, 31 July 2026. The default when a page needs a browser. Auto-waiting removes most of the sleep-and-pray code that makes Selenium scripts flaky, and the release bundles pinned builds: Chromium 151.0.7922.34, Firefox 153.0, WebKit 26.5. Pinning is the underrated part. Your run in November behaves like your run in August.

Selenium 4.46.0, 11 July 2026. The veteran, and the only one here with two decades of answered questions behind it. Selenium Manager now fetches drivers on its own, removing the most common setup failure. Our WebDriver tips are written against Java, but the traps are the same in Python.

Chrome changed underneath both of them. Google's own headless documentation states that "since Chrome 132.0.6793.0 the old Headless mode is only available as a standalone binary named chrome-headless-shell". Anything you read before 2025 about --headless=old, or about headless Chrome missing features headful Chrome has, describes a browser that is no longer shipped.

SeleniumBase 4.51.11, 7 August 2026. A framework over Selenium with two stealth modes: UC Mode, built on undetected-chromedriver, and the newer CDP Mode, which the project says "bypasses bot-detection with Chromium-based browsers" and demonstrates against the BrowserScan and Sannysoft pages. Treat published passes as the floor of what a site knows about you, not the ceiling.

nodriver 0.50.3, 13 May 2026. Talks to Chrome over the DevTools Protocol with no WebDriver in the path, async throughout, and its README calls itself "the official successor of the Undetected-Chromedriver python package."

undetected-chromedriver 3.5.5, February 2024. No release in two and a half years, no push since July 2025, 1,143 issues open. The earlier version of this article called it "largely dated." The dates say something firmer.

Three tools now do most of the work that paragraph used to describe. zendriver 0.15.5 is a fork of nodriver, created in its maintainers' words "to add new features, compile unmerged bugfixes, and increase community engagement." patchright 1.61.2 is a patched Playwright, drop-in. camoufox 0.5.4 is a modified Firefox rather than a modified driver, and it is the rare project honest enough to head its own README with a warning: "This project is under development. It may not be suitable for stable production use."

Frameworks

Scrapy 2.17.0, 7 July 2026. Fifteen years old, maintained by Zyte with more than 500 contributors, and quietly rebuilding its own foundations. The 2.17 release notes add HTTP/2 and SOCKS proxy support to an experimental httpx-based download handler, and 2.16 added an experimental mode that runs without a Twisted reactor at all. Twisted is why Scrapy has felt like a separate universe from the rest of async Python, so that matters more than a point release usually does. Minimum Python is 3.10. Our Scrapy series starts from a blank project.

Crawlee for Python 1.9.1, 6 August 2026. Apify's framework, past 1.0 and shipping weekly. Its AdaptivePlaywrightCrawler decides per URL whether plain HTTP would have done, and pays for a browser only when it would not. That is the right default for a mixed site.

MechanicalSoup 1.4.0, 30 May 2025. Sessions, forms, link following, no JavaScript. Small and stable rather than stalled.

Scrapling 0.4.13, 9 August 2026. The earlier version of this article described it as an adaptive parser. That was wrong in the direction that matters, because it made the project sound like a BeautifulSoup alternative. Scrapling is a framework: a plain Fetcher with TLS impersonation, a StealthyFetcher aimed at Cloudflare-class defenses, a DynamicFetcher over Playwright, sessions with proxy rotation, a Scrapy-shaped spider layer, adaptive selectors that relocate an element after a redesign, and an MCP server for driving it from an LLM. 69,000 stars, BSD-3-Clause. Its first release was October 2024, which is the other half of the sentence.

What breaks when the job gets big

A scraper that runs once on your laptop and one that runs nightly against ten thousand URLs fail for different reasons. The list above solves the first problem. Here is what the second costs.

Browsers do not scale the way HTTP clients do. One Chromium process per worker, each with its own renderer, is the difference between a crawl that fits on one small box and a crawl that needs a cluster. The benchmark above is the useful frame: an HTTP fetch plus an lxml parse costs milliseconds of your CPU, a browser page load costs seconds of somebody's. Before reaching for Playwright, open the page with JavaScript disabled and look for a __NEXT_DATA__ script tag, a window.__NUXT__ assignment, or an XHR returning the same data as JSON. On a large share of modern sites the browser is optional and nobody checked.

Your client is a fingerprint before it is a request. TLS cipher ordering, the HTTP/2 SETTINGS frame, header order: all read and hashed before your carefully chosen User-Agent is looked at. That is why curl_cffi exists, and why "rotate your user agent" belongs to about 2016.

The ground rules changed in July 2025. Cloudflare announced on 1 July 2025 that every new domain signing up is asked whether to allow AI crawlers, with blocking as the offered default, alongside a Pay Per Crawl marketplace. A default that flips at that layer reaches more sites in a quarter than any individual robots.txt policy will in a decade.

The traffic numbers explain the mood. The 2026 Bad Bot Report from Thales, published 30 April 2026 on 2025 traffic, put automated traffic at 53% of everything it observed: 40% bad bots, 13% benign automation. It also reported AI-driven bot activity rising 12.5-fold, with daily blocked requests going from 2 million to 25 million. Your polite nightly crawl arrives in that flood and is sorted by machines that have never heard of you.

One C library sits under a surprising amount of this. lxml binds libxml2. Nokogiri ships libxml2. PHP's DOM extension is libxml2. On 17 June 2025 the maintainer, Nick Wellnhofer, announced that libxml2 would stop accepting embargoed security reports: issues get published immediately and fixed when someone has time, with no deadlines. He had already resigned as libxslt maintainer and wrote that it is unlikely that project will ever be maintained again, adding that triage cost him several unpaid hours a week while the companies shipping the library in their operating systems paid nothing. Parser CVEs now surface as public bugs with no coordinated patch waiting, so pin your versions and read the advisory feed instead of assuming a distribution update is on its way.

And the legal frame is not a library question. Meta Platforms v. Bright Data ended on 23 January 2024 with Judge Chen in the Northern District of California granting summary judgment for Bright Data, holding that Meta's terms did not bind a logged-off visitor collecting public data. The case everyone half-remembers ended the other way: hiQ won its CFAA argument at the Ninth Circuit, lost on contract, and finished in 2022 with a $500,000 judgment and a permanent injunction against scraping LinkedIn. Being logged out matters. Being logged in matters more. No library changes either.

None of the five problems above is solved by a better import line. They are staffed, or they are handed to a managed extraction service.

JavaScript / Node.js

HTTP clients: the native fetch for anything simple; axios 1.19.0 (29 July 2026) for the familiar API, at the price of a dependency doing what the platform already does; got 15.1.0 for retries and hooks; undici 8.10.0 (3 August 2026) from the Node.js team, which is what fetch is built on and worth using directly for connection-pool control.

Parsers: cheerio 1.2.0 (23 January 2026), jQuery syntax over static HTML and the right default; jsdom 30.0.1 when you need a real DOM, at several times the cost; parse5 8.0.1 and htmlparser2 12.0.0 underneath both.

Browser automation: Puppeteer 25.5.0 (4 August 2026) for Chrome, Playwright 1.62.1 for everything else.

One correction, and it is the one to carry out of this section. puppeteer-extra with its stealth plugin is the reflex recommendation in every Node scraping tutorial. Both packages last published in March 2023: puppeteer-extra 3.3.6 and puppeteer-extra-plugin-stealth 2.11.2, with 235 issues open. You are shipping evasions written against Chrome 110 into a world running Chrome 151.

Frameworks: Crawlee 3.18.0 (4 August 2026) from Apify, with queues, proxy rotation, session pools and automatic HTTP-to-browser switching. The most complete thing in the Node column by a distance. node-crawler 2.1.1 is the older, smaller option and still ships.

PHP

HTTP clients: Guzzle, on 8.0.2 (5 August 2026) with the 7.15 line in maintenance and past a billion installs; Symfony HttpClient inside Symfony; native cURL for fingerprint control.

Parsers: Symfony DomCrawler for CSS and XPath; DiDOM for something smaller and quicker.

Simple HTML DOM is where the honest label goes. It has the lowest barrier to entry in PHP and it digests broken markup other parsers reject. Its newest published version is 2.0-RC2, from 9 November 2019. A release candidate that has stood for nearly seven years is a finished project. Use it knowingly.

Navigation and browser: Symfony BrowserKit's HttpBrowser follows links and posts forms in pure PHP with no JavaScript. Goutte, which older articles still recommend by name, was archived on 1 April 2023, and its final README says so plainly: "As of v4, Goutte became a simple proxy to the HttpBrowser class from the Symfony BrowserKit component." For real rendering, Symfony Panther drives Chrome or Firefox over WebDriver on top of php-webdriver.

Frameworks: Roach PHP 3.2.1 (21 March 2025) is a genuine Scrapy analog with spiders, middleware and item pipelines, on PHP 8.2 and up. crwlr v3.5 composes crawlers out of prebuilt steps instead, and needs PHP 8.1.

Go (Golang)

HTTP clients: net/http covers most of it; resty adds retries and a fluent API.

Parsers: goquery v1.12.0 (15 March 2026), jQuery syntax over net/html, needing Go 1.25 or later; htmlquery for XPath.

Browser automation: chromedp 0.15.1 (1 April 2026) speaks DevTools Protocol directly and is the maintained choice. Rod has the friendlier API and wins most ergonomics comparisons, but its last tagged release is v0.116.2 from 12 July 2024. Commits continue; releases do not. Pinning a two-year-old tag against a browser that ships every four weeks is a decision, so make it deliberately.

Frameworks: Colly v2.2.0 (27 March 2025) is the standard for static crawling in Go, and its own front page claims more than 1,000 requests per second on a single core. Geziyor adds JS rendering. Ferret queries pages in its own declarative language, and its repository carries a notice that the default branch holds an in-progress v2 while v1 sits on a branch of its own. Read that as mid-rewrite.

C / C++

There are almost no high-level frameworks here. People assemble an HTTP client and a parser by hand, for speed and for memory that does not move.

HTTP clients: libcurl, which calls itself "probably the most portable, most powerful and most often used network transfer library on this planet" and is the foundation almost every other language's client rests on; CPR, a C++ wrapper over it shaped like Python requests.

Parsers: libxml2, mature and now under the maintenance policy described above; lexbor, an Apache-2.0 browser engine as a library; pugixml for small XML jobs like sitemaps and RSS.

Gumbo is the one to strike from your notes. Google's HTML5 parser appears in nearly every C++ scraping article written since 2013. The repository was archived by its owner on 21 January 2026, and the README shipping with it reads: "This project has been unmaintained since 2016 and should not be used." A maintained fork lives at codeberg.org/gumbo-parser/gumbo-parser. The Google repository is a museum piece.

No JavaScript execution here comes out of the box.

C# / .NET

HTTP clients: the built-in HttpClient, with the usual caution against instantiating it per request; RestSharp for a friendlier REST surface.

Parsers: HtmlAgilityPack 1.12.4 (October 2025), 346 million downloads, tolerant of broken markup and XPath-capable; AngleSharp 1.7.1 (5 August 2026), which follows the W3C and WHATWG specifications and gives querySelector semantics identical to a browser's. AngleSharp is the more actively developed by some margin.

Browser automation: PuppeteerSharp 25.5.0 (5 August 2026) tracks upstream Puppeteer version for version, which is rare in a port; Playwright for .NET is Microsoft's own.

Frameworks: DotnetSpider is the Scrapy-shaped option. Abot is a configurable crawler whose GitHub repository lists no published releases at all, so read the NuGet history first.

Java

HTTP clients: Apache HttpClient 5, and OkHttp 5.3.1 (16 November 2025), which since 5.0 ships separate JVM and Android artifacts.

Parsers: Jsoup 1.23.1 (30 July 2026) is the whole story in Java. Parse, CSS-select, clean, and it fetches too. Quarterly releases.

Browser automation: HtmlUnit 5.4.0 (7 August 2026) is a headless browser written in Java, running Rhino for JavaScript, with a test suite exercising jQuery, htmx, MooTools and Dojo. Its Maven coordinates moved to org.htmlunit; the old net.sourceforge.htmlunit group is stale. Selenium and Playwright for Java drive real browsers when Rhino is not enough, which a 2026 single-page app usually is.

Frameworks: WebMagic 1.0.3 (10 February 2025) is the Scrapy-style option. Crawler4j is still recommended by name in roundups; its last release is 4.4.0 from 27 March 2018.

Ruby

HTTP clients: Faraday for middleware and adapter swapping; HTTParty when you want three lines.

Parsers: Nokogiri 1.19.4 (18 June 2026), past 1.2 billion downloads, CSS and XPath, and the library that makes Ruby a reasonable scraping language at all. It ships libxml2, so the maintenance note above applies to it directly.

Browser automation: Ferrum 0.17.2 (24 March 2026) drives Chrome over CDP with no WebDriver, ChromeDriver or Selenium in the chain. Watir is the older Selenium wrapper, and its last gem release is 7.3.0 from 4 August 2023. Check what Selenium version it pins before building on it.

Navigation and frameworks: Mechanize 2.14.0 (5 January 2025) still handles cookies, redirects, links and forms without a browser. Kimurai 2.2.0 (27 January 2026) is the Scrapy-style framework, now on Ruby 3.2, and that release is a reversal: the gem was dormant for years and most write-ups still call it abandoned. Vessel, from the Ferrum authors, is the other option.

Rust

HTTP clients: reqwest 0.13.4 (25 May 2026), 631 million downloads, async and the default; ureq 3.4.0 for blocking code with a small dependency tree.

Parsers: scraper 0.27.0, CSS selectors over Servo's html5ever; the select crate has not published since 0.6.1 in March 2025.

Browser automation: fantoccini 0.22.1 and thirtyfour 0.37.4 over WebDriver; chromiumoxide 0.9.1 and headless_chrome 1.0.22 over CDP.

Frameworks: spider 2.53.4 (30 July 2026) publishes releases at a pace no other framework here matches.

Library comparison table

Read from PyPI, npm, crates.io, Packagist, RubyGems, NuGet or the project's own release feed on 10 August 2026.

Scrapy and Roach PHP parse static HTML alone and reach JavaScript through an add-on: scrapy-playwright (0.0.48, July 2026) and Browsershot.

Dead ends, renames and quiet shelves

• google/gumbo-parser, archived 21 January 2026. The fork at codeberg.org/gumbo-parser/gumbo-parser is the live one.
• Goutte, archived 1 April 2023 after becoming a proxy to Symfony BrowserKit. Any tutorial installing fabpot/goutte is dated.
• undetected-chromedriver, 3.5.5 in February 2024, no push since July 2025, 1,143 open issues. Its own author's successor is nodriver.
• puppeteer-extra and puppeteer-extra-plugin-stealth, both stopped at March 2023.
• Crawler4j (March 2018), Simple HTML DOM (November 2019), Watir (August 2023), Rod (July 2024, commits continue).
• Abot publishes no GitHub releases, and Ferret is mid-rewrite with v2 on the default branch.
• Kimurai goes the other way: dormant for years, then 2.2.0 in January 2026. Roundups calling it dead are behind.

How to assemble a working stack

• A static page, or an API you found in the network tab. An HTTP client and a parser: requests plus parsel, Guzzle plus DomCrawler, net/http plus goquery. This covers more sites than people expect, and it is the only configuration that stays cheap at volume.
• Content built by JavaScript, with clicks and scrolling. Playwright first, in any language it supports. Puppeteer if you are in Node and care only about Chrome. Look for an embedded JSON state blob before you accept the cost.
• Thousands of pages on a schedule. A framework: Scrapy, Crawlee, Colly, Roach PHP. The queue, the deduplication and the resume-after-crash are the parts you would otherwise write badly twice.
• A site that blocks you. A separate layer, independent of the language you picked. TLS impersonation at the client (curl_cffi), a patched browser when JavaScript is being fingerprinted (patchright, camoufox, nodriver), and rotating residential proxies under both. Expect to spend more time here than on extraction.

Picking libraries is an afternoon. Keeping a large crawl alive across anti-bot changes, proxy churn and quarterly redesigns is the work, which is why the team that writes the scraper often ends up buying the output through a data-as-a-service arrangement instead. Either way, the version numbers on this page will be wrong in six months. The habit of checking them will not.

Changelog

August 2026. Rechecked every package against its registry or release feed. Added our own parser benchmark and corrected the claim that selectolax is fastest at scale: it wins on parsing, loses on selection. Rewrote the Scrapling entry, which described a framework as a parser. Flagged Gumbo's archival, Goutte's, and the stalled state of puppeteer-extra, undetected-chromedriver, Crawler4j, Simple HTML DOM, Watir and Rod. Noted httpx's twenty-month silence against requests shipping inline types. Added zendriver, patchright, camoufox, the curl_cffi free-versus-paid split, Chrome's removal of old headless, Cloudflare's July 2025 default, the 2026 Thales figures and the libxml2 maintenance change.