Best Programming Language for Web Scraping: Full Comparison

Seven stacks parsed the same 182,884-byte page for this article. Rust finished in 4.1 ms, Python with lxml in 4.1 ms, PHP's built-in DOM in 5.7 ms, Node with cheerio in 35.4 ms, Python with BeautifulSoup in 73.1 ms. Fetching that same page over HTTPS took 55 ms on a fast connection with no proxy in the way. The slowest parser on the list costs less than two network round trips. The fastest one saves you time you cannot spend on anything.

That is the shape of the whole question. The best programming language for web scraping is a real decision with real consequences, and it is the third most important one you make. Two others come first, and each of them moves your costs by a factor of ten or more.

This article is an overview rather than a tutorial, and each language links to a hands-on walkthrough. Every version number, release date and price below was read from the project's own repository, package registry or pricing page on 10 August 2026. Several tools that older comparisons still recommend, this article's own earlier version included, turned out to have stopped moving years ago.

What the choice actually decides

Three things determine what a scraping project costs, in this order.

Do you need a browser. A headless Chrome instance costs hundreds of megabytes of RAM and a whole process per page in flight, against a few kilobytes and the 55 ms we measured for an HTTP request. You do not have to take the size of that gap on faith from us, because the vendors who sell both price them side by side. Zyte API charges $0.13 to $1.27 per 1,000 requests for an HTTP response body and $1.01 to $16.08 per 1,000 for a browser-rendered one, tier for tier, on pay-as-you-go pricing read on 10 August 2026. That is between eight and thirteen times the price for the same page. Scrapfly prices a plain datacenter fetch at 1 credit and a JavaScript render at 5. Nobody charges that spread out of malice; it is what the resource actually costs them.

Does the target fight back. The 2026 Thales Bad Bot Report, covering 2025 traffic, puts automated clients at 53% of all internet traffic and bad bots specifically at 40%, up three points year over year, with 17.2 trillion bad-bot requests blocked. Sites have responded, and the response is not written in your language or ours. It is TLS and HTTP/2 fingerprinting, behavioural scoring and IP reputation. Against that, the language you write in is close to irrelevant; the transport library underneath it is not.

Which language. After those two, this one. It decides how pleasant the work is, how much code you write by hand, and how the thing behaves when you point it at ten thousand URLs instead of ten.

There is a fourth question hiding inside the third, and most comparisons skip it: whether the browser-automation project you need has an official binding for your language at all. That question eliminates more candidates than performance ever will, and it gets its own section below.

The measurement

We ran the same extraction task in four languages and seven parser configurations. The point was not to crown a winner but to find out how much of the total the parser is worth at all.

Method. A single 2-vCPU Linux box, 8 GB of RAM. Two input documents. The first is a real page, https://pypi.org/project/requests/, saved on 10 August 2026: 182,884 bytes, 2,312 tags, 288 anchor elements of which 284 carry an href. The second is a synthetic listing built to look like a large catalogue page, 4,733,537 bytes containing 20,000 product blocks, each with a name, a price, a link and a paragraph of text. The task on the real page: parse it, select every a[href], read the href and the text of each. Every stack returned exactly 284 links, which is the check that they were doing the same work. The task on the synthetic page: parse, select div.product span.price, read the text, and all stacks returned 20,000. Timings are the median of 30 runs on the small page and of fewer runs on the large one, measured inside the process, excluding file reads.

Versions under test: Python 3.11.15 with lxml 6.1.0, selectolax 0.4.11 and BeautifulSoup 4.14.3; Node.js 22.22.2 with cheerio 1.2.0; PHP 8.4.21 with the bundled DOM extension on libxml 2.9.14; Rust 1.95.0 with scraper 0.27.0 compiled in release mode.

Read the first column again. Between compiled Rust and PHP's built-in parser, supposedly opposite ends of the performance world, there are 1.6 ms. Between the fastest and the slowest way of parsing in Python there are 69 ms, forty-three times that spread. The parser you pick inside a language matters more than the language you picked. Nobody writes that on a comparison page, because it is not a comparison of languages.

Now the part that changes how you write selectors. On the 4.7 MB document, lxml driven through its CSS layer with the descendant combinator div.product span.price took 32.9 seconds. The same document, the same library, the same 20,000 nodes, with the child combinator div.product > span.price, took 87.7 ms. One character, 373 times the runtime.

The cause is visible in one line. lxml.cssselect translates a descendant combinator into XPath containing /descendant-or-self::*/, and that step is evaluated against every node under every match. The child combinator translates to a plain /. A direct XPath, //div[@class='product']/span[@class='price'], took 82.0 ms, confirming that lxml itself is not slow; the translation is.

Across ten thousand pages of that size, the descendant form costs 91 hours and the child form costs 15 minutes. That is not a language problem, a hardware problem or a scale problem. It is a space bar.

Measured on one machine, on two documents, with the versions listed above. Your absolute numbers will differ. The ratios are what to take away, and the 373x reproduces in ten lines of Python.

What we compare on

Every language below gets looked at through the same five questions, plus one the original version of this comparison left out.

• Learning curve. How much you write by hand before the first row of data lands.
• Concurrency and speed. Behaviour at thousands of parallel requests, and memory at that point.
• JavaScript execution. Whether it can render a page the way a browser does, which SPAs, infinite scroll and lazy-loaded content require.
• Page interaction. Clicking, scrolling, filling and submitting forms, waiting for elements.
• Getting past defenses. How easily it can present as a real browser: TLS fingerprint (JA3 and now JA4), HTTP/2 fingerprint, headless markers, proxy handling.
• Official binding support. Whether Playwright, Selenium and Puppeteer ship a binding for the language, or whether you are on a third-party port.

One rule cuts across all six. Light HTTP clients are fast and cheap and do not run JavaScript. Browser-driven tools run JavaScript, click and scroll, and cost hundreds of megabytes per instance. Almost every language has both, and the interesting question is never which language but which half of it you need.

Which languages the browser tools actually support

If your target needs a browser, your shortlist is already made, and not by you.

Playwright ships official bindings for exactly four languages: JavaScript and TypeScript, Python, Java, and .NET. That list is on its own documentation and it has not grown. Selenium ships official bindings for Java, Python, C#, Ruby and JavaScript, with Kotlin documented as "use the Java bindings". Puppeteer is a Node project with no official port anywhere else.

Everything outside those lists is a community binding, and community bindings are a different risk profile rather than a disaster. Go talks to Chrome through chromedp over the DevTools Protocol directly and does it well. PHP has Symfony Panther and chrome-php. .NET has PuppeteerSharp, which is a port rather than an official binding and which nonetheless shipped 25.5.0 on 5 August 2026, one day after upstream Puppeteer shipped 25.5.0. That is not what an abandoned port looks like.

The practical reading: Python, JavaScript, Java and C# are the four languages where the browser layer is somebody else's maintenance burden. In Go, PHP, Ruby, Rust and C++ it is partly yours.

Python

Still the default, and the ecosystem is the reason. There is a maintained tool for every step, and the gap between the first line of code and the first row of data is shorter than anywhere else.

Main tools:

• requests / httpx: HTTP requests. Note the state of httpx before you plan around it: the stable line has not moved since 0.28.1 in December 2024, with 1.0 development builds published through September 2025 and no stable 1.0 as of August 2026. It works; it is not where the momentum is.
• BeautifulSoup 4.15.0, lxml 6.1.1, parsel 1.11.0, selectolax 0.4.11: HTML parsing. The benchmark above is the argument for reaching past BeautifulSoup once volume matters.
• Scrapy 2.17.0, 7 July 2026: request queues, retries, pipelines, the whole crawl. Recent releases added official Python 3.14 support in 2.16.0 and an experimental path that runs without a Twisted reactor at all, with an HttpxDownloadHandler that now speaks HTTP/2 and SOCKS proxies.
• Selenium 4.46.0, 11 July 2026 and Playwright 1.62.0, 31 July 2026: driving a real browser. Playwright is the better default for new work: async, auto-waiting, a smaller API surface.
• curl_cffi 0.16.0, 1 August 2026: TLS and HTTP/2 fingerprint impersonation without launching a browser, with HTTP/3 fingerprints added in 0.15.0. This is the single highest-leverage package in Python scraping and it has no equivalent in most other languages.
• Crawlee for Python 1.9.1, 6 August 2026 and nodriver 0.50.3: frameworks and stealth drivers that switch between plain HTTP and a browser per request.

Pros: the shortest path from nothing to working; readable code that a colleague can maintain; coverage from static pages to aggressive defenses without changing language; the largest body of worked examples in existence. Playwright and Selenium handle JavaScript, clicks and forms. Scrapy handles industrial volume.

The GIL objection needs updating. Every comparison written before late 2025, ours included, listed the global interpreter lock as Python's structural weakness. PEP 779 was accepted on 16 June 2025 and free-threaded Python became officially supported in Python 3.14, released 7 October 2025. It is no longer experimental. The caveats are real and worth stating plainly: it is supported, not default, so you install a separate free-threaded build; the acceptance criteria allowed up to a 15% single-thread slowdown and 20% more memory; and a C extension has to declare itself compatible or the interpreter re-enables the lock. For scraping specifically the change matters less than it sounds, because scraping is I/O-bound and asyncio already solved that. It matters for the CPU half: parsing and post-processing across cores in one process, without the serialisation tax of multiprocessing.

Cons: raw parsing is slower than compiled languages, though by 1 to 2 ms on a real page rather than by the order of magnitude the phrase implies; browser tools are memory-hungry, which is a property of Chrome rather than of Python; and the sheer number of options means the wrong one is always within reach, as the 55 ms BeautifulSoup row shows.

→ Detailed tutorial with an example: Web scraping with Python

JavaScript / Node.js

The natural choice when the target is JavaScript-first, and increasingly the choice when it is not, because the browser-automation projects live here first and get ported outward.

Main tools:

• axios or native fetch with cheerio 1.2.0, January 2026: requests plus jQuery-shaped parsing of static HTML.
• Puppeteer 25.6.0, 11 August 2026: driving Chrome over the DevTools Protocol.
• Playwright 1.62.1, 30 July 2026: the same job across Chromium, Firefox and WebKit.
• Crawlee 3.18.0, 4 August 2026: an Apify framework that switches between a cheap HTTP request and a full browser per request, which is the correct architecture for a mixed site and a pain to build yourself.
• jsdom 30.0.1, July 2026: DOM emulation without a browser process, for pages whose scripts are simple enough to survive it.

Node 24 "Krypton" is the active LTS as of August 2026, with Node 26 as current.

Pros: first-class headless browser support, since this is where Puppeteer and Playwright are written rather than ported; an event loop that handles thousands of concurrent I/O-bound requests without ceremony; one language across the front end you are reading and the scraper reading it, which genuinely helps when you are reverse-engineering a site's own XHR calls.

Cons: cheerio parses but does not execute, and at 35.4 ms on a 183 KB page it is the slowest non-BeautifulSoup option we measured, roughly nine times lxml; CPU-heavy post-processing needs worker threads; and out-of-the-box TLS fingerprint control is weaker than Python's curl_cffi.

One correction that matters. Round-ups still recommend puppeteer-extra-plugin-stealth as the standard evasion layer. Its last release on npm is 2.11.2, published 1 March 2023, and the parent puppeteer-extra last shipped the same day. Three and a half years is a long time in a field where Chrome ships every four weeks. The packages that have moved since are rebrowser-puppeteer, puppeteer-real-browser and patchright, the last of which released 1.61.1 in June 2026. We are not endorsing any of them; we are pointing out that the one everybody names has not shipped a line since early 2023.

→ Detailed tutorial with an example: Web scraping with JavaScript / Node.js

PHP

Better than its reputation for this work, and the benchmark is the reason: PHP's bundled DOM extension parsed and extracted in 5.7 ms, within 1.6 ms of compiled Rust, with nothing installed. PHP 8.5 shipped 20 November 2025; 8.4 and 8.5 are both in active support.

Main tools:

• native cURL or Guzzle: HTTP.
• DOMDocument with DOMXPath, built in, no dependency, and quick.
• Symfony DomCrawler with BrowserKit and its HttpBrowser class: navigation plus parsing by CSS selectors and XPath.
• Symfony Panther 2.4.0, January 2025, php-webdriver, chrome-php: real browsers for dynamic content.
• DiDOM: a lightweight parser with a friendlier API than raw DOM.
• Async, if you need it: ReactPHP and Amp.

Two libraries to stop recommending. Goutte was archived on 1 April 2023, and its own README says so in one sentence: "As of v4, Goutte became a simple proxy to the HttpBrowser class from the Symfony BrowserKit component." Use BrowserKit directly. Simple HTML DOM is the more common mistake, because it still appears in tutorials written this year: its last release on SourceForge is 1.9.1, dated 20 October 2019. That predates PHP 8 entirely. DiDOM or the built-in DOM extension do the same job on a supported runtime.

Pros: cURL is in every installation; the parser is fast and already there; Guzzle with DomCrawler is a short, legible path for static pages; Panther drives a real browser when you need one.

Cons: concurrency is the weak point, and the async add-ons are real but are not what the average PHP codebase is built around; the scraping ecosystem is thinner than Python's or Node's; Panther is heavy; and PHP has no equivalent of curl_cffi, so TLS fingerprint work means dropping to a patched curl or going through an external service.

→ Detailed tutorial with an example: Web scraping with PHP

Go (Golang)

The choice when the bill is measured in machines rather than hours. Go compiles to one binary, and goroutines let a modest box hold a very large queue. Go 1.26 landed 10 February 2026.

Main tools:

• net/http from the standard library, which is genuinely enough for a lot of jobs.
• Colly: the crawl framework, with queues, caching, limits and retries. Its own site claims, verbatim, "Fast (>1k request/sec on a single core)", which is the project's own figure and not one we reproduced.
• goquery v1.12.0, 15 March 2026: jQuery-style parsing.
• chromedp v0.16.0, 14 July 2026: Chrome over the DevTools Protocol. This is the one to reach for.
• Rod: browser automation with a friendlier API.

Liveness, since Go tooling attracts obituaries. Colly is the one people write off, and the write-off is wrong: v2.3.0 was published to the module proxy on 4 December 2025, though the GitHub releases page stops at v2.2.0, which is why the project looks dead if you check the wrong page. Rod is the opposite case: its latest tagged version is v0.116.2, 12 July 2024, over two years old at the time of writing. It works, and a stable API on a stable protocol can legitimately stand still, but calling it "the modern alternative" without that date attached is how a comparison goes stale. Ferret is worth knowing about and not worth starting on: its most recent release is v2.0.0-alpha.36 from 24 July 2024, and v2 has been in alpha ever since. Surf, which appears in most Go scraping lists including our previous one, has no dated release we could verify and no recent activity we could confirm.

Pros: concurrency for very little code and very little memory; Colly for high-volume crawling of static pages; chromedp for the dynamic half; one static binary to deploy; and the best story anywhere for TLS fingerprint control at the transport level, because utls lets you construct a ClientHello that matches a real browser.

Cons: no official Playwright or Selenium binding, so the browser layer is chromedp or a community port; a smaller ecosystem, which means more code you write and own; fewer turnkey answers for the hardest anti-bot systems.

→ Detailed tutorial with an example: Web scraping with Go

The C family: C, C++, and C

Three languages that share four characters of name and almost nothing else. The previous version of this article managed to head this section "C, C++, and C", which is a typo that survived because nobody reads their own headings. Here is what each is actually for.

C

Almost nobody writes a scraper in C, and the reason is not difficulty but redundancy: the libraries everyone else calls are already written in C. libcurl is underneath Python's curl_cffi, PHP's cURL extension, and half the HTTP clients in this article. libxml2 is underneath lxml and underneath PHP's DOMDocument; the PHP build used for the benchmark above reports libxml 2.9.14. C's role here is the layer below scraping, not scraping.

That shared foundation is worth one paragraph of attention. On 15 September 2025 libxml2's maintainer, Nick Wellnhofer, posted on the GNOME developer forum: "I'm stepping down as maintainer of libxml2 which means that this project is more or less unmaintained for now," committing only to fix regressions in the 2.15 release through the end of 2025. Whether a permanent successor has been confirmed since is not something we could establish from outside the project. If you are choosing a language on the strength of its parser, it is worth knowing that several of the candidates share one, and that one is short of hands.

C++

For pipelines where the per-page cost is multiplied by a number large enough to buy hardware.

Main tools: libcurl or CPR (1.14.2, February 2025), a libcurl wrapper shaped like Python's requests, for HTTP; libxml2, pugixml (1.16, June 2026, twenty years old and still shipping) and lexbor (v3.0.0, March 2025) for parsing.

Gumbo is gone, and this is the correction that matters most in this section. Google's gumbo-parser was archived on 21 January 2026, and the README says, in the project's own words, "This project has been unmaintained since 2016 and should not be used." It is still recommended in current articles, ours previously included. A community fork continues on Codeberg. For new work, lexbor is the live HTML5-conformant option, and it is the same engine that backs Python's selectolax, which is how a Python package ended up 125 times faster than lxml's CSS layer on our large document.

Pros: the lowest achievable per-page cost; total control over memory and connections; a single deployable artifact.

Cons: no JavaScript execution without embedding a browser engine or driving an external one; no official binding to any browser-automation project; and a development cycle measured in weeks where Python's is measured in hours. Choose it when the arithmetic demands it and not before.

C# / .NET

Underrated for this work, and the download counts say so more clearly than any review.

Main tools:

• HtmlAgilityPack 1.12.4, October 2025, about 346 million downloads: forgiving parser with XPath support.
• AngleSharp 1.7.1, 5 August 2026, about 295 million downloads: standards-compliant HTML and CSS parsing with querySelector. The more actively developed of the two by a wide margin.
• PuppeteerSharp 25.5.0, 5 August 2026, Playwright for .NET (an official binding) and Selenium WebDriver: real browsers, all three current.

Two frameworks to skip. ScrapySharp was archived on 21 March 2026. DotnetSpider last shipped 5.1.7 in July 2024 and has roughly 114,000 total downloads across its entire history, against HtmlAgilityPack's 346 million. The gap is not a popularity contest; it is the difference between a library the ecosystem depends on and one it does not. In .NET the working pattern is a browser driver plus a parser, not a crawl framework.

Pros: static typing catches extraction bugs before the crawl starts; real threads and async/await with no interpreter lock to work around; processes that run for weeks without attention; and an official Playwright binding, which puts .NET in the four-language club above.

Cons: more ceremony than Python for a fifty-line job; a smaller community specifically around scraping, so you will read more source and fewer blog posts. Kotlin and Scala sit in a similar place on the JVM, where the Java walkthrough applies to them unchanged.

Additional languages

Java

Built for the pipeline that has to run for two years and be handed to someone else.

Main tools: Jsoup 1.23.1, 30 July 2026, still shipping parser speed and memory improvements after more than fifteen years; HtmlUnit 5.4.0, 7 August 2026, a headless browser written in Java with its own JavaScript engine; Selenium and Playwright for Java, both official bindings; Apache HttpClient for requests.

One detail that breaks builds quietly: HtmlUnit's Maven coordinates are now org.htmlunit:htmlunit. Copy-pasted snippets still carry the old net.sourceforge.htmlunit group, which resolves to a version frozen years ago.

Pros: the maturity is real, the threading model is honest, and both major browser-automation projects support it officially. Cons: verbose, and HtmlUnit's JavaScript support is close to but not the same as a real browser's, which surfaces on exactly the sites you most wanted it for.

→ Detailed tutorial with an example: Web scraping with Java

Ruby

Concise, fast to start, and with better maintenance than its quiet reputation suggests.

Main tools: Nokogiri 1.19.4, 18 June 2026, with about 1.24 billion downloads, one of the most used parsers in any language; Mechanize 2.14.0, January 2025, for navigation and forms; Ferrum 0.17.2, 23 March 2026, driving Chrome over CDP; and Selenium, which has an official Ruby binding. Watir still appears in every Ruby scraping list, this article's previous version included, but its last gem release is 7.3.0 from 4 August 2023, so treat it as settled rather than current.

Pros: a pleasant path from idea to working script, and a parser with a billion downloads behind it. Cons: no Playwright binding; concurrency is workable rather than good; and the number of people solving scraping problems in Ruby is small enough that you will be reading source rather than Stack Overflow.

→ Detailed tutorial with an example: Web scraping with Ruby

Rust

The fastest thing we measured, and the measurement is also the argument against reaching for it reflexively.

Main tools: reqwest 0.13.4 (HTTP), scraper 0.27.0 (CSS selectors, spiritually BeautifulSoup with types), tokio 1.53 (async runtime), fantoccini and thirtyfour (WebDriver), headless_chrome, and the spider crate for full crawls.

Rust tied lxml on the real page, 4.1 ms against 4.1 ms, and won the large one, 107 ms against 221 ms. On the page shape you will actually meet, there was nothing to win. On the pathological one it is a factor of two, against a Python configuration that took 32.9 seconds if you wrote the selector the obvious way. The honest summary: Rust removes an entire class of ways to be slow, and you pay for that in development time and in a browser layer with no official binding.

Pros: C++-level throughput without C++-level memory bugs; excellent concurrency on tokio; a compiled artifact that behaves the same on day 400 as on day 1. Cons: ownership and borrowing are a real tax on the first project; no official Playwright or Selenium binding; and for JavaScript-heavy targets you end up driving Chrome through a third-party crate or an external service.

→ Detailed tutorial with an example: Web scraping with Rust

R

Overlooked in scraping comparisons, and rightly first choice when the scrape is step one of an analysis rather than a product.

Main tools: rvest 1.0.5 (August 2024), httr2 for requests, polite for rate limiting and robots.txt, and chromote for a real browser.

The useful development since most comparisons were written is read_html_live(), added in rvest 1.0.4, which renders through chromote and returns a live document you can click and scroll before extracting. R is no longer static-only.

Pros: the data lands in a data frame and goes straight into modelling and plotting with no export step; polite makes considerate crawling the default rather than a virtue you have to remember. Cons: high-concurrency crawling is not what the runtime is for; the anti-bot toolbox is close to empty; and RSelenium, which older R tutorials still lead with, is no longer where the ecosystem points, since rvest's own live-page function goes through chromote instead.

Summary table

Versions and dates read from each project's own repository or registry on 10 August 2026.

What breaks at ten thousand pages

A script that works on twenty pages and a crawler that survives a night are different programs, and the difference is not the language.

Memory, and it is the browser's. A headless Chrome instance costs hundreds of megabytes and a persistent process. Twenty of them will exhaust a small box regardless of whether the code around them is Go or Python. This is the one place where the language does help, because Go and Rust let you hold a large queue in the same box that runs the browsers. It helps by making room for Chrome, not by being fast.

The retry you did not write. At ten thousand pages, a 1% failure rate is a hundred pages, and if failures are silent your dataset is quietly 99% complete with no record of which 1% is missing. Scrapy and Crawlee give you this for free. Every hand-rolled crawler in every language re-implements it, badly, on the second week.

The selector you wrote at 2 am. Our 373x finding is the cheapest lesson in this article. One wasted second per page across ten thousand pages is close to three hours. Thirty-three wasted seconds, which is what the descendant combinator cost on a 4.7 MB document, is 91 hours. Profile the extraction once, on the largest page you expect, before you scale it. It takes ten minutes.

Rate limits, which are the actual ceiling. Above a few requests per second per host you are into proxy pools, and the pool costs more than the compute. At that point the interesting number is not requests per second but successful requests per dollar, and every language reaches the same ceiling from below.

The page that changed. Sites redesign. A crawler that reads div.product > span.price breaks on the day the class becomes product-card. Static typing does not save you here; nothing does except an assertion that the row count is in the range you expect, and an alarm when it is not. Write that assertion on day one, in whatever language you chose.

Where the language stops mattering

The harder the target defends itself, the less the top half of this article matters.

Modern defenses inspect the TLS ClientHello and derive a JA3 or JA4 fingerprint, then compare it against the User-Agent you claim. Python's requests produces a ClientHello shaped by OpenSSL rather than by Chrome, whatever User-Agent header you set on top of it, and that mismatch on its own is enough to be scored as automation. The countermeasures are specific packages rather than languages: curl_cffi in Python, utls in Go, a patched curl elsewhere. Note the asymmetry, since it is the single most practical thing in this article: the tooling for fingerprint control is unevenly distributed, and it is the one axis where the language you chose really does close doors.

Then there is the arithmetic of doing it yourself. On ScrapingBee the documented credit costs are 1 for a plain request, 5 with JavaScript rendering, 10 for a premium proxy, 25 for premium plus JavaScript, and 75 for stealth. On the $49 Freelance plan with 250,000 credits, that is about 1.5 cents per stealth request, or a tenth of a cent per rendered page. ZenRows starts at $16 a month for 45,000 credits, and Scrapfly at $30 for 200,000. Those prices are read from the vendors' own pages on 10 August 2026, and they are what your own proxy pool, browser farm and fingerprint maintenance are competing against.

The honest framing is a build-versus-buy question rather than a technical one. Writing the scraper is a day. Keeping it alive against a target that actively resists is a standing commitment, and the language you wrote it in changes that commitment very little. Where the target fights back hard and the data matters more than the pipeline, a managed data extraction service delivers the finished rows regardless of the stack behind it, and a data-as-a-service arrangement removes the pipeline from your side of the wall entirely. Where the target is cooperative and the volume is modest, a hundred lines of Python and a cron entry will outlive the contract.

How to choose

• You want the shortest path to working, and the widest set of escape hatches when the site fights back. Python. It is the default for good reasons, and curl_cffi alone justifies it.
• The target is a full SPA, and you will spend your time reading its own JavaScript. Node. Puppeteer and Playwright are written here and ported outward, so you are always on the version without the port lag, and reading the site's bundle in the language you are writing is worth more than it sounds.
• Thousands of pages an hour, on hardware you are paying for. Go. Not because parsing is faster, which is worth milliseconds, but because ten thousand queued jobs and twenty browsers fit on one box.
• The scraper must run for two years and be handed to someone else. C# or Java. Both have official Playwright bindings, real threads, and a type system that turns a schema change into a compile error instead of a null. If the JVM is already where you live, start from the WebDriver walkthrough.
• PHP. If the codebase is already PHP, this is a legitimate choice rather than a compromise: the built-in parser measured 5.7 ms, which is faster than cheerio by a factor of six. Just size the concurrency honestly.
• Rust or C++. When per-page cost times volume exceeds the cost of the extra development weeks. That threshold is higher than most people estimate, and the benchmark above is why.
• Scraping is step one of an analysis. R, or Python with pandas. Keeping collection and modelling in one workflow saves more time than any parser will.

Four questions cover almost every real case. Does it need a browser. Does the site fight back. Is anyone maintaining this in two years. Does it already exist in a language your team writes. Answer those and the language usually names itself.

Corrections in this update

Written in August 2026 against the previous version of this comparison, which was accurate enough when published and had drifted in several specific ways.

• Gumbo removed. Google's gumbo-parser was archived on 21 January 2026 and describes itself as unmaintained since 2016. lexbor is the replacement recommendation.
• Simple HTML DOM removed. Last release 1.9.1, October 2019, predating PHP 8.
• ScrapySharp and DotnetSpider removed. Archived March 2026 and last shipped July 2024 respectively.
• The GIL entry rewritten. Free-threaded Python has been officially supported since 3.14, October 2025.
• Colly defended. v2.3.0 reached the module proxy in December 2025, though its GitHub releases page suggests otherwise.
• Rod and Ferret dated. v0.116.2 from July 2024 and a v2 alpha from July 2024, described as current in the previous version without those dates.
• The section heading "C, C++, and C" fixed, along with the two per-language tutorial links in it that both pointed at the Java article.
• Benchmarks added, because a comparison of languages that contains no measurements is a comparison of reputations.

In upcoming articles we will walk each language through a concrete target, including pagination, session handling and what to do when the first block arrives.