Selenium Web Scraping: Pros and Cons of WebDriver

Ten thousand pages pulled through a real browser cost roughly $64 in residential proxy traffic. The same ten thousand pages fetched as raw HTML cost about 80 cents. Compute for the browser run adds fifteen cents, and nobody notices it. Those figures come from median page weights and published per-gigabyte proxy prices, both read on 13 August 2026, and the arithmetic is in the next section. Speed is what every pros-and-cons list argues about. Bytes are what most projects actually pay for.

Selenium WebDriver was built to test web applications. It became a scraping tool by adoption rather than by design, and every entry in its ledger traces back to one fact: you are running an entire browser, from outside, over a protocol written for test suites. This article is the ledger on its own, without the code. What you get, what it costs, and which of the standard objections have quietly expired. For the setup, the measurements and the extraction patterns, see our in-depth Selenium guide. For the ecosystem behind the name, the IDE, Grid and the history, see what Selenium WebDriver is.

Everything below was re-checked against Selenium 4.47.0, released on 10 August 2026 in all five official bindings on the same day, and against vendor and standards documentation read on 13 August 2026.

The bill nobody itemises

A browser downloads the page. The whole page: scripts, fonts, hero images, ad tags, the analytics beacon. An HTTP client downloads the document and stops. The gap between those two behaviours is the single largest cost line in browser-based scraping, and it almost never appears in a comparison table.

The 2025 Web Almanac page weight chapter puts the median desktop page at 2,862 KB transferred, of which 1,058 KB is images and 697 KB is JavaScript. The median HTML document is 35 KB. Residential proxy traffic on Webshare runs $3.50/GB at 1 GB and $2.25/GB on the 100 GB monthly tier; Oxylabs advertises $6/GB at 5 GB, $4/GB at 125 GB and $2.50/GB on a 1 TB commitment. Take $2.25/GB, the friendlier end of that range, and a thousand pages look like this:

Now the compute, which is the number people worry about instead. A DigitalOcean basic droplet with 2 vCPUs and 4 GiB is $24 a month, or $0.03571 an hour, with 4,000 GiB of transfer included. Assume two concurrent headless browsers on those vCPUs and three seconds a page, which is generous rather than optimistic. That is 2,400 pages an hour and 1.5 cents per thousand pages.

Bandwidth through residential exits costs about 430 times what the machine costs. The datacenter bytes are effectively free and the residential bytes are not, which tells you what you are buying: not bandwidth, but whose address the request leaves from. Any honest comparison of scraping tools is mostly a comparison of how many bytes each one drags through a metered exit.

Two qualifications keep this from being a slogan. Datacenter proxies cost far less per unit and are the first thing a defended site blocks, so the residential price is the one that applies to targets worth a browser. And blocking images is one line of Chrome options that removes about a third of the payload without touching the DOM you came for. Do that before anything else on this page.

The advantages, re-checked

It runs the page, and that is the whole of it

The reason people reach for Selenium is unchanged. On a single-page app, or any site that assembles content from XHR calls after the initial HTML, a requests-style fetch returns a shell. A browser runs the scripts, fires the calls, builds the final DOM, and hands you what a person would see. When the underlying API calls are obfuscated, signed by the page's own JavaScript, or spread across a dozen chained requests, letting the browser run the page is far cheaper in engineering time than unpicking it.

One retraction, and it matters more than the rest of this article. The earlier version of this piece said that because WebDriver drives a real browser, its traffic looks like ordinary browsing, and called that Selenium's single biggest selling point on sites that scrutinise their traffic. That is wrong, and it has been wrong for years. Running a real browser buys you rendering. It does not buy you anonymity, because the automation is announced by the protocol itself, before your code runs. Rendering and trust are two different purchases, and only one of them comes with the tool.

It drives the browser people actually have

This is the underrated one. Selenium launches the Chrome, Firefox, Edge or Safari already installed on the machine, through a driver published by that browser's own project. Playwright downloads its own builds instead: open-source Chromium by default, a Firefox that merely "matches the recent Firefox Stable build", and a WebKit "derived from the latest WebKit main branch sources". Its channel option exists precisely so you can opt back into "branded Google Chrome and Microsoft Edge browsers available on the machine".

For testing, bundled builds are a feature, because they pin the environment. For scraping they are a liability: a fingerprint taken from a build nobody runs is a fingerprint that stands out. Safari makes the point sharply. Selenium's own documentation notes that "unlike Chromium and Firefox drivers, the safaridriver is installed with the Operating System". One safaridriver --enable and you are driving the browser a real Mac user has.

It is a specification, not a product

WebDriver became a W3C Recommendation on 5 June 2018, and the living specification is maintained by the Browser Testing and Tools Working Group with the browser vendors in the room. That has a practical consequence people only appreciate after it bites them elsewhere: no single vendor can retire the protocol under you.

Compare the alternative. The Chrome DevTools Protocol is a Chrome implementation detail, and when Mozilla stopped enabling CDP by default in Firefox 129, everything built on it for Firefox stopped working. Selenium deprecated Firefox CDP in 4.27 and 4.28 and removed it in 4.29. A protocol maintained by one vendor is a dependency on that vendor's roadmap. A W3C specification with four implementations is not.

Five bindings, and that is still a moat

Selenium publishes official bindings for Java, Python, C#, JavaScript and Ruby, and 4.47.0 shipped for all five on 10 August 2026. Playwright lists four official languages: JavaScript and TypeScript, Python, Java, .NET. No Ruby, no PHP. Puppeteer is Node.

If your shop writes Java or C#, that gap is the whole decision, and it is why Selenium remains standard in large enterprises long after the scraping crowd moved on. The language-specific walkthroughs cover what changes between them: Selenium with Java end to end, six WebDriver tips in Java for the patterns that survive real sites, and a worked C# example.

It acts, not just reads

Selenium clicks, types, scrolls, submits, handles dropdowns and waits for elements. That is what reaches content behind logins, infinite scroll and "load more" buttons, and an HTTP client cannot fake it without rebuilding the site's client-side state machine by hand. Screenshots come free with a rendering engine, and they are worth more for debugging a selector that broke at three in the morning than for anything visual.

The drawbacks, re-checked

Slow, and a good share of that is self-inflicted

The browser floor is real. Loading a page means waiting for scripts, stylesheets and fonts before you can touch an element, and by default driver.get() does not return until the load event fires. Against an HTTP request and a parser, that is an order of magnitude per page, sometimes two.

What most write-ups miss is that the second order of magnitude is usually the scraper's own doing. Every find_element and every .text is a separate HTTP round trip to the driver process, so a list page read element by element pays that toll hundreds of times. Our measurements on a 600-item page are in the in-depth guide: the same 600 values took 3.9 seconds field by field and 5.2 milliseconds in one script call. The browser costs what it costs. The 750x on top of it is a coding pattern.

Heavy where it is metered

Each worker is a browser: a few hundred megabytes of resident memory, real CPU, a driver process beside it, and a profile directory that grows. Run four to make up for the latency and you are running four browsers, which is why serious jobs live in containers behind a pool such as Selenium Grid.

The memory is annoying and cheap. The bandwidth is neither, and the first section priced it. When someone says Selenium does not scale, this is what they mean, whether or not they have done the sums.

Detection got worse, and the fixes went stale

An automated browser announces itself before your code runs, and the anti-bot industry is built on the difference. Property checks such as navigator.webdriver are the shallow end; current systems read protocol side effects, TLS and HTTP/2 fingerprints, and behaviour. The markers themselves, including which famous ones no longer exist, are enumerated in the in-depth guide, and the systems that consume them in our anti-bot analysis. Two points belong on the cost side of the ledger.

The first is that running the page cuts both ways. A browser that executes site JavaScript to pass a challenge also fires the site's analytics. You show up in their reports as a session shaped nothing like a reader, and no vendor product was needed to notice you.

The second is the maintenance bill, and here the previous version of this article was giving bad advice. It named three packages as countermeasures. Read from their package pages on 13 August 2026, undetected-chromedriver last released 3.5.5 in February 2024, selenium-stealth last released 1.0.6 in November 2020, and selenium-wire last released 5.1.0 on 15 October 2022, with a PyPI page that still reads as though it were current. None of them is tracking what shipped this year. Anything built on them is a snapshot of the arms race as it stood when the maintainer moved on.

Nothing in the box is a scraper

Selenium gives you a browser on a leash. It does not give you a queue, a retry policy, a rate limiter, a deduplicating URL frontier, a parser, or a storage layer. Frameworks like Scrapy ship all of that and no browser; Selenium ships the browser and none of that. Building the missing half is the part of a Selenium project that overruns.

The sharpest example is proxy authentication. The WebDriver specification's proxy capability defines exactly seven keys: proxyType, proxyAutoconfigUrl, httpProxy, noProxy, sslProxy, socksProxy and socksVersion. There is no username and no password anywhere in it. Nearly every commercial proxy provider authenticates by credentials rather than by allow-listed address, so the workarounds have been an on-the-fly browser extension that answers the auth dialog, a local forwarding proxy, or a package like selenium-wire that has not shipped since 2022. A protocol written for test suites had no reason to anticipate this, and for eight years it did not.

What actually changed, with dates

The trade-offs are stable. Four of the specific complaints in circulation are not, and three of them died between 2022 and 2026.

• Driver management, gone. Selenium 4.6 shipped Selenium Manager as Beta 1 on 4 November 2022, described by the project as "a new tool that helps to get a working environment to run Selenium out of the box". It detects the browser, works out the matching driver, downloads it and caches it. The ritual of hand-placing chromedriver on PATH is over, with one caveat: it resolves versions over the network, so a locked-down build agent can still hand you a mismatch at runtime.
• The headless split, gone, and the advice about it is now wrong. Chrome's old headless really was a different browser. Since Chrome 132.0.6793.0 it ships separately as chrome-headless-shell, and plain --headless starts the same binary real users run. The --headless=new flag that tutorials still copy, this article's earlier version included, adds nothing to a current Chrome. Headless is still fingerprintable in other ways, so none of this is a stealth win.
• Internet Explorer, mostly gone, and the "mostly" is the correction. The IE11 desktop application went out of support on 15 June 2022. Microsoft "is committed to supporting Internet Explorer mode in Microsoft Edge through at least 2029", and Selenium still ships a driver for it. Saying IE is dead and buried was too tidy. It is dead as a scraping target and alive as a compliance obligation.
• The detection surface, worse. This is the one that moved against Selenium, and it moved twice: from property checks a script could patch, to protocol side effects and network-layer fingerprints that no client-side patch reaches.

WebDriver BiDi, and why it changes the argument

The oldest structural complaint about WebDriver is that it is one-directional. You send a command, you get a response, and the browser cannot tell you anything you did not ask for. No event stream, no console messages, no network visibility. That is why an entire generation of Selenium scrapers reached for the Chrome DevTools Protocol, and why selenium-wire existed.

WebDriver BiDi is the fix, and it is a joint specification rather than one vendor's protocol. The Editor's Draft of 5 August 2026 defines ten modules over a WebSocket connection: session, browser, browsingContext, emulation, network, script, storage, log, input and webExtension. Selenium's own BiDi documentation states the direction without hedging: the project "is updating its entire implementation from WebDriver Classic to WebDriver BiDi (while maintaining backwards compatibility as much as possible)".

Three of the items in this article's drawback list are addressed there directly. The network module defines network.addIntercept and network.continueRequest for blocking and rewriting requests, which is how you stop paying for images and ad tags in a browser-neutral way. It defines network.addDataCollector and network.getData, which is response bodies without CDP. And it defines network.continueWithAuth, which is the proxy credential problem that Classic left unsolved for eight years.

The catch is the ordinary one for a moving specification: coverage lags the document. It is an Editor's Draft, not a Recommendation, the bindings expose different amounts of it, and Selenium's Java BiDi classes were still marked beta as of 4.46. The direction is not in doubt, because Selenium is not the only project moving that way. Puppeteer supports BiDi for Chrome and Firefox, and "when launching Firefox with Puppeteer, the WebDriver BiDi Protocol is enabled by default". Two competing automation tools converging on one W3C protocol makes the choice between them smaller than it was.

The alternatives, with version numbers

Release dates below were read from each project's package index on 13 August 2026. They are here as signs of life, not as a ranking.

The one to notice is curl_cffi, because it dissolves a false choice. Plenty of sites need no browser and merely reject Python's TLS handshake. A client that reproduces a real browser handshake gets those pages at HTTP cost.

On the browser side, the honest comparison is narrower than the marketing. Playwright's advantages are architectural: one persistent connection instead of a round trip per command, auto-waiting that removes most flaky sleeps, network interception that was first-class from day one, and a genuinely async API. The speed multiples quoted in blog posts are another matter. They trace back to vendor and tutorial benchmarks rather than to any independent measurement, and a Selenium script that crosses the driver boundary once per page closes most of the gap. Pick on the API and the ecosystem, which you can verify yourself.

For the record-and-playback end of the same family, where the trade-offs are different again, see Selenium IDE and web scraping.

Four questions that settle it

Is the value in the HTML or in a JSON call? Fetch the URL with curl and search the response for a price or a SKU. Then open DevTools, filter to Fetch/XHR and read what the front end is calling. Thirty seconds, and it resolves most pages described as JavaScript-heavy.

Do you have to act, or only to read? Logins, multi-step flows, infinite scroll and anything gated by a challenge that has to execute in a browser are the real Selenium territory. Reading is not.

How many pages, and out of whose address? This is the expensive question. At a hundred pages nothing above matters. At a hundred thousand through residential exits, the byte count is your budget, and every megabyte of images the browser fetches is money.

What is your team already shipping? A Java or C# shop gets a working Selenium scraper this week and spends a month adopting a new runtime. That is a legitimate reason to choose the slower tool, and it is the one enterprises actually use.

Three of the four are free to answer. The third one is the invoice.

Where the choice stops being reversible

Tool choices are cheap to revisit until they are not, and the boundary is not where people expect.

What ports for free. Selectors, XPath expressions, the parsing layer, everything downstream of extraction. If your code takes page_source and hands it to a parser, that half moves to any tool that can produce HTML, including one that is not a browser.

What does not port. Waits and synchronisation, frame and shadow-DOM handling, session lifecycle, and the whole apparatus of detection workarounds, which is tool-specific by construction. Anything written against CDP is stranded twice over: it was never portable across browsers, and Selenium removed Firefox CDP in 4.29.

What is genuinely locked in. A Grid deployment, a CI estate, and a language choice made by a team rather than by a project. These are org-shaped, not code-shaped, and they outlive several rounds of tooling fashion.

And the decision itself goes stale. The choice that was correct at a hundred pages a day is often wrong at a hundred thousand, because the cost that dominated was engineering time and the cost that dominates now is traffic. Re-run the four questions when the volume changes by an order of magnitude, not when a new tool is announced. If browser-based extraction at that scale stops being something you want to own, it can be bought as a managed scraping service or, when what you need is the dataset rather than the crawler, as data delivered on a schedule.

The bottom line

Selenium is alive, standardised and unusually well documented, and 4.47.0 landing across five languages on one day in August 2026 settles the "is it dying" question. Its advantages are a real browser's: it runs the page, it acts on the page, it drives the browser your target's users are running, and it does so from more languages than anything else in the category.

The bill arrives in bytes and in detection surface. Two of the classic complaints, driver management and the headless split, are obsolete. One, the round-trip tax, is a coding pattern you control. The remaining two are structural: a browser drags roughly eighty times the payload of an HTTP client through a metered proxy, and it announces its own automation before your first line runs. WebDriver BiDi is closing the protocol gaps, including the proxy-credential hole open since 2018. It closes none of the economics.

So use it deliberately. Check whether the page needs a browser before launching one, cross the driver boundary once per page instead of once per field, block the images, and price your proxies before your servers. When the page genuinely has to run, Selenium earns its keep. When it does not, the cheapest scraper is the one that never opened a window.