Selenium Web Scraping: In-Depth Guide with Examples

Selenium is a browser-automation tool originally built for testing web applications. But because it drives a real browser, it has long been a staple for web scraping — especially of sites that serve their content through JavaScript and won't open with a plain HTTP request.

This guide breaks down how Selenium works, how to use it across different languages, its strengths and weaknesses, and why anti-bot systems detect it so easily. If Selenium web scraping is on your radar, this is the deep dive on the tool itself.

If you're still choosing an approach, start with our overviews of what language to write a scraper in and web scraping libraries — those compare tools at a high level. This article is a deep dive into one of them.

What Selenium is and how it works

Selenium doesn't control the browser "from the inside" — it goes through a separate driver, over the WebDriver protocol (a W3C standard). The chain is always the same:

Your script  ──▶  Selenium binding  ──▶  Driver (chromedriver / geckodriver / …)  ──▶  Browser

• The binding is a library in your language (Python, Java, and so on) that translates calls like driver.get(url) into WebDriver commands.
• The driver is an executable middleman for a specific browser: chromedriver for Chrome, geckodriver for Firefox, msedgedriver for Edge.
• The browser is a real Chrome/Firefox/Edge that renders the page just as it would for an ordinary user.

Drivers used to be downloaded and dropped into PATH by hand, with you babysitting driver-vs-browser version compatibility. Since version 4.6, Selenium ships with Selenium Manager — it finds the browser, then picks and downloads the right driver automatically. As of this writing the current release is Selenium 4.45 (June 2026). In practice, that means a minimal working script today is literally three lines, with none of the old driver wrangling.

The key difference from HTTP scrapers

Ordinary libraries (requests + BeautifulSoup, Scrapy, and the like) simply download the HTML the server returned for the request. If the content is loaded by JavaScript in the browser, it won't be in that HTML.

Selenium, by contrast, launches a full browser: JavaScript executes, XHR/fetch requests fire, and the final DOM is built. The scraper sees exactly what a person sees. That's the main reason to choose Selenium — but you pay for it in resources and speed (more on that below).

Selenium across languages

One of Selenium's biggest advantages is official bindings for several languages at once. The API is nearly identical everywhere; only the syntax differs. That's convenient: a team can write its scraper in the same language as the rest of the project.

Officially supported: Python, Java, JavaScript (Node.js), C# (.NET), and Ruby. For PHP, Go, Rust, and others there are community bindings or wrappers, but they're maintained by the community rather than the core Selenium team.

Python

The most popular language for scraping in general, and for Selenium in particular: a low barrier to entry, a huge ecosystem (pandas, lxml, BeautifulSoup for post-processing), and a wealth of ready examples.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

title = driver.find_element(By.TAG_NAME, "h1").text
print(title)

driver.quit()

Pros: the largest community, ready-made anti-detection add-ons (undetected-chromedriver, selenium-stealth, SeleniumBase), fast development. Cons: the GIL limits real parallelism — to scale, you typically run multiple processes or use Selenium Grid.

Java

Java is Selenium's "native" language (the project historically grew around it) and the standard in large corporate QA teams.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class Scraper {
    public static void main(String[] args) {
        WebDriver driver = new ChromeDriver();
        driver.get("https://example.com");

        String title = driver.findElement(By.tagName("h1")).getText();
        System.out.println(title);

        driver.quit();
    }
}

Pros: true multithreading, high performance, mature build tooling (Maven/Gradle), a good fit for large, long-lived systems. Cons: verbose syntax, slower to write and prototype than Python.

JavaScript (Node.js)

A logical choice if your backend is already on Node. The async model (async/await) maps neatly onto waiting for elements to load.

const { Builder, By } = require("selenium-webdriver");

(async () => {
  const driver = await new Builder().forBrowser("chrome").build();
  await driver.get("https://example.com");

  const title = await driver.findElement(By.css("h1")).getText();
  console.log(title);

  await driver.quit();
})();

Pros: one language for front and back end, native async. Cons: in the browser-automation niche on Node, Selenium has stronger competitors — Puppeteer and Playwright — that many prefer in the JS ecosystem.

C# (.NET)

Popular in enterprise environments on the Microsoft stack.

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

var driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://example.com");

var title = driver.FindElement(By.TagName("h1")).Text;
Console.WriteLine(title);

driver.Quit();

Pros: static typing, good performance, excellent integration with the .NET ecosystem. Cons: fewer scraping-oriented examples and third-party anti-detection libraries than Python.

Ruby

require "selenium-webdriver"

driver = Selenium::WebDriver.for :chrome
driver.get "https://example.com"

title = driver.find_element(tag_name: "h1").text
puts title

driver.quit

Pros: concise, expressive syntax. Cons: a relatively small community for scraping tasks, fewer ready-made solutions.

PHP, Go, Rust, and others

PHP has php-webdriver/php-webdriver; Go and Rust have community wrappers. They work, but: they're community-maintained, lag the official bindings on new features, and are less well documented specifically for scraping. If your project's language is one of these, it often makes more sense to consider an alternative (for example, a headless browser via CDP, or an HTTP scraper) rather than dragging in Selenium.

Pros of Selenium

• Works with dynamic sites. Executes JavaScript and sees the final DOM — something HTTP scrapers fundamentally cannot get without browser emulation.
• Mimics a real user. Clicks, text input, scrolling, hovering, tab switching, cookie handling — all just like a live person.
• Cross-language. The same approach in Python, Java, JS, C#, Ruby — you can stay in your project's stack.
• Cross-browser. Chrome, Firefox, Edge, Safari through a single API.
• Maturity and documentation. The project is over 15 years old, with an enormous community; an answer to almost any question is already on Stack Overflow.
• Full control of the browser. Screenshots, executing arbitrary JavaScript, access to the Chrome DevTools Protocol, network interception.

Cons of Selenium

• Slow and heavy. Launching a full browser per session eats a lot of CPU and RAM. At scale that's tens or hundreds of times more expensive in resources than an HTTP request.
• Scales poorly "as is." To scrape thousands of pages in parallel you need Selenium Grid, a pool of containers, or the cloud — a whole separate infrastructure.
• Brittle. The scraper is tied to page structure and load timing. Change the markup or shift the delays, and you need explicit waits (WebDriverWait), or everything breaks.
• Harder to operate. Browser and driver versions must match (Selenium Manager eases some of this pain, but not all), and running a browser in Docker takes careful setup.
• And above all — easy to detect. The next section is devoted to this.

Why Selenium is easy to detect

This is arguably the single most important practical problem. Modern anti-bot systems (Cloudflare, DataDome, Imperva/Incapsula, PerimeterX, and others) can recognize an automated browser by a host of signals. And out of the box, Selenium leaves dozens of such traces.

The main tells that give Selenium away

1. The navigator.webdriver flag. The best-known marker. When launched through WebDriver, the browser sets navigator.webdriver = true. For a normal user this property is false or undefined. A single line of JavaScript on the site's side is enough to flag the visit as a bot.

2. The $cdc_ and $wdc_ variables. ChromeDriver injects service variables into the document (for example, ones starting with $cdc_). A detector simply looks for their presence on the document/window object — and finds them.

3. Traces of the Chrome DevTools Protocol (CDP). Selenium talks to Chrome over CDP. Enabling the Runtime domain (Runtime.Enable) is detected separately — so-called "second-generation" detection, which most simple anti-detection libraries do not close.

4. Inconsistencies in navigator. If you spoof the User-Agent to an iPhone but navigator.platform stays Win32, the core count (hardwareConcurrency) is atypical for the claimed device, and the plugin/language lists look "sterile" — the anti-fraud engine catches the logical mismatch instantly.

5. Signatures of the WebDriver commands themselves. Calls to execute_script/execute_async_script leave recognizable traces in the JavaScript call stack, which can also fingerprint automation.

6. Behavioral and network signals. Perfectly even speed, no mouse movement, zero history and cookies, plus a TLS fingerprint (JA3/JA4) and header order that differ from a real browser. This is the "third generation" of detection.

The "generations" of detection — so you know what you're up against

It helps to keep this gradation in mind:

• Generation 1 — API-level checks. navigator.webdriver, $cdc_ variables, and the like. Closed by anti-detection libraries.
• Generation 2 — protocol-level (CDP) detection. For example, watching for Runtime.Enable. Most popular patches no longer cure this.
• Generation 3 — TLS fingerprinting, behavioral analysis, per-client ML models. Fundamentally can't be closed with client-side patches — here you need proxies, rotation, behavior emulation, and quality fingerprints.

Tools to "hide" Selenium

You can't remove detection entirely, but you can lower the odds of a block. The most common choices:

• undetected-chromedriver (Python) — patches the driver binary (removing the cdc_ string), hides navigator.webdriver, and tunes the options. Long considered the "gold standard," but it mostly closes first-generation problems.
• selenium-stealth (Python) — a set of JS patches: it spoofs navigator.webdriver, plugins, languages, the WebGL vendor, and so on. Less actively maintained than UC.
• SeleniumBase UC Mode — adds a disconnect/reconnect trick: on sensitive actions it temporarily detaches ChromeDriver from the browser, bypassing part of the CDP detection.
• nodriver — the successor to undetected-chromedriver from the same author. It works directly over CDP, without chromedriver and the WebDriver layer, which eliminates a range of markers. This is no longer really "Selenium" but a separate approach.
• Manual tricks — the --disable-blink-features=AutomationControlled flag, disabling enable-automation, spoofing navigator.webdriver via Page.addScriptToEvaluateOnNewDocument, and aligning the User-Agent with the browser's real navigator properties.

You can check how well your browser is "hidden" on public test benches like bot.sannysoft.com or fingerprinting services.

An important caveat

None of these libraries are a silver bullet. Anti-bot companies study the open-source bypasses: exactly which patches undetected-chromedriver applies is visible right in its repo. What defeats a defense today may stop working after the next update — with no warning. So for serious scraping, teams almost always pair Selenium with quality rotating proxies, sensible delays, warmed-up profiles with cookies, and human-behavior emulation. And even then, against third-generation defenses, bare Selenium is often not enough — which is where a managed CAPTCHA-solving and proxy pipeline earns its place.

When to reach for Selenium, and when not to

Selenium is justified when:

• the site is built entirely on JavaScript and you can't get the content without a browser;
• you need to emulate a complex user scenario (login, multi-step forms, infinite scroll);
• volumes are moderate, not millions of pages an hour;
• you already have a Java/C#/Python stack and want to stay in it.

Better to look elsewhere when:

• the data is available over plain HTTP/an API — then requests/Scrapy/httpx will be many times faster and cheaper;
• you need maximum scale and speed;
• the goal is to bypass a serious anti-bot defense: here Playwright or Puppeteer (more modern, with better browser control and convenient stealth plugins) often win, and sometimes nodriver or a specialized API does.

Bottom line

Selenium is a powerful, versatile tool: it works with dynamic sites, supports many languages and browsers, and has a huge community and excellent documentation. That makes it a solid choice for scraping complex JavaScript pages and for scenarios that mimic a real user.

But it comes with two serious costs: resource intensity (and thus speed and scaling problems) and easy detectability. Out of the box, Selenium leaves dozens of automation traces, and while you can partially mask them, that's often not enough against modern anti-bot defenses.

The practical takeaway is simple: if you can get the data with an HTTP request, use an HTTP scraper. If a browser is unavoidable, Selenium fits well — but budget for proxies, rotation, human-like behavior, and anti-detection, and for the most protected targets, honestly compare it against Playwright, Puppeteer, and newer solutions.

If you'd rather skip the anti-bot arms race entirely, scraping.pro delivers the finished dataset as a done-for-you scraping service — you get the structured data, we handle the browsers, proxies, and detection.