Selenium Web Scraping with Java: A Full Tutorial

The previous version of this tutorial finished with a complete, runnable program. It no longer runs. The host it pointed at, testing-ground.scraping.pro, has stopped resolving: a query to Google's public resolver on 13 August 2026 comes back with status 3, no answer section, only the SOA record for scraping.pro. Not a line of the Java was wrong. The address simply went away.

That is how Selenium tutorials die, and the target is only the first of three places where they rot. The second is the version number pinned in pom.xml, which was 4.27.0 here and is twenty releases behind. The third is the browser: the headless flag every guide prints has quietly stopped being the spelling Chrome documents. Everything below is rebuilt against Selenium 4.47.0, released 10 August 2026, and Chrome stable 151.0.7922.138, and every target in it answered when we checked it on 13 August 2026.

This is the build-and-run guide: an empty project to a paginating scraper that writes a CSV. The habits that keep such a scraper alive over months — highlighting nodes while you debug, fluent waits, JavaScript click fallbacks, download preferences, user-agent overrides — are the subject of the companion piece, six practical WebDriver tips, and are not repeated here.

What you are actually talking to

driver.findElement(By.id("usr")) looks like a method call into a browser. It is an HTTP request. Your Java process serializes the locator to JSON, sends POST /session/{session id}/element to a small server that the driver binary started on localhost, and the driver translates that into whatever private protocol its browser speaks. The endpoints, the payloads and the error codes are a W3C document, currently a WebDriver Level 2 Working Draft dated 2 July 2026. When nothing matches, the driver answers HTTP 404 with the error code no such element, and the Java bindings turn that into NoSuchElementException.

Four consequences follow, and they explain most of the rest of this article.

• Every locator call costs a round trip. Scraping a hundred quotes with three findElement calls apiece is three hundred request/response pairs against the driver, plus one navigation per page. Locally that is cheap and invisible. Through a Grid node in another data centre it is the dominant cost of the run, which is why you fetch a list once with findElements and drill into each result, rather than re-querying the document per field.
• A WebElement is a handle, not a node. It refers to an entry in the driver's element cache for that session. Re-render the list underneath it and the handle points at nothing.
• quit() ends a session, not a variable. The browser is a separate operating-system process that outlives your JVM if you never send the command.
• A session is single-threaded. Two threads sharing one driver interleave commands on one browser and produce nonsense. One driver per thread.

Setup: two coordinates and a JDK floor

Modern Java scraping starts with a build file, not a folder of downloaded JARs. Maven:

<dependencies>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>4.47.0</version>
    </dependency>
</dependencies>

Gradle:

implementation 'org.seleniumhq.selenium:selenium-java:4.47.0'

That version is not a guess. Maven Central's metadata for the artifact lists <release>4.47.0</release> with lastUpdated of 20260810, and the project's own downloads page prints "Stable: 4.47.0 (August 10, 2026)". By the time you read this the number will have moved. The API used below has been stable across 4.x.

Which JDK. The documentation is coy about it: the install page says only "View the minimum supported Java version here" and links a line in the build config. That line, in .bazelrc, reads build --javacopt="--release 11" under the comment "We target java 11 by default", while the build toolchain itself runs --java_language_version=25. So Java 11 is the floor for the published bytecode and anything newer runs it. The code here is written for JDK 21 and leans on Path.of and the stream API, which want 11 or later in any case.

The driver binary is not your problem any more. Since 4.6, Selenium Manager ships inside the bindings, works out which browser you have, and fetches a matching driver. Three details that the one-line version of this advice leaves out and that will cost you an afternoon on a build server:

It writes to ~/.cache/selenium and reads se-config.toml from the same directory, with SE_-prefixed environment variables as a third configuration layer. A container that starts with an empty home directory downloads a driver on every run.

Since 4.11 it manages browsers as well as drivers, and accepts channel labels such as stable, beta, dev, canary and esr. On a machine with no browser at all it will go and get one.

It has an --offline flag that "disables network requests and downloads". In an air-gapped environment that flag plus a pre-warmed cache is the difference between a working image and a build that hangs on a download. The documentation still carries the word Beta in its title, and Selenium Manager acts as a fallback: supply your own driver on PATH and it stays out of the way.

Launching Chrome, and the options object that goes nowhere

Here is the minimum to open a page. Read the third line carefully.

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
options.addArguments("--window-size=1920,1080");

WebDriver driver = new ChromeDriver();          // <- options are silently ignored

That shape — build an options object, then construct the driver without it — appears in this article's own previous version and in a large fraction of the Selenium tutorials you will find by searching. It compiles. It runs. It opens a visible browser window at the default size and every argument you set is discarded, because new ChromeDriver() builds its own empty options object. The failure is invisible on a laptop, where a window opening is normal, and shows up on a headless server as a driver that cannot start a display. The fix is one argument:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

public class QuickStart {
    public static void main(String[] args) {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless", "--window-size=1366,900");

        WebDriver driver = new ChromeDriver(options);   // options actually applied
        try {
            driver.get("http://quotes.toscrape.com/js/");
            System.out.println(driver.getTitle());
        } finally {
            driver.quit();
        }
    }
}

On the headless flag. Google's headless documentation no longer mentions the =new suffix anywhere. It documents plain --headless, and records that "Since Chrome 132.0.6793.0 the old Headless mode is only available as a standalone binary named chrome-headless-shell". The old implementation was not deprecated so much as moved out of the browser. The suffix is a fossil of the window in which both implementations lived in one binary and you had to say which you meant. Selenium's own Chrome page still prints --headless=new and lists it among "commonly used args"; both pages were live on 13 August 2026. For new code against a current Chrome, write --headless. Whether --headless=new still parses on Chrome 151 is not stated either way in Google's current documentation, and we did not launch a browser to find out. If you are pinning Chrome for Testing builds, last-known-good-versions.json is the file to read; on 11 August 2026 it put stable at 151.0.7922.138.

Window size is not cosmetic. Lazy-loading grids, responsive layouts and anything driven by IntersectionObserver behave differently at 800×600, which is not the size you want to discover after a night of scraping.

Locating elements

Everything else is finding nodes and reading them.

By.id("usr");                              // <input id="usr">
By.className("author");                    // class="author"
By.cssSelector("div.quote span.text");     // CSS - the default choice
By.xpath("//div[@class='quote'][contains(., 'Einstein')]");   // XPath - text matching
By.tagName("a");
By.linkText("Next");

CSS covers most work; drop to XPath when you need to match on text content or walk up the tree, which CSS cannot do. Our guide to CSS selectors for scraping has the syntax.

Two locator bugs were fixed in the release this article is pinned to, and both are the kind that make you doubt your selector rather than your library. The Java changelog for 4.47.0 lists "Fix By.className()/By.id() misescaping non-ASCII leading digits" and "Fix By.name() double String.format on names containing '%'". If your target has class names starting with a digit, non-ASCII identifiers, or form fields with a percent sign in the name, upgrading is the fix.

findElement returns one element and throws when nothing matches. findElements returns a possibly empty list and never throws for absence. That difference is a control-flow tool: testing findElements(...).isEmpty() is how you check for an optional element without wrapping the call in try/catch.

Attribute is not property. This one silently corrupts crawl output. The WebElement javadoc describes getAttribute as returning "the value of the property with the given name, if it exists" and falling back to the attribute, while getDomAttribute "returns the value of the attribute with the given name but not the property with the same name". For href those are different strings. MDN is explicit that the href property "returns the absolute URL corresponding to the element's href attribute", resolved against the document's base URL, while the raw attribute is whatever the markup contains.

// where the markup reads <a href="/js/page/2/">Next</a>
WebElement link = driver.findElement(By.cssSelector("li.next a"));

link.getAttribute("href");      // http://quotes.toscrape.com/js/page/2/   resolved
link.getDomAttribute("href");   // /js/page/2/                             as written
link.getDomProperty("href");    // http://quotes.toscrape.com/js/page/2/   resolved, on purpose

Reach for getAttribute when you want a URL you can request, and getDomAttribute when you are reproducing the page's markup. Picking the wrong one produces a column of relative paths that look fine in a spreadsheet and 404 in the next stage of your pipeline.

Waits, in one screen

On a JavaScript page the node you want does not exist when load fires. It appears when a script finishes. driver.get() returns at the load event and promises nothing about the DOM after it, so a findElement on the next line is a race you will lose intermittently. Thread.sleep is the wrong answer; an explicit wait polls and returns the instant its condition holds.

import java.time.Duration;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

wait.until(ExpectedConditions.presenceOfElementLocated(
        By.cssSelector("div.quote")));

The four conditions that cover most scraping: presenceOfElementLocated for "it is in the DOM", visibilityOfElementLocated for "and it is visible", elementToBeClickable before a click, textToBePresentInElementLocated when the element exists from the start and only its content changes. Pick a timeout around triple the page's normal render time, not three minutes, and never mix implicit and explicit waits. The reasoning behind those two rules, plus FluentWait, polling intervals and ignore lists, is in the companion article; this page uses the plain version and moves on.

Frames and shadow roots: where findElement legitimately sees nothing

Twice a year someone spends an hour on a selector that is provably correct in DevTools and provably absent from driver.findElements. Both times the element is behind a boundary that WebDriver does not cross implicitly.

Iframes. A document inside an <iframe> is a separate browsing context. Selenium's frames documentation puts it plainly: "To interact with the button, we will need to first switch to the frame." You can switch by WebElement, by name or id, or by index, and driver.switchTo().defaultContent() gets you back to the top document. Forget the switch back and every later locator runs against the frame instead of the page. Payment forms, cookie consent dialogs and embedded maps are the usual suspects.

Shadow DOM. Web components hide their internals behind a shadow root, and CSS selectors do not pierce it. WebElement.getShadowRoot() returns "a representation of an element's shadow root for accessing the shadow DOM of a web component" and throws NoSuchShadowRootException when there is none; from that object you search again for what is inside. A root attached in closed mode is not reachable at all, from Selenium or from your own JavaScript, and the only way through is whatever data the component was fed in the first place.

A complete scraper

The target is quotes.toscrape.com/js/, the JavaScript-rendered variant of the sandbox that Scrapy's own tutorial documents. It earns its place here by proving the point: fetch that URL with an HTTP client and you get the navigation, the tag sidebar and the pager, and not one quote. The quote text is written into the DOM by a script. That is the whole argument for a browser, in one page.

Two things to know before you paste. The site does not serve HTTPS: a request to https://quotes.toscrape.com/js/ answers 302 to the http:// URL, checked twice on 13 August 2026. And the markup is documented rather than reverse-engineered: Scrapy's tutorial prints div.quote wrapping span.text, small.author and a.tag, with li.next a for the pager, and the /js/ variant builds that same structure from a script instead of shipping it in the response.

import org.openqa.selenium.By;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class QuoteScraper {

    private static final String START = "http://quotes.toscrape.com/js/";
    private static final int MAX_PAGES = 50;          // a crawl needs a ceiling

    public static void main(String[] args) throws IOException {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless", "--window-size=1366,900");

        WebDriver driver = new ChromeDriver(options);
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
        List<String[]> rows = new ArrayList<>();
        int page = 1;

        try {
            driver.get(START);

            while (true) {
                wait.until(ExpectedConditions.presenceOfElementLocated(
                        By.cssSelector("div.quote")));

                for (WebElement quote : driver.findElements(By.cssSelector("div.quote"))) {
                    String text   = quote.findElement(By.cssSelector("span.text")).getText();
                    String author = quote.findElement(By.cssSelector("small.author")).getText();
                    String tags   = quote.findElements(By.cssSelector("a.tag")).stream()
                            .map(WebElement::getText)
                            .collect(Collectors.joining("|"));
                    rows.add(new String[]{String.valueOf(page), author, text, tags});
                }

                List<WebElement> next = driver.findElements(By.cssSelector("li.next a"));
                if (next.isEmpty() || page >= MAX_PAGES) break;

                WebElement anchor = driver.findElement(By.cssSelector("div.quote"));
                next.get(0).click();
                wait.until(ExpectedConditions.stalenessOf(anchor));   // old DOM is gone
                page++;
            }

            Files.write(Path.of("quotes.csv"), toCsv(rows));
            System.out.println("collected " + rows.size() + " quotes across " + page + " pages");

        } catch (RuntimeException e) {
            File shot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
            Files.copy(shot.toPath(), Path.of("failure.png"),
                       StandardCopyOption.REPLACE_EXISTING);
            throw e;
        } finally {
            driver.quit();
        }
    }

    private static List<String> toCsv(List<String[]> rows) {
        List<String> lines = new ArrayList<>();
        lines.add("page,author,text,tags");
        for (String[] r : rows) {
            lines.add(Arrays.stream(r)
                    .map(v -> '"' + v.replace("\"", "\"\"") + '"')
                    .collect(Collectors.joining(",")));
        }
        return lines;
    }
}

Run it with mvn compile exec:java and you get quotes.csv in the project directory, plus failure.png if something threw. Four choices in there are worth naming, and three of them exist because the older version of this tutorial got them wrong.

The screenshot lands in a catch, not on the happy path. A screenshot of a successful run is a file nobody opens. A screenshot of the DOM at the moment of failure tells you whether you hit a challenge page, an empty state or a layout change, and it is the most useful thing an unattended crawl can leave behind for the morning.

StandardCopyOption.REPLACE_EXISTING is not decoration. Files.copy(source, target) without it throws FileAlreadyExistsException, so the earlier version of this code worked exactly once per directory and then failed on its own leftovers. getScreenshotAs(OutputType.FILE) hands you a temporary file that the driver may clean up, so copy it somewhere you own before doing anything else.

Quoting every CSV field. The quote texts on that page contain commas and typographic quotation marks. A CSV writer that only quotes fields it thinks need it, or that forgets to double embedded quotes, produces a file that opens fine and has one row shifted by a column halfway down. Two lines of escaping now, or an afternoon with a diff later.

throws IOException on main and a finally that always quits. Between them, a failed write is loud and the browser process dies with the program. This is the one the previous version already had right, and it is the line people delete when they refactor.

The pagination race that silently duplicates your data

The loop above has one line that most published versions omit, and its absence is the most expensive bug in this article.

// The version that looks right and is not
while (true) {
    wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("div.quote")));
    scrape(driver.findElements(By.cssSelector("div.quote")));

    List<WebElement> next = driver.findElements(By.cssSelector("li.next a"));
    if (next.isEmpty()) break;
    next.get(0).click();
}

Click a link and navigation starts, but it does not finish before the next Java statement runs. For a moment the browser is still showing the old document. Your wait asks whether a div.quote is present, the old page's quotes are still present, and the condition is satisfied instantly against the page you have already scraped. Depending on timing you re-read page one, or read a half-replaced document, or get a StaleElementReferenceException from a handle that expired mid-loop.

The first two outcomes are the dangerous ones, because nothing fails. The job exits zero and the CSV has a plausible number of rows, some of them read twice and some pages never read at all. In the worst case ten pages of quotes arrive as page one, ten times over.

The fix is to wait for the old document to go away rather than for the new one to look plausible. ExpectedConditions.stalenessOf is documented as "Wait until an element is no longer attached to the DOM", which is precisely the event you care about. Grab a handle on the current page before the click, then block until it dies:

WebElement anchor = driver.findElement(By.cssSelector("div.quote"));
next.get(0).click();
wait.until(ExpectedConditions.stalenessOf(anchor));

Where pagination is done in JavaScript without a navigation — infinite scroll, a Load more button, a client-side router — nothing goes stale, so anchor on a count instead: numberOfElementsToBeMoreThan for a growing list, or urlContains when a router at least updates the address bar. The pattern is the same. Wait for a change you can name, never for time to pass.

The MAX_PAGES ceiling deserves its own line. Some sites render a Next link on every page including the last, and a while (true) against one of those is a crawl that ends when your disk does.

What changes at ten thousand pages

A single browser is easy. The gap between one and a fleet is where scrapers actually fail.

Processes, not objects. Every new ChromeDriver(options) starts a browser and a driver process. Leak one per failure path and a long run exhausts the host before it exhausts the URL list. This is the reason the finally block above is not optional, and it is worth logging the count of live sessions if you pool them.

Containers need shared memory. Chrome uses /dev/shm, and Docker's default allocation for it is small enough that the browser crashes under load with errors that look like anything but the real cause. The docker-selenium README is direct about the workaround: use "the flag --shm-size=2g to use the host's shared memory", a value it calls arbitrary but known to work well. Twenty parallel containers at that ceiling is worth checking against what the host actually has, and --disable-dev-shm-usage as a browser argument is the alternative when you cannot change the run flags.

Write incrementally. The scraper above accumulates rows in memory and writes once at the end, which is fine for ten pages and wrong for ten thousand: one exception on page 9,700 and the run produces nothing. Append per page, key each row by something stable, and make the job resumable. A crawl you can restart in the middle is worth more than a crawl that is twice as fast.

Shard by URL, not by shared driver. One driver per thread, a queue of start URLs, and no shared mutable state between workers. Two threads on one session interleave HTTP commands on one browser and produce results that belong to neither.

Politeness is throughput. Delay between requests, cap concurrency per host, cache aggressively so a re-run does not re-fetch. Beyond that, rotating proxies spread requests across addresses, and sites that challenge you need a CAPTCHA solving step budgeted into the run. Both sit on top of the code above, and both are where the ongoing maintenance lives, because anti-bot defences change on their schedule and not yours. Targets that fight back hard enough are where a managed web scraping service starts to look cheaper than a browser fleet you keep alive yourself.

Sessions and logins. Driving a login form works, and it is also the slowest possible way to authenticate ten thousand requests. Log in once, read the cookies out of driver.manage().getCookies(), and re-inject them into fresh sessions with addCookie after navigating to the domain — cookies cannot be set for a domain the browser is not currently on. If you want a practice target for the form-filling half, the-internet was up on 13 August 2026 and prints the credentials it accepts in its own page body. Read the target's markup for the field selectors rather than copying anyone's ids, this article's included.

Where Selenium stops being the answer

Static HTML. If the data is in the response body, a browser is pure overhead. jsoup parses and queries it with CSS selectors in the same idiom you already used; the current release is 1.23.1, dated 30 July 2026 on the project's news page. One HTTP client plus jsoup handles a surprising share of the jobs people reach for Selenium to do.

A JSON endpoint behind the page. Open DevTools, reload, watch the network requests. If the content arrives as JSON from an XHR call, request that endpoint directly and skip rendering entirely — faster, stabler, and immune to someone renaming a CSS class. Our guide to API scraping covers finding and using those endpoints.

Reading responses off the wire. WebDriver gives you the DOM, not the network. Request interception and response metadata arrive through WebDriver BiDi, whose Java classes were all marked beta in 4.46 and which does not hand back response bodies; the six-tips article linked above has the working code and the caveats.

Hiding that you are automated. The W3C protocol requires the browser to expose navigator.webdriver as true for any automated session, and no option in Selenium turns that off. A user-agent override changes a header and nothing else. If the target is looking for automation rather than for a phone, Selenium is not the layer where you solve it.

A different tool. Playwright for Java drives Chromium, Firefox and WebKit through one API, with request interception and response bodies as first-class features; its Maven Central metadata puts the current release at 1.62.0, last updated 3 August 2026. It is the serious alternative for browser work in Java, and the choice between them is a real one rather than a formality. If you are still choosing a language for the job at all, our comparison of the best languages for scraping is the place to start.

What this was checked against

Every URL linked above was opened on 13 August 2026. The versions will drift within weeks, and the drift is the point: a Selenium tutorial is a snapshot of three moving things, and the one most likely to break first is not the library.

If you keep one habit from this page, keep the staleness check after a click. A broken selector announces itself with an exception on page one. A pagination race announces nothing, fills a file with correct-looking rows, and is discovered a month later by someone who notices that every author in the dataset is Albert Einstein.