Selenium Web Scraping in Java: Six Practical WebDriver Tips

A scraper that works on your laptop and a scraper that survives ten thousand pages overnight are not the same program. The parsing logic is rarely what separates them; the habits around it are — how you confirm you grabbed the right node, how you wait, how you fail, and whether you close the browser. Six of those habits are below.

First, the question the rest of this assumes you have answered: do you need a browser at all? Open DevTools, reload, and watch the Network tab. If the data arrives as JSON from an XHR endpoint, skip Selenium — an HTTP client and Jackson will be an order of magnitude faster and will not want 300 MB of RAM per worker. Selenium earns its cost when the markup only exists after JavaScript has run, when you need a real session and cookie jar, or when the page resists simpler clients.

Everything here is checked against Selenium 4.46.0 (released 11 July 2026), the current stable release; the Maven coordinate is org.seleniumhq.selenium:selenium-java. One thing you no longer manage is driver binaries. Since 4.6, Selenium Manager ships inside the bindings and fetches a matching driver when it cannot find one — no PATH juggling, no reason to add WebDriverManager. It is still labelled beta and acts only as a fallback, so supplying your own driver keeps it out of the way.

1. Outline the node you are about to read

When you are reverse-engineering someone else's DOM, the question you ask forty times a day is whether the selector really matched the thing you think it matched. Drawing a ring around the element before you read it answers that in one glance. WebDriver exposes the page's JavaScript engine through JavascriptExecutor:

public void highlight(WebElement element) {
    ((JavascriptExecutor) driver).executeScript(
        "const s = arguments[0].style;"
      + "s.setProperty('outline', '3px solid #e11d48', 'important');"
      + "s.setProperty('outline-offset', '2px', 'important');",
        element);
}

Two choices there are deliberate. outline rather than border, because a border occupies space in the box model: add two pixels around a cell in a flex row and everything after it shifts, possibly moving the element you were about to click. Outlines paint outside the box and reflow nothing. And writing onto element.style rather than setAttribute("style", …), because the attribute form replaces whatever inline styles the page already had — on a site that positions elements inline, you break the layout you are trying to read.

If you have the widely-copied version of this snippet lying around, check its CSS: it usually sets color: yellow next to a yellow border, making the element's text invisible against white.

Guard it behind a debug flag. Under --headless=new there is nothing to look at anyway; the headless equivalent is a screenshot at the moment of confusion, via ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE).

2. Check that text is present, and know the three ways the check lies

Selenium RC had an isTextPresent helper. WebDriver dropped it, and the substitute is two lines, because getPageSource() hands you the whole serialized document:

public boolean isTextPresent(String text) {
    return driver.getPageSource().contains(text);
}

For scraping, this earns its keep as a page classifier: did I land on results, an empty state, a rate-limit notice, or a challenge page? But three things will make it quietly return the wrong answer.

HTML escaping. You are searching markup, not text. contains("Tom & Jerry") fails on a page that renders Tom & Jerry from Tom & Jerry. Same for quotes, for   where you typed a space, and for anything a CMS entity-encoded. When a presence check mysteriously never matches, this is usually why.

Source is not the rendered page. The javadoc is blunt: "If the page has been modified after loading (for example, by Javascript) there is no guarantee that the returned text is that of the modified page." Most current drivers do serialize the live DOM, but the contract does not promise it.

Presence is not visibility. A hit can come from a <script> payload, a data- attribute, a display: none template, or a preloaded JSON blob that never reaches the screen.

For anything beyond a smoke check, scope the assertion to an element and let a wait handle the timing:

new WebDriverWait(driver, Duration.ofSeconds(10))
        .until(ExpectedConditions.textToBePresentInElementLocated(
                By.cssSelector("[data-testid='results']"), "results found"));

And when you want a node's text rather than a yes or no, be deliberate about which text. getText() returns rendered text — whitespace collapsed, hidden descendants skipped — while getDomProperty("textContent") gives you the whole subtree including nodes CSS has hidden. Scrapers usually want the second and reach for the first out of habit.

3. Never hard-code a sleep, and do not set the timeout in minutes either

The antipattern:

Thread.sleep(180_000);                       // three minutes, every single time
driver.findElement(By.id("username"));

Thread.sleep burns the whole interval even when the node appeared in 300 ms. One wasted second per page across a ten-thousand-page crawl is close to three hours. Set the value too low instead and you get intermittent failures on exactly the pages you most wanted data from.

An explicit wait polls and returns the moment its condition holds. In Selenium 4 the timeout is a java.time.Duration:

import java.time.Duration;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

WebElement field = new WebDriverWait(driver, Duration.ofSeconds(10))
        .until(ExpectedConditions.presenceOfElementLocated(By.id("username")));

Note the ceiling: ten seconds, not three minutes. That number matters more in scraping than in testing. A generous timeout is a cheap safety net across two hundred tests; on a crawl it is a way to spend fifty hours discovering that a selector broke. Pick roughly triple the page's normal render time and let it fail loudly.

Reach for FluentWait when you want the polling interval or the ignore list:

Wait<WebDriver> wait = new FluentWait<>(driver)
        .withTimeout(Duration.ofSeconds(10))
        .pollingEvery(Duration.ofMillis(250))
        .ignoring(StaleElementReferenceException.class);

Ignoring StaleElementReferenceException is the entry that pays for itself. It is the signature scraping failure: you locate a row, a framework re-renders the list underneath you, and your reference points at a node no longer attached to the document. The durable pattern is to re-find inside the wait rather than hold references across a re-render:

String price = new FluentWait<>(driver)
        .withTimeout(Duration.ofSeconds(10))
        .ignoring(StaleElementReferenceException.class)
        .until(d -> d.findElement(By.cssSelector(".price")).getText());

One rule from the official Waiting Strategies page deserves quoting, because breaking it produces timing bugs nobody can reproduce: "Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times." Their example — a 10-second implicit wait alongside a 15-second explicit one timing out at 20 — is arithmetic you do not want to debug. Choose explicit and leave the implicit timeout at its default of zero.

4. Fall back to a JavaScript click, knowing what it skips

Sometimes element.click() does nothing, throws, or never returns, usually because the element is overlapped, mid-animation, or being detached and re-attached by a framework. Dispatching the click through the DOM works around it:

public void jsClick(WebElement element) {
    ((JavascriptExecutor) driver).executeScript("arguments[0].click();", element);
}

In scraping you hit this most on Load more buttons, cookie walls, and pagination parked under a sticky header. But a JavaScript click is not a cheaper native click; it is a different operation. It fires a click event on the node and skips everything WebDriver does first: scrolling the element into the viewport, honouring pointer-events, and refusing loudly when the element genuinely cannot be reached.

In testing, the argument against it is that it conceals real bugs. In scraping the damage is more direct. Plenty of Load more implementations fetch the next batch only once the element has entered the viewport, because they are wired to an IntersectionObserver or a scroll handler. Click it from JavaScript while it sits four thousand pixels down the page and the handler runs against state that was never set up: no request goes out, no rows arrive, and you conclude the site has twenty items when it has two thousand.

So scroll, then try the native click, then fall back:

new Actions(driver).scrollToElement(loadMore).perform();
try {
    new WebDriverWait(driver, Duration.ofSeconds(5))
            .until(ExpectedConditions.elementToBeClickable(loadMore))
            .click();
} catch (ElementClickInterceptedException | TimeoutException e) {
    jsClick(loadMore);                       // last resort
}

Log the fallback. If it starts firing on every page, something structural changed and you want to know before the data does.

5. Silence the download prompt, after checking you still have one

This is where older advice has gone stale, so reproduce the problem before configuring around it. Firefox used to open a What should Firefox do with this file? dialog, and because WebDriver cannot drive native OS chrome, that dialog stopped a scraper dead. Since Firefox 98 it does not appear: Mozilla's documentation states you are "not prompted to choose a helper application or save to disk before downloading a file, unless you have changed your setting."

If you do still meet a prompt — a profile with the old behaviour switched back on, a locked-down corporate build, or a content type Firefox would rather open itself — this is the current set, configured on FirefoxOptions:

import java.nio.file.Files;
import java.nio.file.Path;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;

Path downloads = Files.createTempDirectory("scrape-downloads");

FirefoxOptions options = new FirefoxOptions();
options.addPreference("browser.download.folderList", 2);            // 2 = use the dir below
options.addPreference("browser.download.dir", downloads.toString());
options.addPreference("browser.download.useDownloadDir", true);
options.addPreference("browser.download.always_ask_before_handling_new_types", false);
options.addPreference("browser.helperApps.neverAsk.saveToDisk",
        "application/zip,application/octet-stream,text/csv");
options.addPreference("pdfjs.disabled", true);                      // else PDFs open in the viewer

WebDriver driver = new FirefoxDriver(options);

Three things the copy-pasted recipes get wrong.

browser.download.always_ask_before_handling_new_types is the preference that governs the dialog now, and it already defaults to false. The one you will see everywhere, browser.download.manager.showWhenStarting, targeted the standalone Download Manager window that Firefox retired years ago; setting it today does nothing.

pdfjs.disabled matters more than people expect, and it is the line most recipes omit. Firefox's built-in viewer opens PDFs rather than saving them, and neverAsk.saveToDisk does not override that, so if PDFs are the goal no amount of MIME-type fiddling substitutes for disabling the viewer. A wider family of formats behaves the same way, governed by browser.download.viewableInternally.enabledTypes; when some other type keeps opening in a tab instead of landing on disk, setting that preference to an empty string is the escape hatch (Mozilla bug 1670651 has the history).

neverAsk.saveToDisk matches the Content-Type the server sends, not the file extension. A CSV served as text/plain needs text/plain in the list, so read the response header rather than guess.

A fresh temporary directory per run beats a hard-coded D:\downloads: it makes which file did this page produce? answerable, and it survives the move to Linux CI. You can set the same preferences on a FirefoxProfile and attach it with options.setProfile(profile) — but pick one mechanism, since splitting preferences across both reliably produces settings that appear not to apply.

None of this helps on Grid, where the file lands on the node rather than your machine. Selenium's answer there is the se:downloadsEnabled capability plus HasDownloads: getDownloadedFiles() lists what the browser saved, downloadFile(name, target) pulls one back to the client, deleteDownloadableFiles() cleans up between sessions.

6. Override the user agent, and be clear about what it hides

Sometimes the mobile layout is the one carrying the data you want, and most servers branch on the User-Agent header, so the preference is enough to get it:

FirefoxOptions options = new FirefoxOptions();
options.addPreference("general.useragent.override", MOBILE_UA);

WebDriver driver = new FirefoxDriver(options);

MOBILE_UA should come from a real, current browser, not from an article. Every user-agent string printed in a blog post is a fossil the day after publication, and a stale one is itself a signal — a request claiming Firefox 13 in 2026 stands out far more than one claiming nothing unusual. Read navigator.userAgent in the console of a device you actually have and keep the value in configuration, not in code. MDN documents the header's structure under User-Agent.

Now the limits, because this tip is the one most often oversold. The override changes a header. It does not change the viewport, touch-event support, device-pixel ratio, or what matchMedia reports, so a site branching on any of those still serves you the desktop tree. On Chromium it is worse: client hints (Sec-CH-UA and friends) derive from the real build, so a bare UA swap leaves two contradictory accounts of what browser you are.

And navigator.webdriver stays true. The W3C WebDriver specification requires the browser to expose that flag for any automated session, and no preference turns it off. If the site is looking for automation rather than for a phone, this tip does nothing for you.

The ground has also moved underneath the alternative: the CDP route to the same effect (Network.setUserAgentOverride) is gone for Firefox. Selenium deprecated Firefox CDP in 4.27 and 4.28 and removed it in 4.29, following Firefox's decision to stop enabling CDP by default from Firefox 129. WebDriver BiDi is the replacement.

Beyond the six: what actually breaks at scale

Quit the driver. None of the snippets above are complete without this, and omitting it is the most common way a scraper takes down its own host. Every new FirefoxDriver(…) starts a browser process; quit() ends both it and its driver:

WebDriver driver = new FirefoxDriver(options);
try {
    // ... scrape
} finally {
    driver.quit();
}

close() closes one window; quit() ends the session. Miss it on an exception path and you leak a browser per failure — at a few hundred megabytes each, a long crawl exhausts the machine well before it exhausts the URL list.

Stop rendering what you never read. Three settings, in rough order of payoff: run headless (--headless=new on Chrome, -headless on Firefox); set PageLoadStrategy.EAGER so get() returns at DOMContentLoaded instead of waiting on every tracker and web font; and block the requests you will not use.

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
options.setPageLoadStrategy(PageLoadStrategy.EAGER);

EAGER is the one to try first. On advertising-heavy pages the tail of the load event routinely lasts longer than everything you came for.

For blocking, WebDriver BiDi now offers a browser-neutral way in — a real improvement over per-browser preferences:

import org.openqa.selenium.bidi.module.Network;
import org.openqa.selenium.bidi.network.AddInterceptParameters;
import org.openqa.selenium.bidi.network.InterceptPhase;
import org.openqa.selenium.bidi.network.RequestData;

options.enableBiDi();                         // or setCapability("webSocketUrl", true)
WebDriver driver = new FirefoxDriver(options);

try (Network network = new Network(driver)) {
    network.addIntercept(new AddInterceptParameters(InterceptPhase.BEFORE_REQUEST_SENT));
    network.onBeforeRequestSent(event -> {
        RequestData request = event.getRequest();
        if (request.getUrl().matches(".*\\.(png|jpe?g|gif|webp|svg|woff2?)(\\?.*)?$")) {
            network.failRequest(request.getRequestId());
        }
    });

    driver.get("https://example.com/catalogue");
    // scrape here — the intercept lives as long as this block
}

Two caveats before you build on it. As of 4.46 every BiDi class in the Java bindings is marked beta, and that release moved connection handling onto RemoteWebDriver while deprecating the old accessors — expect the surface to keep shifting. And BiDi does not hand you response bodies: ResponseData exposes status, headers and MIME type, but its getContent() returns an Optional<Long>, a size rather than a payload. If you hoped to skim JSON off the wire and skip the DOM, you still have to request the URL yourself.

Be a guest. Read /robots.txt, cap concurrency per host, put a real delay between requests, and cache so a re-run does not re-fetch. A browser-based scraper is expensive for the site as well as for you, since every page you open pulls the full asset set. Check the terms you are operating under, and where a documented API exists, prefer it: faster, stabler, and immune to someone renaming a CSS class.

Where each technique fits

Further reading

• Selenium project and downloads: https://www.selenium.dev/downloads/
• WebDriver documentation: https://www.selenium.dev/documentation/webdriver/
• Waiting strategies (implicit, explicit, fluent): https://www.selenium.dev/documentation/webdriver/waits/
• Browser options and page load strategies: https://www.selenium.dev/documentation/webdriver/drivers/options/
• WebDriver BiDi network module: https://www.selenium.dev/documentation/webdriver/bidi/w3c/network/
• Selenium Manager: https://www.selenium.dev/documentation/selenium_manager/
• Full Java API reference: https://www.selenium.dev/selenium/docs/api/java/
• Maven artifact: https://central.sonatype.com/artifact/org.seleniumhq.selenium/selenium-java

None of this is exotic, and none of it replaces well-chosen locators. It is the difference between a scraper you babysit and one you schedule. If you change one thing today, change the timeout from tip 3: on a ten-thousand-page crawl, a three-minute ceiling turns a broken selector into a fifty-hour mystery, while ten seconds turns it into a failure on page one.