Selenium Web Scraping with Java: A Full Tutorial

A complete, up-to-date tutorial on Selenium web scraping with Java: setting up WebDriver the modern way (no manual driver downloads), locating elements, using explicit waits so your scraper doesn't flake, and pulling it all together into a working scraper you can run today.

Selenium is the veteran of browser automation, and Java is the language it was born in. If you need to scrape a site that builds its content with JavaScript — a single-page app, an infinite-scroll feed, anything where the data isn't in the raw HTML — driving a real browser with Selenium is a reliable way to get it. This is a hands-on guide to web scraping using Selenium in Java, rebuilt for Selenium 4/current releases, where a lot of the old friction (downloading chromedriver.exe, matching versions, wiring up build paths by hand) is simply gone.

We'll go from an empty project to a scraper that logs into a page, waits for content, extracts text, and saves a screenshot.

When Selenium is the right tool (and when it isn't)

Selenium launches and controls a genuine browser, so it executes JavaScript exactly like a user's Chrome would. That's its superpower and its cost.

• Use Selenium when the data is rendered client-side, hidden behind logins, or requires clicks, typing, and scrolling.
• Skip Selenium for static pages — a plain HTTP client plus Jsoup is far faster and lighter. And always check DevTools first: if the page pulls its data from a JSON API, call that internal endpoint instead of rendering anything.

If you're weighing this against Python or Node tooling, see our overview of the best languages for scraping. For Java specifically, when you do need a browser, Selenium and Playwright for Java are the two main options; this guide covers Selenium.

1. Project setup with Maven

Modern Java scraping starts with a build tool, not a folder of downloaded JARs. With Maven, add the Selenium dependency to your pom.xml:

<dependencies>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>4.27.0</version>
    </dependency>
</dependencies>

(Use whatever the current 4.x release is; the API below is stable across recent versions.) Gradle users add the equivalent line:

implementation 'org.seleniumhq.selenium:selenium-java:4.27.0'

That's the entire setup. You no longer download the browser driver yourself. Since Selenium 4.6, a component called Selenium Manager detects your installed browser and fetches the matching driver automatically the first time you run. The old routine — grabbing selenium-server-standalone.jar, downloading a driver binary, and pointing a system property at it — is obsolete. As long as Chrome (or Edge/Firefox) is installed, you're ready.

2. Launching a browser (headless)

Here's the minimum to open a page. Running headless (no visible window) is the norm for scraping — it's faster and works on servers with no display:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

public class QuickStart {
    public static void main(String[] args) {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new");   // modern headless mode
        options.addArguments("--window-size=1920,1080");

        WebDriver driver = new ChromeDriver();     // driver auto-managed
        driver.get("https://quotes.toscrape.com/js/");
        System.out.println(driver.getTitle());
        driver.quit();                             // always quit to free the process
    }
}

A few essentials:

• driver.get(url) blocks until the page's load event fires — but that does not guarantee JavaScript-rendered content has appeared yet. That's what waits are for (section 4).
• Always call driver.quit() (ideally in a finally block). It closes every window and ends the driver process; forgetting it leaves zombie browsers eating memory.
• --headless=new is Chrome's current headless implementation and behaves much closer to a real browser than the legacy mode.

3. Locating elements

Everything in Selenium scraping comes down to finding elements with a By locator, then reading text or attributes off them. The main strategies:

import org.openqa.selenium.By;

By.id("usr");                        // <input id="usr">
By.className("author");              // class="author"
By.cssSelector("div.quote span.text");   // CSS — usually the best choice
By.xpath("//div[@id='case_login']/h3");  // XPath — for text/structure matches
By.tagName("a");
By.linkText("Next");

Prefer CSS selectors for most work — they're concise and fast — and drop to XPath when you need to match on text content or navigate up the tree. If you're new to selectors, our guide to CSS selectors for scraping covers the syntax.

Read data off a found element:

import org.openqa.selenium.WebElement;

WebElement el = driver.findElement(By.cssSelector("span.text"));
String text = el.getText();                 // visible text
String href = el.getAttribute("href");      // any attribute
List<WebElement> all = driver.findElements(By.cssSelector("div.quote")); // many

Note the difference: findElement returns one element and throws NoSuchElementException if nothing matches; findElements returns a (possibly empty) list and never throws for "not found."

4. Explicit waits — the key to a non-flaky Selenium scraper

This is the single most important section. On a JavaScript site, the element you want often doesn't exist at the moment the page "loads" — it appears a fraction of a second later when a script runs. If you call findElement too early, you get an exception. The wrong fix is Thread.sleep(3000); the right fix is an explicit wait that polls until a condition is true, then continues immediately.

import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;
import java.time.Duration;

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

// Wait until the quotes have been rendered by JavaScript
wait.until(ExpectedConditions.presenceOfElementLocated(
        By.cssSelector("div.quote")));

Common conditions:

• presenceOfElementLocated — element is in the DOM (rendered).
• visibilityOfElementLocated — in the DOM and visible.
• elementToBeClickable — visible and enabled (use before .click()).
• textToBePresentInElementLocated — waits for specific text to load.

Use explicit waits liberally and never hard-code sleeps. This one habit is the difference between a scraper that runs reliably at 3 a.m. and one that fails intermittently for no obvious reason.

5. A complete working scraper

Let's put it together into a real, runnable Selenium scraper. This program opens a login page, submits credentials, waits for the confirmation message, extracts it, saves a screenshot for auditing, and writes the result to a file — a pattern that generalizes to most "log in, then read data" jobs.

import org.openqa.selenium.*;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;

public class LoginScraper {

    public static void main(String[] args) throws Exception {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new", "--window-size=1366,900");

        WebDriver driver = new ChromeDriver();
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

        try {
            // 1. Open the target page
            driver.get("https://testing-ground.scraping.pro/login");

            // 2. Fill and submit the login form
            driver.findElement(By.id("usr")).sendKeys("admin");
            driver.findElement(By.id("pwd")).sendKeys("12345");
            driver.findElement(By.cssSelector("input[value='Login']")).click();

            // 3. Wait for the result message, then extract it
            WebElement status = wait.until(
                ExpectedConditions.visibilityOfElementLocated(
                    By.cssSelector("#case_login h3")));
            String message = status.getText().trim();
            System.out.println("Status: " + message);

            // 4. Persist the extracted text
            Files.writeString(Path.of("status.txt"), message);

            // 5. Save a screenshot for the audit trail
            File shot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
            Files.copy(shot.toPath(), Path.of("screenshot.png"));

        } finally {
            driver.quit();   // runs even if something above throws
        }
    }
}

Run it (mvn compile exec:java or straight from your IDE) and you'll get status.txt with the extracted message and screenshot.png in the project folder. Note what changed from the classic version of this tutorial: no FirefoxDriver and manual JAR wiring, no Apache Commons IO for the file copy (java.nio.file handles it), an explicit wait instead of hoping the page is ready, and a finally block so the browser always shuts down.

6. Extracting many records and paginating

Real scraping means looping over lists and following "next" links. The shape:

import java.util.ArrayList;
import java.util.List;

List<String> quotes = new ArrayList<>();

while (true) {
    wait.until(ExpectedConditions.presenceOfElementLocated(
            By.cssSelector("div.quote")));

    for (WebElement q : driver.findElements(By.cssSelector("div.quote"))) {
        String text = q.findElement(By.cssSelector("span.text")).getText();
        String author = q.findElement(By.cssSelector("small.author")).getText();
        quotes.add(author + " — " + text);
    }

    // Follow pagination if a "Next" link exists; otherwise stop
    List<WebElement> next = driver.findElements(By.cssSelector("li.next a"));
    if (next.isEmpty()) break;
    next.get(0).click();
}

System.out.println("Collected " + quotes.size() + " quotes");

Using findElements for the "Next" link is the clean way to test existence without a try/catch: an empty list means the last page.

7. Staying unblocked at scale

A single Selenium browser is easy; a hundred of them scraping a defended site is not. As you scale, layer in:

• Realistic options — a real User-Agent, a normal window size, and a locale, so the browser doesn't look automated. Anti-bot systems flag the default headless fingerprint.
• Rotating proxies — spread requests across IPs to avoid rate limits and bans. Datacenter proxies are cheap; residential proxies survive tougher targets.
• Rate limiting and retries — pause between requests and back off on errors instead of hammering the server.
• CAPTCHA handling — for sites that challenge you, budget for a solving step.

These sit on top of the code above and are where most of the ongoing maintenance lives, because anti-bot defenses keep changing.

FAQ

Do I still need to download ChromeDriver for Selenium in Java? No. Since Selenium 4.6, Selenium Manager downloads and matches the correct driver automatically. Just add the selenium-java dependency and have a browser installed.

Why does findElement throw even though the element is on the page? It's almost always a timing problem: the element hadn't rendered yet when you queried. Wrap the lookup in a WebDriverWait with an ExpectedConditions check rather than searching immediately after get().

Selenium or Jsoup for Java scraping? Use Jsoup for static HTML — it's much faster and needs no browser. Use Selenium when the content is rendered by JavaScript or requires interaction like logging in, clicking, or scrolling.

Can Selenium run without opening a visible window? Yes — add --headless=new to ChromeOptions. It runs with no UI, which is faster and works on headless servers.

Selenium makes a Java scraper straightforward to write; keeping it running against shifting layouts, IP bans, and anti-bot updates is the ongoing work. If you'd rather receive clean data than maintain a browser fleet, scraping.pro offers this as a done-for-you web scraping service, with proxies, retries, and monitoring included.