Amazon Scraping with Selenium WebDriver in Java

Amazon is the reference target for anyone learning product-data extraction: enormous catalog, rich listings, and — because pricing, ratings, and availability change constantly — a genuine reason to automate. It's also one of the harder sites to scrape. The catalog is rendered with heavy JavaScript, the markup shifts, and Amazon runs aggressive bot detection. This tutorial covers Amazon scraping with Selenium WebDriver in Java end to end: a modern project setup, locating product fields reliably, walking pagination, writing results to CSV, and the anti-blocking measures that separate a demo from something that survives more than a handful of requests.

We use Selenium because it drives a real browser, which means it executes the JavaScript that builds Amazon's product grid — something a raw HTTP client can't do. If you're new to the library, our Selenium web scraping in Java guide and the what is Selenium WebDriver primer give useful background.

A note on responsible scraping

Before any code: scrape responsibly. Amazon's Terms of Service restrict automated access, so this tutorial is for learning the mechanics and for legitimate uses such as monitoring your own listings or public prices at modest volume. Take only public data, throttle your requests, don't attempt to bypass authentication, and consider Amazon's official Product Advertising API where it fits your use case. If you need Amazon data at commercial scale, a compliant managed service is usually the right answer — more on that at the end. Our overview of web scraping legality is worth a read.

Why Amazon is hard to scrape

Going in with clear eyes:

• JavaScript rendering. Much of a listing loads client-side, so you need a browser (or a rendering service), not just HttpClient.
• Shifting, A/B-tested markup. Class names and layouts vary by region, category, and experiment. Selectors that work today can break tomorrow, so build them defensively.
• Bot detection. Datacenter IPs, automation fingerprints, and fast request patterns trigger CAPTCHAs ("Enter the characters you see") and "Sorry, something went wrong" walls.
• Localization. Currency, language, and available fields differ across amazon.com, .co.uk, .de, and so on.

None of this makes Amazon un-scrapable — it makes discipline mandatory.

Project setup (Selenium 4, Maven)

The old way of doing this — downloading a driver binary, pointing a system property at it, instantiating FirefoxDriver — is obsolete. Selenium 4 ships Selenium Manager, which resolves and downloads the correct driver for your installed browser automatically. Add one dependency:

<!-- pom.xml -->
<dependencies>
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.28.0</version>
  </dependency>
</dependencies>

That's the whole setup. No webdriver.chrome.driver system property, no manual ChromeDriver download. (If you prefer to pin the driver explicitly, WebDriverManager still works, but for most projects Selenium Manager is one less thing to maintain.)

Configuring the browser

Create a Chrome driver with options tuned for scraping — a realistic User-Agent, a normal window size, and flags that keep Chrome stable on a server:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import java.time.Duration;

public class AmazonDriver {
    public static WebDriver create(boolean headless) {
        ChromeOptions options = new ChromeOptions();
        if (headless) {
            options.addArguments("--headless=new");
        }
        options.addArguments("--window-size=1920,1080");
        options.addArguments("--no-sandbox");
        options.addArguments("--disable-dev-shm-usage");
        options.addArguments("--disable-blink-features=AutomationControlled");
        options.addArguments(
            "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
            "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36");

        WebDriver driver = new ChromeDriver(options);
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(5));
        return driver;
    }
}

--disable-blink-features=AutomationControlled removes one of the obvious "I'm automated" signals; --headless=new is Chrome's current, more faithful headless mode. Note that Amazon detects vanilla headless more readily than a headed browser — during development, run headed (create(false)) so you can see CAPTCHAs and layout differences.

Locating product data

The original 2013 version of this scraper hit a hardcoded tag page and pulled innerHTML with a regex. That's brittle. The modern approach targets Amazon's search-results structure with CSS selectors and explicit waits.

On a search results page, each product is a container carrying a data-asin attribute. Inside it live the title, price, and rating. These selectors are commonly used and reasonably durable, but verify them against the live page — Amazon changes markup, so always confirm in DevTools before trusting a selector:

• Result container: div[data-component-type="s-search-result"]
• ASIN (product ID): the container's data-asin attribute
• Title: h2 span (the visible product name)
• Price: span.a-price span.a-offscreen (the full price as text, e.g. $29.99)
• Rating: span.a-icon-alt (e.g. 4.5 out of 5 stars)
• Product URL: h2 a or a.a-link-normal[href]

A small model class and an extractor that reads one result card:

record Product(String asin, String title, String price,
               String rating, String url) {}

import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.NoSuchElementException;

public static Product parseCard(WebElement card) {
    String asin = card.getAttribute("data-asin");
    String title = text(card, By.cssSelector("h2 span"));
    String price = text(card, By.cssSelector("span.a-price span.a-offscreen"));
    String rating = text(card, By.cssSelector("span.a-icon-alt"));
    String url = attr(card, By.cssSelector("h2 a"), "href");
    return new Product(asin, title, price, rating, url);
}

// null-safe helpers: a card may lack a price (out of stock) or rating (new item)
private static String text(WebElement scope, By by) {
    try { return scope.findElement(by).getText().trim(); }
    catch (NoSuchElementException e) { return null; }
}
private static String attr(WebElement scope, By by, String name) {
    try { return scope.findElement(by).getAttribute(name); }
    catch (NoSuchElementException e) { return null; }
}

The null-safe helpers matter: not every card has every field. Sponsored placements, out-of-stock items, and brand-new products routinely miss a price or rating, and calling getText() on a missing element throws. Catching NoSuchElementException per field keeps one odd card from killing the whole run.

Waiting for content

Because the grid renders with JavaScript, wait for the results to appear before reading them. Use an explicit WebDriverWait rather than Thread.sleep:

import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(15));
wait.until(ExpectedConditions.presenceOfElementLocated(
    By.cssSelector("div[data-component-type='s-search-result']")));

Handling pagination

The original scraped a single page and left "add a couple of lines" as an exercise. Here's how it's actually done. Amazon paginates search results with a &page=N URL parameter and a "Next" link at the bottom. The most robust approach is to drive the page parameter directly in a loop, stopping when a page returns no results:

import java.util.ArrayList;
import java.util.List;

public static List<Product> scrapeSearch(WebDriver driver,
                                         String keyword, int maxPages) {
    List<Product> all = new ArrayList<>();
    String base = "https://www.amazon.com/s?k=" +
                  java.net.URLEncoder.encode(keyword,
                      java.nio.charset.StandardCharsets.UTF_8);

    for (int page = 1; page <= maxPages; page++) {
        driver.get(base + "&page=" + page);

        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(15));
        try {
            wait.until(ExpectedConditions.presenceOfElementLocated(
                By.cssSelector("div[data-component-type='s-search-result']")));
        } catch (org.openqa.selenium.TimeoutException e) {
            System.out.println("No results on page " + page + " — stopping.");
            break;   // ran past the last page, or hit a block
        }

        List<WebElement> cards = driver.findElements(
            By.cssSelector("div[data-component-type='s-search-result']"));
        for (WebElement card : cards) {
            Product p = parseCard(card);
            if (p.title() != null) all.add(p);
        }
        System.out.printf("Page %d: %d products (running total %d)%n",
                          page, cards.size(), all.size());

        // polite, randomized delay between pages
        sleep(2500 + (long) (Math.random() * 2500));
    }
    return all;
}

private static void sleep(long ms) {
    try { Thread.sleep(ms); } catch (InterruptedException ignored) {}
}

Driving &page=N is more reliable than clicking "Next," because it avoids stale-element issues when the DOM re-renders and it degrades gracefully — when a page has no result cards you've either reached the end or been blocked, and either way you stop.

Writing results to CSV

Persist as you go, or at the end. A minimal, quote-safe CSV writer:

import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;

public static void writeCsv(List<Product> products, String path) throws IOException {
    try (BufferedWriter w = Files.newBufferedWriter(Path.of(path))) {
        w.write("asin,title,price,rating,url\n");
        for (Product p : products) {
            w.write(String.join(",",
                q(p.asin()), q(p.title()), q(p.price()),
                q(p.rating()), q(p.url())));
            w.write("\n");
        }
    }
}

// wrap in quotes and escape embedded quotes so commas in titles don't break columns
private static String q(String s) {
    if (s == null) return "";
    return "\"" + s.replace("\"", "\"\"") + "\"";
}

Product titles are full of commas, so quoting every field is not optional — skip it and your columns shift the first time a title contains a comma. For larger jobs, write to a database instead; see our notes on storing scraped data.

Tying it together

public static void main(String[] args) throws IOException {
    WebDriver driver = AmazonDriver.create(false);   // headed while developing
    try {
        List<Product> products = scrapeSearch(driver, "wireless headphones", 5);
        writeCsv(products, "amazon_products.csv");
        System.out.println("Saved " + products.size() + " products.");
    } finally {
        driver.quit();   // always quit — a leaked browser keeps running
    }
}

driver.quit() in a finally block is non-negotiable: an unclosed browser process lingers and, across many runs, exhausts server memory.

Avoiding blocks

A working scraper and an unblocked scraper are different achievements. Amazon will flag automation quickly without these measures:

• Rotate IP addresses. The single biggest factor. Datacenter IPs get blocked fast; route requests through rotating proxies — residential or mobile for stubborn blocks — so traffic doesn't all come from one address. Our guide on changing the WebDriver IP address shows the wiring.
• Throttle and randomize. The jittered delay between pages above is the minimum. Add variation to actions, and cap concurrency. Hammering guarantees a block.
• Reduce your fingerprint. A realistic User-Agent, a normal viewport, and disabling the automation flag (all set above) remove the obvious tells. Headed mode is detected less than headless.
• Handle CAPTCHAs. Amazon serves image challenges when suspicious. Detect the CAPTCHA page and route it to a CAPTCHA solving service, or back off and retry later from a fresh IP.
• Vary and validate. Watch for the "Sorry" and "Robot Check" pages; when you hit them, slow down, rotate IPs, and don't just plough on collecting garbage HTML. Our deep dive on anti-scraping protection covers the full arsenal.

Selenium vs. the alternatives in Java

Selenium is the standard, but it isn't the only Java option:

• Playwright for Java — modern API with built-in auto-waiting; often more stable and faster to write than Selenium for browser scraping.
• HtmlUnit — a headless Java browser with limited JavaScript support; lighter, but it struggles with Amazon's heavy JS.
• Amazon Product Advertising API — the official, compliant route for associates; no scraping needed where it covers your fields.

For a browser-driven approach today, Playwright for Java is well worth evaluating alongside Selenium. Both beat the old Firefox-plus-regex pattern comfortably.

FAQ

Is scraping Amazon legal? Amazon's Terms of Service restrict automated access, and legality depends on jurisdiction, the data, and how you use it. Scraping public data at modest volume for research or monitoring is generally lower-risk than large-scale commercial harvesting; don't bypass authentication, and prefer the official Product Advertising API where it fits. Review the applicable terms and laws for your jurisdiction before you run at scale.

Why does my Amazon scraper get blocked so fast? Almost always the IP and the request pattern. One datacenter IP firing rapid, evenly spaced requests is an obvious bot. Rotate residential proxies, add randomized delays, cap concurrency, and reduce your automation fingerprint.

Do I need Selenium, or can I use plain HTTP? Amazon renders much of a listing with JavaScript, so a plain HTTP client often gets incomplete HTML. A browser (Selenium or Playwright) executes that JavaScript. If you find a stable JSON endpoint the page calls, you can sometimes skip the browser — but expect defenses there too.

How do I handle Amazon's CAPTCHA? Detect the challenge page, then either solve it through a CAPTCHA-solving service or back off and retry from a fresh IP after a delay. Persistent CAPTCHAs usually mean your IPs and pacing need fixing first.

Do the CSS selectors in this guide always work? No — treat them as a well-informed starting point. Amazon changes markup and runs A/B tests, and selectors vary by region and category. Always confirm against the live page in DevTools and build defensively with null-safe field extraction.

Wrapping up

Modern Amazon scraping in Java is far cleaner than it was a decade ago: Selenium 4 with Selenium Manager removes the driver-wrangling, CSS selectors plus explicit waits replace brittle regex-on-innerHTML, and driving the &page=N parameter makes pagination robust. The genuinely hard part is staying unblocked — rotating proxies, randomized pacing, fingerprint hygiene, and CAPTCHA handling do more for your success rate than any parsing trick. When you need Amazon data reliably and at scale without running that arms race yourself, scraping.pro offers it as a managed service, including ongoing competitor price monitoring so you get clean, structured product data on a schedule instead of maintaining scrapers.