Web Scraping in Java with HtmlUnit

Web scraping is the act of fetching data from a website by downloading its HTML and parsing out the parts you care about. You can do it by hand, but the term almost always means the automated version: request a page, extract the fields, and save them somewhere useful for analysis. This guide covers web scraping in Java with HtmlUnit, a mature headless browser that handles HTTP, cookies, forms, and — when you need it — JavaScript, all from a single library and without a real browser installed.

If you are weighing your options first, our broader guide to web scraping with Java compares HtmlUnit against jsoup, Selenium, and Playwright. This article is the deep dive on HtmlUnit itself.

Web scraping vs. APIs

When a site wants to expose data to developers, it usually builds an API: a set of HTTP endpoints that return JSON or XML. An e-commerce API might list products at https://api.shop.com/products and a single product at https://api.shop.com/products/123. If a clean API exists and covers your needs, use it — it is easier and more stable than scraping.

The problem is that most sites offer no API, or one that is rate-limited, incomplete, or paywalled. Building and maintaining an API is a real cost, so plenty of data is only ever available as rendered HTML. That is where scraping comes in. The good news: almost anything you can see in a browser can be scraped. For the bigger picture, see what is web scraping.

Why HtmlUnit?

HtmlUnit is a GUI-less (headless) browser written in pure Java. It performs HTTP requests, models the page as a DOM you can query, manages cookies and sessions, submits forms, and can execute JavaScript through the bundled Rhino engine. Compared with the alternatives:

• jsoup is lighter and faster but is only an HTML parser — it does not run JavaScript or maintain browser state.
• Selenium / Playwright drive a real Chrome or Firefox, so they handle any modern site, but they need a browser binary, more memory, and more setup.
• HtmlUnit sits in the middle: no external browser, low overhead, good for form-driven and mostly-static sites, with optional JavaScript for lighter dynamic pages.

A note on the project's history: HtmlUnit moved to the org.htmlunit group ID a few years back (it was formerly net.sourceforge.htmlunit). Older tutorials still show the old coordinates and the old asText() method — both have changed. Use the modern coordinates and methods shown below.

Setup: Maven and Java

You need a current LTS JDK (Java 17 or Java 21 are the safe choices in 2026) and Maven. Add HtmlUnit to your pom.xml. Always check Maven Central for the latest release rather than copying a version blindly:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>4.x.x</version> <!-- use the current release -->
</dependency>

We will also export our results as JSON with Jackson:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.17.x</version>
</dependency>

A first request

Everything in HtmlUnit starts with a WebClient. It is configurable — proxy settings, the browser it emulates, redirect handling, and whether CSS and JavaScript run. For a mostly-static target, turning JavaScript and CSS off makes pages load faster and cuts noise. WebClient is AutoCloseable, so use try-with-resources so it is always shut down.

We will scrape Hacker News (https://news.ycombinator.com/) as a worked example — it has a clean, table-based layout that is ideal for learning selectors.

import org.htmlunit.WebClient;
import org.htmlunit.html.HtmlPage;

public class FirstRequest {
    public static void main(String[] args) {
        String baseUrl = "https://news.ycombinator.com/";
        try (WebClient client = new WebClient()) {
            client.getOptions().setCssEnabled(false);
            client.getOptions().setJavaScriptEnabled(false);
            // Suppress noisy JS warnings if you re-enable JavaScript later:
            client.getOptions().setThrowExceptionOnScriptError(false);

            HtmlPage page = client.getPage(baseUrl);
            System.out.println(page.asXml()); // the full rendered HTML
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

HtmlPage holds the parsed document. page.asXml() dumps the HTML, which is handy while you are figuring out the structure.

Selecting elements

HtmlUnit gives you several ways to reach an element:

• getHtmlElementById(String id) — by id.
• getFirstByXPath(String xpath) — the first XPath match.
• getByXPath(String xpath) — all matches, as a List.
• querySelector / querySelectorAll — CSS selectors, if you prefer them.

Open Hacker News, right-click a post, and choose Inspect. Each story lives in a <tr class="athing"> row; the score, author, and comment count sit in the following <tr>. Since the rows have no convenient IDs, XPath is the natural fit. The plan: grab every athing row, then pull each field with a relative XPath.

import org.htmlunit.html.*;
import java.util.List;

List<HtmlElement> rows = page.getByXPath("//tr[@class='athing']");

if (rows.isEmpty()) {
    System.out.println("No items found");
} else {
    for (HtmlElement row : rows) {
        String rankText = ((HtmlElement) row.getFirstByXPath(".//span[@class='rank']"))
                .asNormalizedText().replace(".", "");
        int rank = Integer.parseInt(rankText);

        long id = Long.parseLong(row.getAttribute("id"));

        HtmlAnchor titleLink = row.getFirstByXPath(".//span[@class='titleline']/a");
        String title = titleLink.asNormalizedText();
        String url = titleLink.getHrefAttribute();

        // The subtext row is the immediate sibling of the story row:
        HtmlElement author = row.getFirstByXPath(
                "./following-sibling::tr[1]//a[@class='hnuser']");
        String by = (author != null) ? author.asNormalizedText() : "(none)";

        HtmlElement scoreEl = row.getFirstByXPath(
                "./following-sibling::tr[1]//span[@class='score']");
        int score = (scoreEl != null)
                ? Integer.parseInt(scoreEl.asNormalizedText().replace(" points", ""))
                : 0;

        System.out.printf("%d. %s (%s) — %d points by %s%n", rank, title, url, score, by);
    }
}

A few things to note versus older HtmlUnit code:

• Use asNormalizedText(), not the deprecated asText(). It collapses whitespace the way a browser would.
• Hacker News restructured its markup over the years — modern rows use span.titleline and span.rank. Always inspect the live page; selectors from an old tutorial rot.
• Null-check anything that can be missing (job posts have no author or score). A single unguarded cast will throw and kill your loop, so guard each optional field.

Exporting to JSON

Printing to the console is fine for debugging, but you will want structured output. Define a plain Java record (a modern replacement for the old getters-and-setters POJO) and let Jackson serialize it.

public record HackerNewsItem(
        String title, String url, String author,
        int score, int rank, long id) {}

Then serialize inside the loop:

import com.fasterxml.jackson.databind.ObjectMapper;

ObjectMapper mapper = new ObjectMapper();
// build the item from the fields extracted above:
HackerNewsItem item = new HackerNewsItem(title, url, by, score, rank, id);
System.out.println(mapper.writeValueAsString(item));

Create the ObjectMapper once and reuse it — it is thread-safe and relatively expensive to construct. To write a whole list to a file:

mapper.writerWithDefaultPrettyPrinter()
      .writeValue(new java.io.File("hn.json"), items);

Submitting forms and logging in

Forms are where HtmlUnit shines, because it keeps cookies and session state between requests. Fetch the page, locate the form and its fields, set values, and "click" the submit button. The returned HtmlPage is the post-submit page, cookies already applied.

HtmlPage login = client.getPage("https://example.com/login");
HtmlForm form = login.getFormByName("loginForm");

form.getInputByName("username").type("myuser");
form.getInputByName("password").type("mypass");

HtmlPage afterLogin = form.getButtonByName("submit").click();
// afterLogin now carries the authenticated session for subsequent getPage() calls

For scraping behind authentication in general — session handling, tokens, and staying logged in — see our guide to scraping login-protected sites.

Handling JavaScript

If a target renders content with JavaScript, enable the engine and give the page time to run:

client.getOptions().setJavaScriptEnabled(true);
client.getOptions().setThrowExceptionOnScriptError(false);

HtmlPage page = client.getPage(url);
client.waitForBackgroundJavaScript(10_000); // wait up to 10s for XHR/timers
String html = page.asXml();

Be realistic about HtmlUnit's JavaScript support. Its engine is not a modern V8/Blink, so heavy single-page apps built on current React, Vue, or Angular often will not render correctly, and you may see script errors. HtmlUnit copes well with lighter jQuery-era pages and simple XHR, but for a genuinely JS-heavy site you have two better options:

1. Find the hidden API. Open the browser Network tab, watch for XHR/fetch calls returning JSON, and request those endpoints directly. This is faster and more reliable than rendering anything. See scraping dynamic content.
2. Use a real browser via Selenium or Playwright for Java. See Selenium web scraping.

Being a good citizen: proxies, delays, headers

To scrape at any scale without getting blocked, slow down and blend in:

• Set a realistic User-Agent and emulate a current browser: new WebClient(BrowserVersion.CHROME).
• Add delays between requests and cap concurrency — hammering a server is both rude and a fast route to a ban.
• Rotate IPs for larger jobs. HtmlUnit takes a proxy in the WebClient constructor; pair it with a pool of rotating proxies.
• Respect robots.txt and Terms of Service, and avoid collecting personal data without a lawful basis (GDPR, CCPA/CPRA).

try (WebClient client = new WebClient(BrowserVersion.CHROME, "proxy.host", 8080)) {
    // ...scrape...
}

If you hit CAPTCHAs, that is a signal you are being detected; back off, and if you must continue, look at CAPTCHA solving services.

FAQ

Is HtmlUnit still maintained in 2026? Yes. It is actively developed under the org.htmlunit group ID. If you are on an old net.sourceforge.htmlunit version, migrate — the API and coordinates have changed.

HtmlUnit or jsoup? Use jsoup when the data is in the initial HTML and you just need fast parsing and CSS selectors. Use HtmlUnit when you need browser behavior: cookies, form submission, multi-step navigation, or light JavaScript.

HtmlUnit or Selenium/Playwright? HtmlUnit needs no browser binary and is lighter, so it wins for form-driven or mostly-static sites. For modern JavaScript-heavy apps, drive a real browser with Selenium or Playwright.

Why do I get NullPointerException on some rows? Optional fields (author, score) are missing on certain items. Null-check every getFirstByXPath result before you cast or read it.

Can HtmlUnit render a React or Angular single-page app? Often not fully — its JavaScript engine is not a modern browser engine. Prefer the hidden-API approach or a real headless browser for those.

Wrapping up

HtmlUnit remains one of the most practical ways to do Java web scraping when you need real browser behavior without the weight of a full browser: HTTP, cookies, forms, and light JavaScript in one dependency. Start with JavaScript disabled for speed, lean on XPath or CSS selectors, use asNormalizedText(), null-check optional fields, and export clean JSON with Jackson. When a site outgrows HtmlUnit's script engine, move to hidden APIs or a real headless browser.

If you would rather have the data than build and maintain the scraper, scraping.pro runs this as a done-for-you web scraping service — you get clean, structured output on a schedule and skip the anti-bot arms race entirely.