Web Scraping with Java: The Complete Guide

Web scraping with Java means collecting data from web pages programmatically — everything from downloading a single page to running a multithreaded crawler with proxy rotation, TOR, and workarounds for the usual roadblocks. This guide to web scraping in Java is built from simple to advanced: each section builds on the one before it, so you can read straight through or jump to the part you need. If you have ever wondered how to scrape a website in Java the right way, this is the end-to-end walkthrough — from your first HTTP request to a production-ready scraper.

Table of Contents

1. Introduction: What Web Scraping Is, and the Legal Landscape
2. Fetching the Page: HTTP Clients
3. Libraries for Parsing Content
4. Character Encodings and Charsets
5. Response Status and Headers
6. Working with Cookies
7. HTTPS and SSL
8. Using Proxies
9. Scraping Through TOR
10. Multithreading
11. JavaScript Rendering: Dynamic Sites
12. Anti-Bot Defenses: User-Agent, Delays, Retries
13. Storing URLs and Queues
14. Ready-Made Crawler Frameworks
15. Pros and Cons of Java for Scraping

1. Introduction: What Web Scraping Is, and the Legal Landscape

Web scraping is the automated extraction of data from web pages. It helps to split the process into two stages that are easy to confuse:

• Fetching (crawling) — retrieving HTML (or JSON/XML) over HTTP. This is the job of an HTTP client.
• Parsing (extraction) — turning raw HTML into a structure and picking out the data you want via selectors. This is the job of a parser.

Keep those two responsibilities separate and the rest of this guide falls into place: pick a solid HTTP client, hand its output to a good parser, then scale out.

The legal and ethical landscape

Before you write a line of code, keep a few things in mind. This is not legal advice — it is basic hygiene:

• robots.txt — a file at the site root (https://site.com/robots.txt) where the owner declares what crawlers are allowed to visit. It rarely carries legal weight on its own, but ignoring it is bad form and a common trigger for a ban.
• Terms of Service (ToS) may explicitly prohibit automated collection. Violating ToS is a contractual risk, and in some jurisdictions can escalate into something more serious.
• Personal data. Collecting personal data is regulated — GDPR in the EU and UK, CCPA/CPRA in California, and similar laws elsewhere. Tread carefully, and avoid scraping personal data unless you have a lawful basis.
• Server load. Aggressive scraping is effectively a denial-of-service attack on someone else's server. Add delays, cap your concurrency, and respect Retry-After and 429/503 responses.

For context in the US, the hiQ Labs v. LinkedIn litigation and the Computer Fraud and Abuse Act (CFAA) have shaped how courts view scraping publicly available data. The short version: scraping public data is not automatically a crime, but bypassing authentication, ignoring a cease-and-desist, or harvesting personal data can create real liability. Technically Java lets you do almost anything — the responsibility for what you collect and why is yours.

2. Fetching the Page: HTTP Clients

This is the foundation: without a correctly retrieved response, there is nothing to parse. Java gives you several HTTP client options.

2.1. The built-in java.net.http.HttpClient (Java 11+)

The modern, standard client. No dependencies, supports HTTP/2, and offers both synchronous and asynchronous modes.

import java.net.URI;
import java.net.http.*;
import java.time.Duration;

HttpClient client = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(10))
        .followRedirects(HttpClient.Redirect.NORMAL)
        .version(HttpClient.Version.HTTP_2)
        .build();

HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://example.com"))
        .timeout(Duration.ofSeconds(15))
        .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
        .header("Accept-Language", "en-US,en;q=0.9")
        .GET()
        .build();

HttpResponse<String> response =
        client.send(request, HttpResponse.BodyHandlers.ofString());

System.out.println("Status: " + response.statusCode());
String html = response.body();

An important gotcha with encoding: BodyHandlers.ofString() with no argument decodes the body as UTF-8. If the site uses a different charset (say, windows-1251 or ISO-8859-1), you will get garbled text here — see section 4. To avoid this, it is common to grab the bytes and determine the charset separately:

HttpResponse<byte[]> resp =
        client.send(request, HttpResponse.BodyHandlers.ofByteArray());
byte[] raw = resp.body();   // decode later, once you know the charset

2.2. OkHttp

A popular third-party library (from Square). Convenient API, connection pools, interceptors, and easy handling of proxies and cookies. A good default for serious scraping.

// build.gradle: implementation("com.squareup.okhttp3:okhttp:4.12.0")
import okhttp3.*;

OkHttpClient client = new OkHttpClient.Builder()
        .connectTimeout(Duration.ofSeconds(10))
        .readTimeout(Duration.ofSeconds(15))
        .build();

Request request = new Request.Builder()
        .url("https://example.com")
        .header("User-Agent", "Mozilla/5.0 ...")
        .build();

try (Response response = client.newCall(request).execute()) {
    int code = response.code();
    byte[] bytes = response.body().bytes();   // again: prefer bytes
}

2.3. Apache HttpClient 5

A mature, powerful, slightly more verbose library. Fine-grained control over connections, pools, and authentication. You will often find it in enterprise codebases.

// org.apache.httpcomponents.client5:httpclient5:5.x
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.classic.*;
import org.apache.hc.core5.http.io.entity.EntityUtils;

try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
    HttpGet httpGet = new HttpGet("https://example.com");
    byte[] body = httpclient.execute(httpGet, response -> {
        int status = response.getCode();
        return EntityUtils.toByteArray(response.getEntity());
    });
}

2.4. Fetching directly with jsoup

jsoup (see section 3) can download a page on its own. That is handy for prototypes, but its built-in HTTP engine is weaker on flexibility (proxies, pools, fine-tuning), so for production work the common pattern is: fetch with a powerful client, then parse with jsoup.

Document doc = Jsoup.connect("https://example.com")
        .userAgent("Mozilla/5.0 ...")
        .timeout(15_000)
        .get();

Which one to choose

3. Libraries for Parsing Content

Once you have the HTML, you need to extract the data. Do not parse with regular expressions — HTML is not a regular language, and a regex-based parser breaks on any non-standard markup.

3.1. jsoup — the workhorse

jsoup (the current version in 2026 is 1.22.2) implements the WHATWG HTML5 specification and builds the same DOM a browser would. It supports CSS selectors and XPath, and it forgives "dirty" HTML.

// implementation("org.jsoup:jsoup:1.22.2")
import org.jsoup.Jsoup;
import org.jsoup.nodes.*;
import org.jsoup.select.Elements;

Document doc = Jsoup.parse(html, "https://example.com"); // 2nd arg = baseUri for absolute links

// CSS selectors
Elements links = doc.select("a[href]");
for (Element link : links) {
    String text = link.text();
    String absUrl = link.absUrl("href");  // absolute URL
}

// Pinpoint selection
Element title = doc.selectFirst("h1.article-title");
String price = doc.select("span.price").text();

// Attributes
String img = doc.selectFirst("img").attr("src");

A few common selector tricks:

doc.select("div.product");              // by class
doc.select("#main-content");            // by id
doc.select("ul.menu > li");             // direct children
doc.select("a[href^=https]");           // attribute starts with
doc.select("table tr:nth-child(2n)");   // pseudo-selectors
doc.select("p:contains(Price)");        // by text

3.2. HtmlUnit — a "headless browser"

HtmlUnit is a GUI-less browser written in Java. It can execute JavaScript (with limitations), which sometimes lets you pull data from dynamic sites without reaching for heavyweight Selenium. Slower than jsoup, but more capable.

3.3. When you don't need HTML at all

Very often the data on a page is loaded by a separate request to an internal API (JSON). Open DevTools, go to the Network tab, find the relevant XHR/fetch call — and parse clean JSON with Jackson or Gson. This is far more reliable and faster than picking apart HTML. Always check for this path first.

// com.fasterxml.jackson.core:jackson-databind
ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(jsonString);
String name = root.path("data").path("name").asText();

4. Character Encodings and Charsets

The single most common source of garbled text — output like café instead of café, or a row of ��� replacement characters — has one root cause: the bytes were decoded with the wrong charset. This bites hardest on legacy pages served as ISO-8859-1 (Latin-1), Windows-1252, or windows-1251, and on any text with non-ASCII or accented characters.

Why it happens

HTTP delivers bytes. To get a string, you must decode those bytes with the correct charset. The charset may be declared in:

1. the HTTP header Content-Type: text/html; charset=windows-1251;
2. a <meta charset="..."> or <meta http-equiv="Content-Type"> inside the HTML;
3. nowhere at all (in which case you have to guess).

If you read the body as UTF-8 but the site is actually ISO-8859-1 (or another legacy code page), you get mojibake.

Rule: work with bytes, set the charset explicitly

jsoup solves this almost automatically if you hand it bytes or an InputStream rather than a finished string — it reads the charset itself from the header or the <meta> tag:

// Correct: jsoup detects the encoding from meta/charset
import java.io.ByteArrayInputStream;
import java.io.InputStream;

byte[] bytes = response.body();  // from HttpClient/OkHttp as byte[]
InputStream in = new ByteArrayInputStream(bytes);
Document doc = Jsoup.parse(in, null, "https://example.com");
//                             ^^^^ null = auto-detect charset

If the charset is known in advance, pass it explicitly:

Document doc = Jsoup.parse(in, "windows-1251", "https://example.com");

Manual decoding

When you are not parsing through jsoup, decode the bytes yourself:

import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;

// if you know the charset
String html = new String(bytes, Charset.forName("windows-1251"));

// for UTF-8
String html2 = new String(bytes, StandardCharsets.UTF_8);

Detecting the charset automatically

If the encoding is declared nowhere, charset-detection libraries help: juniversalchardet (a port of Mozilla's universalchardet) or ICU4J (CharsetDetector).

// org.apache.tika:tika-core can also detect a charset
import org.apache.tika.parser.txt.CharsetDetector;

CharsetDetector detector = new CharsetDetector();
detector.setText(bytes);
String charset = detector.detect().getName(); // e.g. "windows-1251"
String html = new String(bytes, charset);

Encoding checklist

• Never decode a response to a string as UTF-8 "blindly."
• Hand jsoup an InputStream/byte[], not a String.
• Save files with an explicit charset: Files.write(path, html.getBytes(StandardCharsets.UTF_8)).
• The console can mangle output too — set -Dfile.encoding=UTF-8 (already the default on Java 18+, per JEP 400) and check your terminal's charset.

5. Response Status and Headers

Don't parse the body blindly — first confirm the server actually returned a page, and not a redirect, a CAPTCHA, or an error.

HttpResponse<byte[]> resp = client.send(request, BodyHandlers.ofByteArray());

int status = resp.statusCode();
HttpHeaders headers = resp.headers();

// reading specific headers
String contentType = headers.firstValue("Content-Type").orElse("");
String server      = headers.firstValue("Server").orElse("");
long length        = headers.firstValueAsLong("Content-Length").orElse(-1);

// all headers
headers.map().forEach((k, v) -> System.out.println(k + ": " + v));

What to watch for:

Useful response headers: Content-Type (type and charset), Set-Cookie (see section 6), Location (redirect), Retry-After (pause on 429/503), and ETag/Last-Modified (for conditional requests and incremental crawling).

In OkHttp it is all analogous: response.code(), response.header("Content-Type"), response.headers().

6. Working with Cookies

Cookies are needed for sessions, authentication, passing "are you a bot?" checks, and preserving state between requests.

6.1. The built-in HttpClient + CookieManager

import java.net.*;

CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);

HttpClient client = HttpClient.newBuilder()
        .cookieHandler(cookieManager)   // cookies are now stored automatically
        .build();

// after a few requests, inspect the store:
CookieStore store = cookieManager.getCookieStore();
store.getCookies().forEach(c ->
        System.out.println(c.getName() + "=" + c.getValue()));

Now cookies travel between requests on their own — handy for login: first a POST with the username/password, then requests to protected pages using the same client.

6.2. OkHttp + CookieJar

// The simplest route is a ready-made PersistentCookieJar or your own in-memory CookieJar
OkHttpClient client = new OkHttpClient.Builder()
        .cookieJar(new JavaNetCookieJar(cookieManager))
        .build();

6.3. jsoup — passing cookies manually

jsoup stores cookies in the Connection.Response object, and you can forward them by hand:

Connection.Response login = Jsoup.connect("https://site.com/login")
        .data("username", "user", "password", "pass")
        .method(Connection.Method.POST)
        .execute();

Map<String, String> cookies = login.cookies();

Document page = Jsoup.connect("https://site.com/profile")
        .cookies(cookies)      // pass the session along
        .get();

Pitfall: in a multithreaded scraper, one shared session can "leak" state between threads. For different accounts/proxies, give each thread or worker its own cookie store.

7. HTTPS and SSL

Most sites are on HTTPS. Normally the built-in HttpClient/OkHttp handles it all: they verify the certificate against the trusted root CAs shipped with the JDK. You only need extra code in two situations.

7.1. Self-signed and "problematic" certificates

Occasionally a target site has a self-signed or expired certificate. It is tempting to disable verification:

// WARNING: this disables all SSL verification — for tests/trusted networks only!
TrustManager[] trustAll = new TrustManager[]{
    new X509TrustManager() {
        public void checkClientTrusted(X509Certificate[] c, String a) {}
        public void checkServerTrusted(X509Certificate[] c, String a) {}
        public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; }
    }
};
SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, trustAll, new SecureRandom());

HttpClient client = HttpClient.newBuilder()
        .sslContext(sc)
        .build();

Never do this in production — it opens the door to man-in-the-middle attacks. The correct approach is to add the specific certificate to a custom truststore:

keytool -import -alias mysite -file mysite.crt -keystore custom.jks

KeyStore ks = KeyStore.getInstance("JKS");
try (InputStream in = Files.newInputStream(Path.of("custom.jks"))) {
    ks.load(in, "changeit".toCharArray());
}
TrustManagerFactory tmf =
        TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, tmf.getTrustManagers(), null);

7.2. TLS fingerprinting (JA3) — advanced anti-bot

Serious defenses (Cloudflare and others) distinguish clients by their TLS fingerprint (JA3/JA4): the set of ciphers and extensions in the ClientHello. A stock Java client's fingerprint is "not browser-like," and it is easy to flag as a bot. Faithfully forging it on plain JDK is hard; in practice people use tools like curl-impersonate/uTLS-style wrappers, or route traffic through a real browser (section 11). It is a niche topic, but an important one for "hard" sites.

8. Using Proxies

Proxies let you spread load across different IPs, get around geo-restrictions, and reduce the risk of an IP ban during mass scraping. Types: datacenter (cheap, easily detected), residential (expensive, looks like a real user), and mobile (the "cleanest"). For sustained scraping you will usually want rotating proxies rather than a single static IP.

8.1. The built-in HttpClient

import java.net.*;

HttpClient client = HttpClient.newBuilder()
        .proxy(ProxySelector.of(new InetSocketAddress("proxy.host", 8080)))
        .build();

With proxy authentication by username/password:

Authenticator auth = new Authenticator() {
    @Override protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication("user", "pass".toCharArray());
    }
};
HttpClient client = HttpClient.newBuilder()
        .proxy(ProxySelector.of(new InetSocketAddress("proxy.host", 8080)))
        .authenticator(auth)
        .build();

JDK quirk: proxying may require the system property -Djdk.http.auth.tunneling.disabledSchemes="", otherwise Basic authentication on the CONNECT tunnel silently fails.

8.2. OkHttp (often more convenient for proxies)

Proxy proxy = new Proxy(Proxy.Type.HTTP,
        new InetSocketAddress("proxy.host", 8080));

OkHttpClient client = new OkHttpClient.Builder()
        .proxy(proxy)
        .proxyAuthenticator((route, response) -> {
            String credential = Credentials.basic("user", "pass");
            return response.request().newBuilder()
                    .header("Proxy-Authorization", credential)
                    .build();
        })
        .build();

OkHttp also supports Proxy.Type.SOCKS — which we will need for TOR.

8.3. Rotating proxies

For mass scraping you keep a pool of proxies and cycle through them, or swap on a ban. The simplest rotation:

List<Proxy> pool = loadProxies();
AtomicInteger idx = new AtomicInteger();

Proxy next() {
    return pool.get(idx.getAndIncrement() % pool.size());
}

In practice you also add: a liveness check for proxies, a ban-list of "burned" ones, binding a session/cookies to a specific proxy, and a dedicated OkHttpClient per proxy (they are cheap when they share a connection pool, but reuse them rather than recreating them per request).

9. Scraping Through TOR

TOR provides anonymity and free IP rotation, but it is slow, and many sites block exit nodes. It suits targeted tasks, not high-speed scraping.

9.1. Setup

1. Install TOR (the tor package or Tor Browser).
2. TOR runs a SOCKS5 proxy locally, usually on 127.0.0.1:9050 (Tor Browser uses 9150).
3. To rebuild the circuit (get a new IP), use the ControlPort (9051) with the NEWNYM command.

A minimal torrc:

SocksPort 9050
ControlPort 9051
CookieAuthentication 1

9.2. Requests through TOR (SOCKS5)

Proxy torProxy = new Proxy(Proxy.Type.SOCKS,
        new InetSocketAddress("127.0.0.1", 9050));

OkHttpClient client = new OkHttpClient.Builder()
        .proxy(torProxy)
        .build();

Request req = new Request.Builder()
        .url("https://check.torproject.org/api/ip")  // confirm we're on TOR
        .build();

try (Response resp = client.newCall(req).execute()) {
    System.out.println(resp.body().string()); // {"IsTor":true,...}
}

Important: for SOCKS5, DNS resolution must go through the proxy (remote DNS), otherwise the DNS query "leaks" from your real IP. OkHttp with Proxy.Type.SOCKS handles this; with plain JDK, watch out for DNS leaks.

9.3. Changing IP with the NEWNYM command

import java.io.*;
import java.net.Socket;

void newTorIdentity() throws IOException {
    try (Socket s = new Socket("127.0.0.1", 9051);
         PrintWriter out = new PrintWriter(s.getOutputStream(), true);
         BufferedReader in = new BufferedReader(
                 new InputStreamReader(s.getInputStream()))) {

        out.println("AUTHENTICATE \"\"");   // or a password, if set
        in.readLine();                       // wait for 250 OK
        out.println("SIGNAL NEWNYM");        // new circuit
        in.readLine();
    }
}

After NEWNYM, wait a moment (TOR takes a few seconds to build a new circuit) and don't fire the command too often.

10. Multithreading

Scraping is an I/O-bound task (most of the time you're waiting on the network), so parallelizing yields a huge speedup. The catch: don't take down the target site, and don't get yourself banned.

10.1. ExecutorService — the classic

import java.util.concurrent.*;

ExecutorService pool = Executors.newFixedThreadPool(10);
List<Future<String>> futures = new ArrayList<>();

for (String url : urls) {
    futures.add(pool.submit(() -> fetchAndParse(url)));
}

for (Future<String> f : futures) {
    try {
        String result = f.get(30, TimeUnit.SECONDS);
        // save the result
    } catch (Exception e) {
        // log + retry/skip
    }
}
pool.shutdown();

10.2. Virtual threads (Java 21+) — perfect for I/O

Virtual threads give you thousands of "cheap" threads idling on the network — exactly what scraping needs:

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (String url : urls) {
        executor.submit(() -> fetchAndParse(url));
    }
} // close() waits for all tasks to finish

10.3. CompletableFuture + the async HttpClient

List<CompletableFuture<HttpResponse<byte[]>>> calls = urls.stream()
        .map(u -> client.sendAsync(buildRequest(u), BodyHandlers.ofByteArray()))
        .toList();

CompletableFuture.allOf(calls.toArray(new CompletableFuture[0])).join();

10.4. What you absolutely must account for

• Per-domain concurrency limits. 200 threads hitting one site equals a DoS and an instant ban. Throttle it — for example, a Semaphore per domain.
• Throttling. Add delays/jitter between requests (see section 12).
• Thread safety. Don't share a jsoup Document across threads; use ConcurrentHashMap/ConcurrentLinkedQueue for shared collections and AtomicInteger/LongAdder for counters.
• Backpressure. Don't dump a million tasks into the queue at once — use a bounded queue (ArrayBlockingQueue) and CallerRunsPolicy so the producer slows down.
• One OkHttpClient/HttpClient per application. They are thread-safe and hold a connection pool; spawning one per request is an anti-pattern.

11. JavaScript Rendering: Dynamic Sites

jsoup only sees the HTML that came back in the response. If the site is an SPA (React/Vue/Angular) and the data is drawn in by JavaScript in the browser, it won't be in the raw HTML. Your options:

1. Find the internal API (see 3.3) — almost always the best path: faster, more stable, no browser.
2. Selenium WebDriver — drives a real browser (Chrome/Firefox via WebDriver). Heavy and slow, but it sees everything a user sees.
3. Playwright for Java — a modern alternative to Selenium from Microsoft: faster, a nicer API, and strong support for dynamic content and waits.
4. HtmlUnit — a lightweight built-in browser in Java; executes JS with limitations, but needs no external binaries.

// Selenium (headless Chrome)
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
driver.get("https://spa-site.com");

// wait for an element to appear
new WebDriverWait(driver, Duration.ofSeconds(10))
    .until(ExpectedConditions.presenceOfElementLocated(By.cssSelector(".item")));

String renderedHtml = driver.getPageSource();
// then hand it to jsoup for convenient parsing:
Document doc = Jsoup.parse(renderedHtml);
driver.quit();

Browser engines also solve some anti-bot problems along the way (a correct TLS fingerprint, running JS challenges), but they pay for it in resources: a single Chrome instance eats hundreds of megabytes of RAM, so scaling them out is expensive.

12. Anti-Bot Defenses: User-Agent, Delays, Retries

The larger the scrape, the higher the chance of a block. Basic "politeness" and camouflage:

• User-Agent. Set a realistic UA from a real browser; for mass jobs, rotate a list of UAs. The default Java UA is a red flag for a site.
• A full set of headers. A real browser sends Accept, Accept-Language, Accept-Encoding, Referer, and Sec-Fetch-*. Imitate them.
• Delays and jitter. A random pause (say 1–5 s) between requests instead of a steady "machine gun."
• Retries with exponential backoff on 429/5xx, respecting Retry-After.
• IP rotation (section 8) — the main tool against IP bans.
• CAPTCHAs. When a CAPTCHA appears, your options are: slow down, change IP, or use CAPTCHA-solving services (2Captcha/Anti-Captcha) — but that is a gray area, so weigh the risks.
• Cloudflare and JS challenges. Often only clearable through a real browser (section 11) or specialized bypass libraries.

A retry skeleton:

int attempts = 0;
while (attempts < MAX_RETRIES) {
    HttpResponse<byte[]> r = client.send(req, BodyHandlers.ofByteArray());
    int code = r.statusCode();
    if (code == 200) return r.body();
    if (code == 429 || code >= 500) {
        long backoff = (long) (Math.pow(2, attempts) * 1000)
                + ThreadLocalRandom.current().nextInt(500); // jitter
        Thread.sleep(backoff);
        attempts++;
    } else {
        break; // 403/404 — a retry won't help
    }
}

13. Storing URLs and Queues

A crawler is, at heart, a traversal of a graph of links. You need two structures:

• The queue (frontier) — URLs still waiting to be crawled.
• The visited set — so you don't hit the same URL twice or loop forever.

In-memory (for small jobs)

Queue<String> frontier = new ConcurrentLinkedQueue<>();
Set<String> visited = ConcurrentHashMap.newKeySet();

if (visited.add(url)) {       // add returns false if it was already there
    frontier.offer(url);
}

Parsing the URL

Before you normalize or filter a link, you need to break it into its parts (scheme, host, port, path, query, fragment). Java has the built-in java.net.URI for this:

import java.net.URI;

URI uri = URI.create("https://Site.com:443/catalog/item?id=7&ref=a#section");

String scheme   = uri.getScheme();    // https
String host     = uri.getHost();      // Site.com
int    port     = uri.getPort();      // 443
String path     = uri.getPath();      // /catalog/item
String query    = uri.getQuery();     // id=7&ref=a
String fragment = uri.getFragment();  // section

// resolving a relative link into an absolute one:
URI abs = uri.resolve("../other");    // https://Site.com:443/other

jsoup does the same thing for links found on a page automatically — link.absUrl("href") (see section 3) returns an already-absolute URL, based on the baseUri. Breaking a URL into components with URI is specifically for the normalization step that follows. The same URL-parsing principles carry over to other language stacks, too.

What matters

• URL normalization before deduplication (building on URL parsing): strip #... anchors, sort query parameters, lowercase the host, and decide about http/https and the trailing slash. Otherwise site.com/a and site.com/a/ count as different.
• Deduplication at scale. Keeping hundreds of millions of URLs in a HashSet is unrealistic memory-wise — use a Bloom filter (Guava's BloomFilter) or move the set into Redis/a database.
• Persistence. For long crawls, the queue and visited set must survive a restart: Redis (List/Set), Kafka/RabbitMQ as a task queue, a relational DB, or an embedded store.
• Priorities. Sometimes you need a PriorityBlockingQueue (important sections first) or a crawl-depth limit.
• Distribution. At multi-machine scale, move the queue and dedup into an external broker/Redis so workers don't duplicate work.

A typical architecture: a producer pulls a URL off the frontier, a pool of workers fetches and parses it, and the extracted links are filtered, normalized, checked against the visited set, and put back on the frontier; the data is written to storage.

14. Ready-Made Crawler Frameworks

To avoid writing a crawler from scratch, several ready-made solutions exist:

• crawler4j — a simple multithreaded crawler in Java; quick to get started.
• webmagic — a flexible framework (inspired by Scrapy) with pipelines and a scheduler.
• Apache Nutch — a heavy, scalable, industrial-grade crawler (often paired with Hadoop/Solr).
• StormCrawler — a distributed crawler on top of Apache Storm for streaming processing.

For most practical tasks, the combination of "OkHttp/HttpClient + jsoup + your own thread pool + a Redis queue" is more than enough, and it is more transparent than a framework's "magic."

15. Pros and Cons of Java for Scraping

The upsides of Java

• Performance and concurrency. The JVM handles high concurrency well; virtual threads (Java 21+) take I/O-bound scraping to a new level.
• A mature ecosystem. jsoup, OkHttp, Apache HttpClient, Selenium/Playwright, Jackson — all stable and well documented.
• Reliability and type safety. Strong typing catches errors at compile time; convenient for large, long-lived crawlers.
• Integration with an enterprise stack. Easy to embed in a Spring service and wire up Kafka, databases, and monitoring.
• Cross-platform. One JAR — run it anywhere.

The downsides

• Verbosity. More code than Python; a quick prototype with requests + BeautifulSoup is shorter to write.
• Fewer scraping-specific tools. In Python the scraping ecosystem (Scrapy and friends) is richer and more popular.
• JavaScript sites are painful. Without a browser (Selenium/Playwright) you can't get the dynamic content, and browsers are heavy and hungry.
• The anti-bot arms race. TLS fingerprinting, behavioral analysis, CAPTCHAs — all of it needs ongoing maintenance; a stock Java client is easy to detect.
• Encodings. Legacy charsets like windows-1251 or ISO-8859-1 and non-ASCII text require care (section 4) — a classic source of bugs.
• Fragility. Any scraper breaks when a site's markup changes; selectors need maintenance.

When Java is a good choice

A big, long-lived, high-load crawler; a team already on a Java stack; a need for hard multithreading and infrastructure integration. For a one-off "grab a table from a single page," Python is usually faster in development time.

Want to skip the maintenance treadmill? Building and running a resilient scraper — proxy rotation, anti-bot handling, encoding edge cases, JavaScript rendering — is exactly the kind of work that quietly eats engineering time. If you'd rather just receive clean, structured data, scraping.pro can run this for you as a done-for-you data extraction service (custom scraping, DaaS, and monitoring).

Library versions are current as of 2026 (jsoup 1.22.2). Before you build, check the latest releases on Maven Central — APIs may have changed since.