Java Web Crawler for Email Extraction: A Simple Tutorial

A web crawler is a program that starts from one page, follows the links it finds, and keeps going — visiting page after page across a site. Add a bit of pattern matching on the way and you have an email crawler: a tool that walks a website and pulls out every email address it can find.

This tutorial builds a small but complete Java web crawler for exactly that. It stays on a single domain, keeps a queue of URLs to visit, fetches each page with JSoup, and extracts emails with a regular expression. The full source is at the end. Along the way we will fix the things a naive first version gets wrong — duplicate work, missing politeness, weak regex — so what you copy is worth running.

New to this? A crawler discovers and follows links; a scraper extracts data from a page. Most real jobs do both. If the distinction is fuzzy, see crawling vs. scraping. For the broader toolkit, our guide to web scraping in Java covers libraries beyond the one used here.

What you need

• JDK 17 or newer. Any current long-term-support Java works; the code uses nothing exotic.
• JSoup — the standard Java library for fetching and parsing HTML. It is actively maintained (1.18.x at the time of writing), handles messy real-world markup, and gives you CSS-selector queries like jQuery. Add it with Maven:

<dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.18.3</version>
</dependency>

Everything else — collections, regex, URI handling — is in the standard library.

How the crawler works

The design is a classic breadth-first crawl:

1. Start with one seed URL and record its host. We only follow links that stay on that host, so the crawl doesn't wander off into the whole internet.
2. Keep a queue of URLs still to visit and a set of URLs already seen (so we never fetch the same page twice).
3. Pop a URL, fetch the page, extract emails from its text, and collect its links.
4. Push any new same-host links onto the queue.
5. Repeat until the queue is empty or a page limit is reached.

The original version of this crawler stored URLs in an ArrayList and checked list.contains(url) before adding each one. That works, but contains scans the whole list every time — on a big site it turns the crawl quadratic. Using a HashSet for the "seen" check makes deduplication instant. That single change is the difference between a toy and something you can point at a real site.

Setup: seed, host, and email pattern

We grab the host of the seed URL to scope the crawl, compile the email regex once, and set up our queue and result collections.

String seedUrl = "https://example.com";
String host = URI.create(seedUrl).getHost();   // stay on this host only

// A pragmatic email pattern: local-part @ domain . tld
Pattern emailPattern = Pattern.compile(
    "[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}");

Deque<String> queue = new ArrayDeque<>();  // URLs to visit
Set<String> visited = new HashSet<>();     // URLs already fetched
Set<String> emails  = new LinkedHashSet<>(); // results, de-duplicated, ordered

queue.add(seedUrl);
visited.add(seedUrl);

A note on the regex: the original pattern (...@[a-zA-Z]+[.]{1}[a-zA-Z]{2,4}) misses perfectly valid addresses — subdomains (user@mail.example.com), digits or hyphens in the domain, and modern TLDs longer than four letters (.online, .company). The version above handles all of those. Email validation is famously impossible to do perfectly with regex, but this is a solid, practical middle ground for harvesting.

The main loop

Each iteration fetches one page, harvests emails, then enqueues new links.

int maxPages = 200;              // safety cap
int fetched  = 0;

while (!queue.isEmpty() && fetched < maxPages) {
    String url = queue.poll();

    Document doc;
    try {
        doc = Jsoup.connect(url)
                   .userAgent("Mozilla/5.0 (compatible; EmailCrawler/1.0)")
                   .timeout(10_000)
                   .get();
    } catch (IOException e) {
        System.out.println("Skip (" + e.getMessage() + "): " + url);
        continue;                 // a dead link must not kill the crawl
    }
    fetched++;

    // 1) Extract emails from the visible text
    Matcher m = emailPattern.matcher(doc.text());
    while (m.find()) {
        emails.add(m.group());
    }

    // 2) Collect new same-host links
    for (Element link : doc.select("a[href]")) {
        String next = link.attr("abs:href");   // resolve relative URLs
        next = next.replaceAll("#.*$", "");     // drop fragments
        if (next.isEmpty()) continue;

        String linkHost = URI.create(next).getHost();
        if (host.equals(linkHost) && visited.add(next)) {
            queue.add(next);                    // add() returns false if already seen
        }
    }

    // 3) Be polite — don't hammer the server
    try { Thread.sleep(1000); } catch (InterruptedException ignored) {}
}

Three details make this robust where a first attempt wouldn't be:

• The try/catch around Jsoup.connect means one 404 or timeout doesn't crash the whole run. On a real site you will hit dead links.
• abs:href tells JSoup to resolve relative links (/about) into absolute URLs, so URI.getHost() and deduplication work correctly.
• visited.add(next) returns false when the URL is already in the set, so a single line both checks and records — no separate contains call.

Being a good citizen

An email crawler that ignores the site it visits will get IP-banned fast, and may cross legal lines. Bake in politeness from the start:

• Rate limit. The Thread.sleep(1000) above spaces requests one second apart. Crawling as fast as the network allows is the quickest way to get blocked by anti-bot systems.
• Respect robots.txt. Before crawling a host, fetch https://host/robots.txt and honor its Disallow rules. It signals what the site owner is willing to have crawled.
• Set an honest User-Agent and a timeout, as shown.
• Cap the crawl. maxPages stops an "infinite depth" crawl from running forever on a large site.
• Scale carefully. Threads or an async client speed things up, but multiply your request rate — pair concurrency with per-host rate limiting and, on larger jobs, rotating proxies so a single IP isn't the bottleneck.

A word on legality and consent

Harvesting email addresses to send unsolicited mail is a different activity from crawling, and it is regulated. Bulk unsolicited email runs into CAN-SPAM in the US, CASL in Canada, and GDPR/PECR in the EU and UK, where an email address tied to a person is personal data and scraping it for marketing without a lawful basis can bring real penalties. Use this crawler for legitimate purposes — auditing your own sites, gathering published business contacts you're permitted to use, research — and always check the target's terms of service. The technique is neutral; how you apply it is not.

Full source

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

import java.io.IOException;
import java.net.URI;
import java.util.ArrayDeque;
import java.util.Deque;
import java.util.HashSet;
import java.util.LinkedHashSet;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class EmailCrawler {
    public static void main(String[] args) {
        String seedUrl = "https://example.com";
        String host = URI.create(seedUrl).getHost();

        Pattern emailPattern = Pattern.compile(
            "[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}");

        Deque<String> queue   = new ArrayDeque<>();
        Set<String>   visited = new HashSet<>();
        Set<String>   emails  = new LinkedHashSet<>();

        queue.add(seedUrl);
        visited.add(seedUrl);

        int maxPages = 200, fetched = 0;

        while (!queue.isEmpty() && fetched < maxPages) {
            String url = queue.poll();
            Document doc;
            try {
                doc = Jsoup.connect(url)
                           .userAgent("Mozilla/5.0 (compatible; EmailCrawler/1.0)")
                           .timeout(10_000)
                           .get();
            } catch (IOException e) {
                System.out.println("Skip: " + url + " (" + e.getMessage() + ")");
                continue;
            }
            fetched++;

            Matcher m = emailPattern.matcher(doc.text());
            while (m.find()) emails.add(m.group());

            for (Element link : doc.select("a[href]")) {
                String next = link.attr("abs:href").replaceAll("#.*$", "");
                if (next.isEmpty()) continue;
                String linkHost = URI.create(next).getHost();
                if (host.equals(linkHost) && visited.add(next)) {
                    queue.add(next);
                }
            }

            try { Thread.sleep(1000); } catch (InterruptedException ignored) {}
        }

        System.out.println("Pages fetched: " + fetched);
        System.out.println("Emails found:  " + emails.size());
        emails.forEach(System.out::println);
    }
}

Compile it with JSoup on the classpath, point seedUrl at a site you're allowed to crawl, and it will walk the domain and print every unique address it finds.

Where to take it next

This crawler is deliberately minimal so the mechanics are clear. Real projects usually add:

• Persistence — write results to a database or CSV as you go, rather than holding everything in memory, so a long crawl survives a restart.
• A frontier that scales — for very large sites, back the queue with Redis or a database instead of an in-memory deque.
• A dedicated framework — when a crawl grows beyond a single class, tools like crawler4j or Apache Nutch in the Java world (or Scrapy in Python) give you scheduling, concurrency, and retry handling out of the box.
• JavaScript rendering — JSoup only sees the server's HTML. If a site builds its content in the browser, you need a headless browser to see the same emails a visitor would.

If you'd rather skip the engineering and just receive clean, structured contact data — deduplicated, verified, and delivered on a schedule — that's something we run as a managed web scraping service, with compliance handled up front. Either way, you now have a working Java crawler to build on.