Email Scraping in Python: Build a Simple Email Crawler

A question that comes up again and again is how to collect email addresses from websites automatically. The interest is understandable: anyone who works with outreach, lead research, or contact discovery eventually wonders whether the process can be automated instead of copying addresses by hand. There are ready-made tools for this — for example, the GSA Email Spider is a packaged solution you can buy and run without writing a single line of code.

But if you want to actually understand what happens under the hood — and pick up some practical web-scraping skills along the way — it is surprisingly easy to build your own crawler. In this tutorial we will write one from scratch in Python 3. The script is intentionally small so the core idea stays visible, but it is fully working and will happily pull real addresses off live pages. Once you understand it, extending it with extra features is straightforward.

What the crawler actually does

Before touching any code, it helps to picture the whole flow. The crawler is a classic breadth-first web walker with one extra job bolted on:

1. Start from one or more seed URLs that you provide.
2. Take the next URL from a queue and download the page.
3. Scan the page text for anything that looks like an email address and save it.
4. Find every link on the page and add the new, unseen ones back to the queue.
5. Repeat until the queue is empty (or until you decide to stop it).

That is the entire concept. Everything below is just the plumbing that makes those five steps happen reliably.

What you will need

The script leans on two excellent third-party packages plus a few modules from the standard library:

• Requests — the de-facto standard library for making HTTP requests in Python. It turns "download this page" into a one-line call.
• Beautiful Soup 4 — a forgiving HTML parser that lets you pull elements out of a page without writing fragile string-matching code.

Install both with pip:

pip install requests beautifulsoup4

The rest of the imports — urllib.parse, collections, and re — ship with Python itself, so there is nothing extra to install for those.

Step 1 — Import the libraries

We start by pulling in everything we need:

from bs4 import BeautifulSoup
import requests
import requests.exceptions
from urllib.parse import urlsplit
from collections import deque
import re

Here is what each one is for:

• BeautifulSoup parses the downloaded HTML so we can hunt for links.
• requests fetches pages, and requests.exceptions lets us catch the errors it can raise.
• urlsplit breaks a URL into its parts so we can rebuild relative links into full ones.
• deque gives us an efficient double-ended queue for the list of pages still to visit.
• re provides regular expressions, which we use to recognise email addresses.

Step 2 — Set up the queue of pages to visit

Next we define where the crawl begins. For this walkthrough I am pointing the crawler at the contact page of the Electronic Frontier Foundation, a public-interest organisation that openly lists contact addresses — a clean, real-world target for a demo. You can drop in any starting URL (or several) that you have permission to crawl:

# a queue of URLs still to be crawled
new_urls = deque(['https://www.eff.org/about/contact'])

You might wonder why this is a deque rather than a plain list. A deque supports fast removal from the front (popleft), which is exactly how we will consume URLs — first in, first out — so it fits the access pattern better than a list.

Step 3 — Remember what we have already seen

To avoid downloading the same page twice, we keep a record of every URL we have already handled. A set is ideal here because it stores unique values and checks membership quickly:

# URLs we have already processed
processed_urls = set()

Step 4 — Collect the emails

The harvested addresses also go into a set, which conveniently de-duplicates them for us — if the same address shows up on five different pages, we still only keep one copy:

# the addresses we have found so far
emails = set()

Step 5 — The main loop

Now the work begins. We keep going as long as there is anything left in the queue. Each iteration pulls the next URL off the front and immediately records it as processed so we never revisit it:

# work through the queue until it runs dry
while new_urls:

    # take the next URL and mark it as done
    url = new_urls.popleft()
    processed_urls.add(url)

Step 6 — Work out the base URL

Many links on a page are relative (for example /about instead of the full https://example.com/about). To turn those into something we can actually fetch, we need the page's scheme and host, plus its directory path:

    # break the URL apart so we can resolve relative links later
    parts = urlsplit(url)
    base_url = "{0.scheme}://{0.netloc}".format(parts)
    path = url[:url.rfind('/') + 1] if '/' in parts.path else url

base_url ends up as something like https://www.eff.org, and path holds the folder the current page lives in — both come in handy when we rebuild links.

Step 7 — Download the page (and survive failures)

Real websites are messy: links break, hosts time out, and some URLs are malformed. We fetch the page inside a try block and simply skip to the next one if anything goes wrong, so a single bad link can never crash the whole crawl:

    # download the page, skipping anything that errors out
    print("Processing %s" % url)
    try:
        response = requests.get(url)
    except (requests.exceptions.MissingSchema,
            requests.exceptions.ConnectionError):
        # ignore pages that fail to load
        continue

Step 8 — Pull out the email addresses

With the page in hand, we scan its raw text for anything matching the shape of an email address. The pattern below is a deliberately simple regular expression for matching email addresses — it is not a bullet-proof RFC-compliant validator, but it does the job for crawling and is easy to read:

    # find every address on the page and merge it into our set
    new_emails = set(re.findall(
        r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+",
        response.text, re.I))
    emails.update(new_emails)

The re.I flag makes the match case-insensitive, and using a set again means duplicates on the same page collapse into one entry automatically.

Step 9 — Find more pages to crawl

Extracting emails from one page is only half the story. The "crawling" part is discovering links to other pages and feeding them back into the queue. This is where Beautiful Soup shines — we hand it the HTML and ask for every anchor tag:

    # parse the HTML so we can walk its links
    soup = BeautifulSoup(response.text, "html.parser")

    # look at every <a> element on the page
    for anchor in soup.find_all("a"):

Tip: passing an explicit parser such as "html.parser" (or "lxml" if you have it installed) silences Beautiful Soup's warning and keeps behaviour consistent across machines.

Not every <a> tag actually has an href, so we guard against that:

        # grab the link target, defaulting to empty if there is none
        link = anchor.attrs["href"] if "href" in anchor.attrs else ''

A link that starts with / is relative to the site root, so we prepend the base URL. A link that does not start with http at all is relative to the current folder, so we prepend the path:

        # rebuild relative links into absolute ones
        if link.startswith('/'):
            link = base_url + link
        elif not link.startswith('http'):
            link = path + link

Finally, if we end up with a link we have neither queued nor visited yet, we add it to the back of the queue for later:

        # queue any genuinely new link
        if link not in new_urls and link not in processed_urls:
            new_urls.append(link)

The complete crawler

Put all of those pieces together and you get a compact, working email crawler:

from bs4 import BeautifulSoup
import requests
import requests.exceptions
from urllib.parse import urlsplit
from collections import deque
import re

# a queue of URLs still to be crawled
new_urls = deque(['https://www.eff.org/about/contact'])

# URLs we have already processed
processed_urls = set()

# the addresses we have found so far
emails = set()

# work through the queue until it runs dry
while new_urls:

    # take the next URL and mark it as done
    url = new_urls.popleft()
    processed_urls.add(url)

    # break the URL apart so we can resolve relative links later
    parts = urlsplit(url)
    base_url = "{0.scheme}://{0.netloc}".format(parts)
    path = url[:url.rfind('/') + 1] if '/' in parts.path else url

    # download the page, skipping anything that errors out
    print("Processing %s" % url)
    try:
        response = requests.get(url)
    except (requests.exceptions.MissingSchema,
            requests.exceptions.ConnectionError):
        continue

    # find every address on the page and merge it into our set
    new_emails = set(re.findall(
        r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+",
        response.text, re.I))
    emails.update(new_emails)

    # parse the HTML so we can walk its links
    soup = BeautifulSoup(response.text, "html.parser")

    # look at every <a> element on the page
    for anchor in soup.find_all("a"):
        link = anchor.attrs["href"] if "href" in anchor.attrs else ''
        if link.startswith('/'):
            link = base_url + link
        elif not link.startswith('http'):
            link = path + link
        if link not in new_urls and link not in processed_urls:
            new_urls.append(link)

Going further

The script above is deliberately bare-bones so the logic stays clear. Left as it is, it will crawl outward forever and keep its results only in memory. Here are a few practical upgrades worth adding once you are comfortable with the basics.

Save the results to a file

Right now the addresses vanish when the program ends. Writing them out — for example as CSV — makes the crawl actually useful:

import csv

with open('emails.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    for address in sorted(emails):
        writer.writerow([address])

Put a limit on the crawl

A crawler that follows every link can wander across an entire domain (or the whole web). Cap it with a simple counter so it stops after a fixed number of pages:

MAX_PAGES = 100
count = 0

while new_urls and count < MAX_PAGES:
    count += 1
    # ... rest of the loop ...

Be a polite, well-behaved bot

Hammering a server with rapid-fire requests is rude and can get you blocked. A few small additions go a long way:

• Add a timeout so a slow page cannot hang the crawler: requests.get(url, timeout=10).
• Send a descriptive User-Agent header identifying your bot.
• Sleep briefly between requests with time.sleep(1).
• Check the site's rules first using urllib.robotparser, which reads and interprets a site's robots.txt for you.

import time
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://www.eff.org/robots.txt")
rp.read()

if rp.can_fetch("*", url):
    response = requests.get(
        url,
        timeout=10,
        headers={"User-Agent": "MyEmailCrawler/1.0"},
    )
    time.sleep(1)

Tighten the email pattern

The regex used here is intentionally loose and will occasionally match things that are not real addresses. If precision matters, swap in one of the stricter patterns discussed in this roundup of regular expressions for matching email addresses.

A word on responsible use

A crawler is just a tool, and the same code that powers legitimate contact research can also fuel spam. Before you point this at a real site, make sure you are on solid ground: respect each site's robots.txt and terms of service, crawl gently so you do not overload anyone's server, and remember that data protection rules such as the GDPR and the CAN-SPAM Act govern how harvested addresses may be stored and used. Scraping a public page is one thing; blasting unsolicited mail to everyone you find is another.

Wrapping up

That is all there is to it. In well under a hundred lines you have a functional email crawler that demonstrates the core building blocks of web scraping in Python: queuing URLs, fetching pages with Requests, parsing HTML with Beautiful Soup, and pulling structured data out with regular expressions. Treat it as a starting point — add persistence, depth limits, concurrency, or smarter filtering as your needs grow.

Questions, ideas, or improvements? They are always welcome. Happy (and responsible) crawling!