Email Scraping in Python: Build a Simple Email Crawler

Project Honey Pot has been counting the machines that scrape addresses off web pages since 2004. Its live tally, read on 13 August 2026, stands at 914,764 harvesters identified and 3,870,311,720 unique spam messages received. That is the company you keep the moment you point a script at a contact page. It is also the reason the interesting part of this task is not the Python.

The Python is small. Under a hundred lines will walk a site, pull addresses out of the HTML and hand you a list. This tutorial builds exactly that, runs it against a controlled three-page site, shows the server log, and then rebuilds it, because the small version fails in ways that never show up in the console. Packaged tools exist as well. GSA has sold an Email Spider for Windows for years and its manual still describes the product as "collecting E-Mails as well as phone and fax numbers from websites", though on 13 August 2026 the vendor's own product URL for it lands on a newer product, Scrape Genie. The trouble with every packaged harvester is the same: you cannot see what it does, and you own the consequences anyway. Writing forty lines yourself is the cheaper way to learn where the boundaries of automated collection actually sit.

Everything below was run on Python 3.11.15 with requests 2.34.2 and beautifulsoup4 4.15.0, on 13 August 2026. The current Python at that date is 3.14.7, released 5 August 2026. Nothing in the code needs anything newer than Python 3.6.

Before the code: who owns the address you just saved

Most write-ups on this topic end with a paragraph about being nice. That is backwards. The legal exposure of an email crawler is larger than its technical difficulty, and it lands on you the second the addresses hit disk, not when you send anything.

An email address is personal data if it identifies a person. The UK regulator states the test plainly in its business-to-business marketing guidance: "If you can identify an individual either directly or indirectly it will constitute personal data even if they are acting in their business capacity", giving john.smith@company.com as its own example. A generic info@ box usually is not personal data. A named mailbox at the same company is. A crawler cannot tell the difference, so it collects both and you inherit the harder case.

GDPR: the clock starts when the file is written

Scraping addresses in the EU or UK puts you under Article 14 of the GDPR, the article that covers personal data you did not get from the person. It is worth reading in full; Article 14 as retained in UK law is the easiest version to link. Two deadlines matter to a crawler operator.

You must tell each person you hold their data "within a reasonable period after obtaining the personal data, but at the latest within one month". If you intend to email them, the deadline moves earlier: "at the latest at the time of the first communication". So the notice is not something you bolt on later. Your first message to a scraped address has to carry it.

GDPR does not care that the address was public. In March 2019 the Polish supervisory authority UODO fined Bisnode 943,000 PLN under Article 14 for collecting personal data from public registers and informing only the fraction of people whose email addresses it happened to have. The company argued that postal notice to the rest would be a disproportionate effort. On 20 September 2023 the Polish Supreme Administrative Court dismissed the cassation appeal, case III OSK 2538/21, holding that transparency is the rule and exceptions to it are read narrowly. That is the case every "public data is fair game" argument runs into.

The disproportionate-effort escape hatch also moved in the UK. On 5 February 2026, Schedule 9 of the Data (Use and Access) Act 2025 removed point (b) of Article 14(5) from the UK GDPR and re-enacted its substance as new points (e) and (f), commenced by S.I. 2026/82. The exemption survives, but any guidance or contract clause that cites "Article 14(5)(b)" against a UK controller is now pointing at a paragraph that no longer exists.

CAN-SPAM: harvesting is the aggravating factor

United States law does not ban address harvesting outright, and articles that say it does are wrong. What CAN-SPAM does is make harvesting the thing that turns a civil problem into a criminal one. The FTC's own compliance guide for business lists, among the conduct that can bring criminal penalties including imprisonment, "harvesting email addresses or generating them through a dictionary attack".

The civil arithmetic is worse than most people assume, because the unit is the message. The same guide states: "Each separate email in violation of the law is subject to penalties of up to $53,088." Send one non-compliant blast to ten thousand harvested addresses and the statutory ceiling is a number with nine digits in it.

That $53,088 is still current in August 2026, and the reason is odd enough to be worth knowing. Federal civil penalties are normally re-indexed every January. The OMB memorandum M-26-11, dated 17 April 2026, cancelled the 2026 adjustment because the Bureau of Labor Statistics could not produce the October 2025 CPI data during the shutdown, and told agencies to "continue using the 2025 civil monetary penalty levels as applicable". The figure is frozen, not stale.

PECR: the consent rule, and who it does not cover

In the UK, sending the mail is governed separately from holding the address. Regulation 22 of PECR says a person "shall neither transmit, nor instigate the transmission of, unsolicited communications for the purposes of direct marketing by means of electronic mail unless the recipient of the electronic mail has previously notified the sender that he consents". The soft opt-in in 22(3) only helps if you obtained the address during a sale or negotiation with that same person, which a crawler by definition did not.

There is a real carve-out, and it is the one B2B outreach lives in. The ICO states that "The PECR rule on direct marketing by electronic mail does not apply to corporate subscribers". Mail to a limited company is outside regulation 22. Sole traders and most partnerships are treated as individuals, and UK GDPR still applies to any address that names a person, so the carve-out is narrower than the people quoting it usually think.

Two things changed the stakes here. On 5 February 2026 the Data (Use and Access) Act 2025 added a charity soft opt-in as new regulation 22(3A). The same date brought in Schedule 13, which replaced Schedule 1 of PECR so that breaches of regulations 19 to 24, regulation 22 among them, now draw their ceiling from section 157 of the Data Protection Act 2018. That is the higher maximum amount: "£17,500,000 or 4% of the undertaking's total annual worldwide turnover in the preceding financial year, whichever is higher". PECR used to cap out at £500,000. Enforcement at the old ceiling was already routine: on 20 January 2026 the ICO announced £225,000 of fines in one go, including £105,000 against ZMLUK Limited for 67 million marketing emails sent between January and July 2023 on third-party data.

Canada draws the hardest line

Canada is the jurisdiction where the crawler itself is the offence. The Office of the Privacy Commissioner's CASL compliance guidance defines address harvesting as "collecting electronic addresses, such as those for email, instant messaging and social media by the use of computer programs" and states that "With very limited exceptions, PIPEDA prohibits address harvesting". The same page disposes of the usual justification in one sentence: "It cannot be assumed that people whose electronic addresses are posted online are necessarily interested in receiving commercial offers."

Four jurisdictions, four different answers, and none of them is "public means free". Write the code with that in the back of your head.

What the crawler actually does

The crawler is a breadth-first web walker with one extra job bolted on:

1. Start from one or more seed URLs that you provide.
2. Take the next URL from a queue and download the page.
3. Scan the page for anything that looks like an email address and save it.
4. Find every link on the page and add the new, unseen ones back to the queue.
5. Repeat until the queue is empty, or until a limit you set stops it.

Everything else is plumbing that keeps those five steps from lying to you.

What you will need

Two third-party packages plus a few standard-library modules:

• Requests: the de-facto standard HTTP client for Python. Current release 2.34.2, published 14 May 2026, requires Python 3.10 or newer.
• Beautiful Soup 4: a forgiving HTML parser. Current release 4.15.0, published 7 June 2026; the documentation matches that version.

Install both with pip:

pip install requests beautifulsoup4

The rest of the imports ship with Python: urllib.parse, collections, re and urllib.robotparser.

Step 1. Import the libraries

from bs4 import BeautifulSoup
import requests
import requests.exceptions
from urllib.parse import urlsplit
from collections import deque
import re

BeautifulSoup parses the HTML so you can hunt for links. requests fetches pages, and requests.exceptions gives you the error classes to catch. urlsplit breaks a URL into parts. deque is a double-ended queue with a fast popleft. re recognises the addresses.

Step 2. Three containers, and why each is that type

The whole crawl runs on three variables, and the type of each one carries an argument.

# a queue of URLs still to be crawled
new_urls = deque(['https://www.eff.org/about/contact'])

# URLs already processed
processed_urls = set()

# the addresses found so far
emails = set()

The seed here is the contact page of the Electronic Frontier Foundation, a public-interest organisation that publishes six addresses on it openly. An earlier version of this article gave the URL as /about/contacts in the prose and /about/contact in the code. Only the second one exists; the first returns 404. Use any starting URL you have permission to crawl.

deque is chosen over a list for popleft, which is O(1) where a list's pop(0) is O(n). That reasoning is correct and incomplete, and the gap costs you an hour per crawl later. Hold that thought until the section on scale.

The two set objects give you fast membership tests and free de-duplication. If the same address appears on five pages, you still keep one copy.

Step 3. The main loop

Keep going while the queue has anything in it. Each pass pulls the next URL off the front and records it as processed immediately, so a link that appears twice never causes a second download:

# work through the queue until it runs dry
while new_urls:

    # take the next URL and mark it as done
    url = new_urls.popleft()
    processed_urls.add(url)

Step 4. Work out the base URL

Many links on a page are relative: /about rather than https://example.com/about. To fetch those you need the scheme, the host and the directory the current page lives in:

    # break the URL apart so we can resolve relative links later
    parts = urlsplit(url)
    base_url = "{0.scheme}://{0.netloc}".format(parts)
    path = url[:url.rfind('/') + 1] if '/' in parts.path else url

base_url comes out as something like https://www.eff.org, and path holds the folder. This is the piece that will turn out to be wrong.

Step 5. Download the page and survive failures

Links break, hosts hang, and some URLs are malformed. Fetch inside a try and skip on failure so one bad link cannot end the crawl:

    # download the page, skipping anything that errors out
    print("Processing %s" % url)
    try:
        response = requests.get(url)
    except (requests.exceptions.MissingSchema,
            requests.exceptions.ConnectionError):
        # ignore pages that fail to load
        continue

Step 6. Pull out the email addresses

Scan the raw response text for anything shaped like an address. The pattern is a deliberately simple regular expression for matching email addresses, not a validator:

    # find every address on the page and merge it into our set
    new_emails = set(re.findall(
        r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+",
        response.text, re.I))
    emails.update(new_emails)

re.I makes the match case-insensitive.

Step 7. Find more pages to crawl

Discovering links is the crawling half. Hand the HTML to Beautiful Soup and ask for every anchor:

    # parse the HTML so we can walk its links
    soup = BeautifulSoup(response.text, "html.parser")

    # look at every <a> element on the page
    for anchor in soup.find_all("a"):

Tip: pass an explicit parser such as "html.parser" or "lxml". The Beautiful Soup documentation is direct about why: "If you're planning on distributing your script to other people, or running it on multiple machines, you should specify a parser in the BeautifulSoup constructor."

Not every <a> has an href, so guard against that, rebuild relative links, and queue anything new:

        # grab the link target, defaulting to empty if there is none
        link = anchor.attrs["href"] if "href" in anchor.attrs else ''

        # rebuild relative links into absolute ones
        if link.startswith('/'):
            link = base_url + link
        elif not link.startswith('http'):
            link = path + link

        # queue any genuinely new link
        if link not in new_urls and link not in processed_urls:
            new_urls.append(link)

The complete crawler

from bs4 import BeautifulSoup
import requests
import requests.exceptions
from urllib.parse import urlsplit
from collections import deque
import re

# a queue of URLs still to be crawled
new_urls = deque(['https://www.eff.org/about/contact'])

# URLs we have already processed
processed_urls = set()

# the addresses we have found so far
emails = set()

# work through the queue until it runs dry
while new_urls:

    # take the next URL and mark it as done
    url = new_urls.popleft()
    processed_urls.add(url)

    # break the URL apart so we can resolve relative links later
    parts = urlsplit(url)
    base_url = "{0.scheme}://{0.netloc}".format(parts)
    path = url[:url.rfind('/') + 1] if '/' in parts.path else url

    # download the page, skipping anything that errors out
    print("Processing %s" % url)
    try:
        response = requests.get(url)
    except (requests.exceptions.MissingSchema,
            requests.exceptions.ConnectionError):
        continue

    # find every address on the page and merge it into our set
    new_emails = set(re.findall(
        r"[a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+",
        response.text, re.I))
    emails.update(new_emails)

    # parse the HTML so we can walk its links
    soup = BeautifulSoup(response.text, "html.parser")

    # look at every <a> element on the page
    for anchor in soup.find_all("a"):
        link = anchor.attrs["href"] if "href" in anchor.attrs else ''
        if link.startswith('/'):
            link = base_url + link
        elif not link.startswith('http'):
            link = path + link
        if link not in new_urls and link not in processed_urls:
            new_urls.append(link)

That runs. It is also the version most tutorials stop at, this article's earlier edition included, and stopping there is where the trouble starts.

Running it against a three-page site

Pointing an unfinished crawler at somebody else's server to see what happens is exactly the behaviour this article is arguing against. So we built a three-page site and served it from python3 -m http.server on localhost: an index, a contact page and a team page, cross-linked, with five real addresses between them. One address sits behind a Cloudflare-style obfuscated link, one is written support [at] acme [dot] example, and the markup also carries a retina image called logo@2x.png and a Sentry DSN in a <script> tag. All of that is ordinary markup you will meet on any real site.

The script above, pointed at the root, reported six addresses:

a1b2c3d4e5@o447951.ingest.sentry.io
dana.reid@acme.example
hello@acme.example
legal@acme.example
logo@2x.png
sales@acme.example

Four are real. logo@2x.png is an image file. The Sentry string is a client-side error-reporting key. Two genuine addresses are missing entirely: press@acme.example behind the obfuscated link, and the [at]-spelled one. Six real addresses in the fixture, four of them found, two of the six results junk. On a site you can read in ten seconds.

The server log is worse than the output.

     12 requests total
      6 x 200
      6 x 404
      3 GET /mailto:hello@acme.example
      2 GET /mailto:sales@acme.example?subject=Hi
      2 GET /index.html
      2 GET /contact.html
      1 GET /team/
      1 GET /cdn-cgi/l/email-protection
      1 GET /

Twelve HTTP requests for three pages, and half of them are 404s.

The mailto: handling is the bug that produces most of that. A mailto: href does not start with / and does not start with http, so the elif branch treats it as a relative path and glues it onto the current directory. mailto:hello@acme.example becomes https://example.com/mailto:hello@acme.example and gets queued as a page to download. Every contact link on the site turns into a 404 aimed at the server you were trying to be polite to. The addresses are still found, by accident, because they are also in the visible text.

The duplicate GET /index.html and GET /contact.html come from the same string-concatenation logic. A link written ../index.html becomes https://example.com/team/../index.html, which the HTTP layer normalises to /index.html on the wire but which is a different string in processed_urls. The set never sees the collision.

Four separate defects, one root cause: URLs are being manipulated as strings.

What the regex sees and what it misses

The pattern [a-z0-9\.\-+_]+@[a-z0-9\.\-+_]+\.[a-z]+ is close enough for a demo and wrong in four specific directions.

It has no idea what a top-level domain is. Anything followed by a dot and letters qualifies, so logo@2x.png matches, and so does every @1x, @3x and @2x.webp in a modern asset pipeline. The cheap fix is a blocklist of asset extensions. The better fix is requiring at least two labels and a plausible TLD length.

It has no length bound. Our companion article on email validation regexes works through the standards in detail, including why the real ceiling is 254 characters rather than the 320 that Django and half the internet still quote, and why the pattern circulating as the "RFC 5322 Official Standard" rejects addresses that the grammar allows. If precision matters, start there rather than hand-tuning this one.

It cannot see what the browser sees. Cloudflare's email address obfuscation is enabled automatically on signup, and it rewrites every visible address into a link to /cdn-cgi/l/email-protection#<hex> with a small deferred script that restores it. No address exists in the HTML at all. The encoding is not a secret. As Jesse Li's 2018 analysis sets out, the first byte of the hex blob is a random XOR key and every following byte is the plaintext XORed against it:

def cf_decode(token):
    """Decode a Cloudflare /cdn-cgi/l/email-protection#<hex> token."""
    raw = bytes.fromhex(token)
    key = raw[0]
    return bytes(b ^ key for b in raw[1:]).decode("utf-8", "replace")

cf_decode("49393b2c3a3a09282a242c672c31282439252c")
# 'press@acme.example'

Eleven lines to undo it, which tells you how much protection it actually buys. It stops the naive regex and nothing else.

It cannot read English. support [at] acme [dot] example is invisible to any address pattern, and so is an address rendered as a PNG, assembled by JavaScript at runtime, or split across two spans by a CSS trick. Some of these are decodable with more work. None of them are decodable with a regex, and each new trick is a separate special case you maintain forever.

Two conclusions, and they point in opposite directions. mailto: links are the highest-precision source on any page and the original script actively destroys them. Free text is the lowest-precision source and it is the only one the original script uses.

The crawler that survives a real site

This version resolves URLs with urljoin instead of string concatenation, reads mailto: targets directly, decodes Cloudflare tokens, stays on one host, obeys robots.txt including Crawl-delay, sets a timeout, refuses to download anything that is not HTML, and caps how much of a page it will read.

import re
import sys
import time
from collections import deque
from urllib.parse import urljoin, urldefrag, urlsplit, unquote
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

START = sys.argv[1]
MAX_PAGES = 200
MAX_BYTES = 2_000_000
UA = "AcmeContactBot/1.0 (+https://example.com/bot; ops@example.com)"

ADDRESS = re.compile(
    r"(?<![A-Za-z0-9._%+-])"
    r"[A-Za-z0-9._%+-]{1,64}"
    r"@"
    r"(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\.)+"
    r"[A-Za-z]{2,24}"
    r"(?![A-Za-z0-9.-])"
)
ASSET_TLDS = {"png", "jpg", "jpeg", "gif", "svg", "webp", "avif", "css",
              "js", "mjs", "map", "json", "woff", "woff2", "ttf", "ico"}
SKIP_HOSTS = ("sentry.io", "ingest.sentry.io")


def plausible(addr):
    """Reject the shapes that match the pattern but are not addresses."""
    if len(addr) > 254:                      # RFC 5321 path limit
        return False
    local, _, domain = addr.rpartition("@")
    if len(local.encode()) > 64:
        return False
    if domain.rsplit(".", 1)[-1].lower() in ASSET_TLDS:
        return False
    return not domain.lower().endswith(SKIP_HOSTS)


def cf_decode(token):
    raw = bytes.fromhex(token)
    key = raw[0]
    return bytes(b ^ key for b in raw[1:]).decode("utf-8", "replace")


def harvest(soup, html):
    """mailto: first, then obfuscated links, then free text."""
    found = set()
    for a in soup.select('a[href^="mailto:"]'):
        target = unquote(a["href"][7:].split("?", 1)[0])
        found.update(p.strip() for p in target.split(",") if "@" in p)
    for node in soup.select("[data-cfemail]"):
        found.add(cf_decode(node["data-cfemail"]))
    for a in soup.select('a[href*="/cdn-cgi/l/email-protection#"]'):
        found.add(cf_decode(a["href"].split("#", 1)[1]))
    found.update(ADDRESS.findall(html))
    return {e.lower() for e in found if plausible(e)}


session = requests.Session()
session.headers["User-Agent"] = UA

host = urlsplit(START).netloc
robots = RobotFileParser()
robots.set_url("{0.scheme}://{0.netloc}/robots.txt".format(urlsplit(START)))
try:
    robots.read()
except Exception:
    robots.disallow_all = True          # unreachable robots.txt means stop
delay = robots.crawl_delay(UA) or 1.0

frontier = deque([START])
queued = {START}                        # membership lives in the set
seen = set()
emails = set()

while frontier and len(seen) < MAX_PAGES:
    url = frontier.popleft()
    if url in seen:
        continue
    seen.add(url)
    if not robots.can_fetch(UA, url):
        continue

    try:
        r = session.get(url, timeout=(5, 15), stream=True)
        if "html" not in r.headers.get("Content-Type", "").split(";")[0]:
            r.close()
            continue
        body = r.raw.read(MAX_BYTES, decode_content=True)
        r.close()
    except requests.exceptions.RequestException as exc:
        print("skip %s: %s" % (url, type(exc).__name__), file=sys.stderr)
        continue
    finally:
        time.sleep(delay)

    if r.status_code >= 400:
        continue

    html = body.decode(r.encoding or r.apparent_encoding or "utf-8", "replace")
    soup = BeautifulSoup(html, "html.parser")
    emails |= harvest(soup, html)

    for a in soup.find_all("a", href=True):
        href = a["href"].strip()
        if href.lower().startswith(
                ("mailto:", "tel:", "javascript:", "#", "data:")):
            continue
        link, _ = urldefrag(urljoin(url, href))
        if "/cdn-cgi/" in link or urlsplit(link).netloc != host:
            continue
        if link not in queued:
            queued.add(link)
            frontier.append(link)

for address in sorted(emails):
    print(address)

Same fixture, same seed. Five HTTP requests including robots.txt, zero 404s, and five addresses, every one of them real:

dana.reid@acme.example
hello@acme.example
legal@acme.example
press@acme.example
sales@acme.example

Twelve requests down to five, 33% junk down to zero, and the Cloudflare-protected address recovered. One address is still missing: support [at] acme [dot] example, because nothing here reads English. Writing the CSV is four lines on top:

import csv

with open("emails.csv", "w", newline="") as f:
    writer = csv.writer(f)
    for address in sorted(emails):
        writer.writerow([address])

Two details in that code deserve their own sentence. The except clause catches requests.exceptions.RequestException, the base class, rather than the two specific errors the original caught. That matters more than it looks: on requests 2.34.2, ConnectTimeout is a subclass of ConnectionError and is caught by the original, but ReadTimeout, TooManyRedirects, InvalidURL and ChunkedEncodingError are not. A server that accepts your connection and then stops talking crashes the original crawler outright, after the default timeout of never.

And stream=True with an explicit raw.read(MAX_BYTES) is what keeps a 400 MB video, a firmware image or a database dump from being pulled into memory in one gulp because somebody linked to it from a footer.

What breaks at scale

A one-off run against a small site forgives everything. Ten thousand pages overnight does not.

The queue is quadratic and it does not announce itself. if link not in new_urls scans the entire deque. Every new URL costs a walk over every URL already queued, so the total work grows with the square of the frontier. Measured on a single machine, Python 3.11.15 on x86_64, timing only the membership-and-append loop with no network at all:

Your absolute numbers will differ; the shape will not. At a hundred thousand queued URLs the original crawler spends over a minute of pure CPU deciding whether it has seen things, and that minute is invisible because it is spread across the run in microsecond slices. Keeping a mirror set alongside the deque costs one line and removes the curve entirely.

response.text will corrupt your data silently. requests follows the old HTTP rule that a text/* response with no charset parameter is Latin-1. Plenty of servers still send bare Content-Type: text/html. Against a UTF-8 page served that way:

r.headers['Content-Type'] : text/html
r.encoding                : ISO-8859-1
r.apparent_encoding       : utf-8
r.text                    : Contact Björn: bjoern@acme.example
r.content.decode('utf-8') : Contact Björn: bjoern@acme.example

The address survived here because it is ASCII. The name next to it did not, and an internationalised address under RFC 6531 would not either. Decode from r.content with apparent_encoding as the fallback, as the corrected version does.

Nothing bounds the crawl. Without a host check, one link to a news site sends the crawler off to index the web. The corrected version compares urlsplit(link).netloc against the seed's host and drops everything else. If the site publishes a sitemap, crawling the sitemap instead of following links is both faster and far less traffic.

One request at a time is the real wall. At EFF's requested rate a serial crawler manages 120 pages an hour. Concurrency is the answer, and it is a different program: asynchronous scraping in Python covers the asyncio shape, where the per-host rate limit becomes a semaphore rather than a sleep. Concurrency is also where a hobby crawler turns into infrastructure with retries, backoff, proxy rotation and a persistent frontier, and where handing the extraction to a managed extraction service starts to be the cheaper answer.

Politeness you can measure

Politeness is usually described in adjectives. It has numbers.

robots.txt was standardised as RFC 9309 in September 2022, on the Standards Track. Two of its rules are the ones people get backwards. A 4xx on robots.txt means the file is unavailable and "the crawler MAY access any resources on the server". A 5xx means it is unreachable, and then the crawler "MUST assume complete disallow". Most hand-rolled crawlers treat both as permission. The corrected version above sets disallow_all when the fetch raises, which is the conservative reading.

Python's RobotFileParser also exposes crawl_delay(useragent) and request_rate(useragent), both added in Python 3.6, and almost nobody calls them. EFF's own robots.txt carries Crawl-delay: 30 under User-agent: *, and 60 for CCBot. The advice in the earlier version of this article was time.sleep(1). Against that site it was thirty times too fast, and the file that says so is one request away:

rp = RobotFileParser()
rp.set_url("https://www.eff.org/robots.txt")
rp.read()
print(rp.crawl_delay("MyEmailCrawler/1.0"))   # 30
print(rp.can_fetch("MyEmailCrawler/1.0", "https://www.eff.org/about/contact"))

Send a User-Agent that names you and gives a way to reach you. A site owner who can identify your crawler will usually email you before blocking a whole subnet.

Thirty seconds a page is 120 pages an hour. Plan the crawl around that, or crawl something else.

Where this approach stops working

The address is not in the HTML. Contact details behind a form, a login, or a React component that fetches JSON after paint are invisible to requests. You need a browser, and a browser costs roughly two orders of magnitude more per page.

The site fights back. Bot management, rate limiting and IP reputation stop a naive crawler long before the volume gets interesting. That is a different discipline from parsing HTML.

The address is real and useless. info@, sales@, noreply@ and role accounts dominate what a crawler collects. Reaching a named person usually means a source other than the public website.

The address is a trap. Project Honey Pot's whole method is publishing addresses that exist only to be harvested, so that any mail arriving at them proves harvesting. Its 2009 billionth-message report tracked the average gap between an address being taken and its first spam falling from 49 days in 2004 to roughly 21 days by 2009. That report is old, and no comparable public measurement has replaced it, but the mechanism is unchanged: hit one honeypot and your sending domain and IP inherit the reputation.

Delivery has moved, and this is the change most email-harvesting guides have not caught up with. Since 1 February 2024 Google requires bulk senders above 5,000 messages a day to Gmail to authenticate with SPF, DKIM and DMARC, to offer one-click unsubscribe per RFC 8058, and to keep the Postmaster Tools spam rate below 0.30%. Microsoft followed on 5 May 2025 for Outlook.com at the same 5,000-a-day threshold, and a late revision dropped the planned junk-folder stage in favour of outright rejection: 550; 5.7.515 Access denied, sending domain [SendingDomain] does not meet the required authentication level.

Do the arithmetic on that 0.30% before you write the crawler. Three complaints in a thousand is the ceiling. A cold list scraped off contact pages will clear that on the first send, and the penalty falls on the domain, not the campaign, so your invoices and password resets go with it. The technical ceiling on this kind of outreach arrived before the legal one did.

A word on responsible use

A crawler is a tool, and the same code that powers legitimate contact research can also fuel spam. Before you point this at a real site, be on solid ground: respect each site's robots.txt and terms of service, crawl gently so you do not overload anyone's server, and remember that data protection rules such as the GDPR and the CAN-SPAM Act govern how harvested addresses may be stored and used. Scraping a public page is one thing; blasting unsolicited mail to everyone you find is another.

Practically, that means three habits. Keep the source URL and the timestamp next to every address you store, because Article 14(2)(f) requires you to tell people where their data came from and you cannot reconstruct it later. Put your Article 14 notice in the first message rather than a later one. Delete on request, immediately and provably, and log the deletion.

Wrapping up

In well under two hundred lines you have a crawler that queues URLs, fetches with Requests, parses with Beautiful Soup, reads mailto: links before it reads free text, decodes the obfuscation that hides the rest, and stops when robots.txt tells it to. The same walk-and-extract skeleton is the basis of most scraping work, and the Java version of this crawler makes the same shape visible in another language.

The gap between the naive version and the corrected one was twelve requests versus five, two junk hits versus none, and one silently mangled encoding. The gap between the corrected one and something you should actually run against a stranger's website is a legal question, and the honest answer for most outreach use cases in the EU, the UK and Canada is that scraping is the easy part and the compliance is not.