Python Web Scraping: The Complete Guide

Parsing one 2.4 MB HTML page with BeautifulSoup's default parser costs 266 ms of CPU on our test machine. The same page, the same extraction, through lxml costs 25 ms. Across ten thousand pages that is forty-four minutes against four. Neither script looks different from the other in a code review, and neither one throws an error.

Most of what goes wrong in a Python scraper is like that. It does not crash. It returns a page you did not want, decoded with a codec nobody chose, from a queue that quietly re-fetched the same URL under five different spellings. The parts that fail loudly are the easy parts.

This guide walks the whole path, from the first HTTP request to a crawl that survives restarts, and it links out to focused articles for the narrow topics. Every version number, price and default value below was read from PyPI's JSON API, from each project's own source tree, or from the vendor's own pricing page on 13 August 2026. Where we measured something ourselves, the method is described so you can run it again.

The four stages, and the one that eats your afternoon

Every scraper, at any scale, is the same four moves:

1. Fetch. An HTTP request goes out, a response comes back: HTML, XML, JSON, or a block page dressed as one of those.
2. Parse. Data comes out of the markup with CSS selectors, XPath, or regular expressions.
3. Normalize and store. Values get coerced into one shape, deduplicated, and written somewhere.
4. Manage the crawl. URLs go into a queue, come out once, and get retried when the network disagrees.

A one-off script folds all four into ten lines, and you should write that version first. If the site is small and static you may never need anything else; the basic principles of web scraping cover the shape of the job before any library gets involved.

Stage four decides whether the project is pleasant or miserable. Fetching and parsing are solved problems with good libraries. Crawl management is where you invent your own bugs: URLs that look different and are the same, retries that hammer a server already telling you to stop, a queue that lives in RAM and dies with the process at 4 a.m.

The stack, with versions and dates

Every version below was pulled from the PyPI JSON API on 13 August 2026. Release dates are the upload timestamp of the current version.

HTTP clients

Two things a feature table hides. requests is not stagnant: 2.34.0 shipped inline type hints in May 2026, replacing the typeshed stubs, and added support for Python 3.14t and Python 3.15 betas. And httpx has been on 0.28.1 since December 2024, with three 1.0 development builds during 2025 and nothing since September of that year. The library works and is widely deployed. Its release cadence is not what "modern replacement for requests" implies.

HTML and XML parsers

Read the registry, not the project page. lxml.de headlines 7.0.0a1 from 29 May 2026, an alpha, while stable is 6.1.1. The Beautiful Soup home page still lists 4.14.3 from November 2025 while PyPI serves 4.15.0.

Browser automation

Drop Pyppeteer from your shortlist. Its own README opens by calling the repository unmaintained and points readers at playwright-python, and its last release is two and a half years old. The Selenium WebDriver tips still transfer across languages, and Selenium itself is in no danger: 4.47.0 landed three days before this article was rechecked. One mismatch worth an afternoon of your time: the Playwright for Python intro page says Python 3.8 or higher, while the 1.62.0 package metadata declares requires-python >=3.10. Trust the metadata.

Frameworks

Scrapy 2.17.0 shipped 7 July 2026 and calls itself "the world's most-used open source data extraction framework". It is the only option here that gives you queueing, deduplication, retries, throttling and crawl resumption without you writing them. scrapy-playwright 0.0.48 (10 July 2026) bolts a browser onto it for JavaScript-rendered pages. Full walkthrough in the Scrapy tutorial.

Scrapy 2.16.0 removed start_requests(). This is the breaking change to know if you are returning to a Scrapy project after a year away. The method was deprecated in 2.13.0 (8 May 2025) in favour of the async start(), and removed on 19 May 2026, at which point it stopped being called at all. Nothing raises. Scrapy falls back to the default start(), which yields one plain request per entry in start_urls, so whatever lived in your start_requests() (custom headers, a login POST, seeded pagination) silently stops running. With start_urls empty, the crawl finishes in a second having scraped nothing. Support for process_start_requests() in spider middlewares went the same way, along with synchronous process_spider_output() methods. The same release added official Python 3.14 support, and 2.17.0 went on to deprecate scrapy.FormRequest.

A wider survey with the trade-offs spelled out per library lives in the best Python web scraping libraries.

Choosing without a decision tree

• Static page, one-off job: requests plus BeautifulSoup. Write this first, always.
• The same job at ten thousand pages: requests plus lxml, or httpx/aiohttp plus selectolax.
• A whole site, with resume, dedup and retries: Scrapy. The framework is not overhead here, it is the part you would otherwise write badly.
• Content that only exists after JavaScript runs: Playwright, or Selenium if your team already knows it.
• Tens of thousands of concurrent connections: an async approach.
• A site that returns 403 to Python and 200 to Chrome: curl_cffi, before you reach for a browser.

Fetching the page

A request that will not embarrass you:

import requests

url = "https://example.com"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/140.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

response = requests.get(url, headers=headers, timeout=(5, 20))
response.raise_for_status()   # raises on 4xx/5xx
html = response.text

The timeout is a tuple for a reason. timeout=10 is not a deadline for the request. It is a per-socket-operation timeout, applied separately to the connection handshake and to each read. A server that dribbles one byte every nine seconds keeps that request alive forever. The (connect, read) form at least makes the distinction visible. For a real wall-clock ceiling, enforce it outside the call.

There is no retry by default. requests mounts an HTTPAdapter with max_retries=0, so one connection reset ends the request. Wiring retries up takes four lines and is covered further down.

The default connection pool holds ten sockets per host. HTTPAdapter defaults to pool_connections=10 and pool_maxsize=10. Run twenty threads against one domain through a shared Session and urllib3 logs "Connection pool is full, discarding connection" while quietly opening and tearing down sockets outside the pool. Nothing fails. You pay for a fresh TLS handshake on half your requests and never find out unless you turn urllib3's logging on.

If the page is rendered by JavaScript, requests gets you the empty shell and you need a browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="domcontentloaded")
    page.wait_for_selector(".content")   # wait for the data, not for the page
    html = page.content()
    browser.close()

Before you spend a browser on it, check whether the data arrives as JSON from an endpoint the page itself calls. It usually does, it is usually cleaner than the HTML, and it is usually stable across redesigns. That path is covered in API scraping, and the Python side of it in parsing JSON in Python.

Where requests stops working, and why it is not your headers

A site returns 200 in your browser and 403 to your script. You copy the browser's User-Agent, then every header the browser sends, then the cookies. Still 403. This is the point where most Python scraping tutorials, this article's earlier version included, tell you to add more headers and move on.

Headers are not the signal. Before a byte of HTTP is sent, your client completes a TLS handshake, and its contents and ordering differ between OpenSSL as Python builds it and BoringSSL as Chrome ships it. The cipher list, extension set, supported groups and ALPN offer form a fingerprint. Above it, the HTTP/2 SETTINGS frame, window-update increment and pseudo-header order form a second one. Anti-bot products read both. A requests call announcing itself as Chrome 140 over a Python TLS stack is a contradiction that costs nothing to detect.

curl_cffi is the practical answer in Python. It binds a fork of curl-impersonate and, in its README's words, "can impersonate browsers' TLS/JA3 and HTTP/2 fingerprints" while keeping an API that "mimics the requests API, no need to learn another one". Version 0.16.0 went up on 1 August 2026.

from curl_cffi import requests as cffi_requests

r = cffi_requests.get("https://example.com", impersonate="chrome")
print(r.status_code, len(r.text))

One import clears a large class of 403s at a fraction of a headless browser's cost. It does not defeat behavioural scoring, JavaScript challenges, or a bot-management platform running a per-customer model; those have their own article on how sites protect themselves and how it gets bypassed. Persistent challenge walls may need CAPTCHA solving, a per-solve cost worth pricing before you commit to a target.

Parsing, and what the speed claims actually mean

The advice everywhere is: pass "lxml" to BeautifulSoup instead of "html.parser" and get an order of magnitude. Half of that is true, and it is the less useful half.

We measured it. Page: https://pypi.org/simple/lxml/, fetched 13 August 2026, 2,364,601 bytes containing 4,700 anchors. Task: return every href. All five approaches return the same 4,700 strings. Timings are CPU time via time.process_time(), minimum of 21 runs, on Python 3.11.15 with beautifulsoup4 4.15.0, lxml 6.1.1 (libxml2 2.14.6), parsel 1.11.0 and selectolax 0.4.11.

Measured on a single shared machine under variable load, which is why these are CPU times rather than wall clock. The same run on a 75 KB page gave the same ordering and near-identical ratios: 1.0x, 1.4x, 5.6x, 11.1x, 12.9x. Your absolute numbers will differ. The shape will not.

Switching BeautifulSoup's parser to lxml buys 1.4x, not 10x. The 10x comes from not using BeautifulSoup. What lxml speeds up is the tokenizing, and tokenizing is not where BeautifulSoup spends its time. It spends it building a Python object for every node, and that cost is the same whichever tokenizer fed it. If your profile says parsing is the problem, changing the features argument is not the fix. Dropping to lxml.html or selectolax is.

That does not make BeautifulSoup the wrong choice. It is the friendliest API in the ecosystem, it forgives markup that would make a strict parser give up, and on the 75 KB pages most people actually scrape the whole question is 10 ms against 1 ms. There is a walkthrough on scraping tables with BeautifulSoup.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")

title = soup.find("h1").get_text(strip=True)
links = [a["href"] for a in soup.select("a.product-link")]
price = soup.select_one(".price").text

lxml is the same job with a different bill. Full XPath, C-speed tokenizing, and an API that assumes you know what a tree is:

from lxml import html as lxml_html

tree = lxml_html.fromstring(html)
titles = tree.xpath('//h2[@class="title"]/text()')
prices = tree.xpath('//span[@class="price"]/text()')

The deep dive is in Python scraping with lxml. One trap it is worth naming here: lxml enforces a 10,000,000-byte limit on a single text node and, on hitting it, may stop processing the rest of the document without raising. BeautifulSoup 4.14.3 added a huge_tree=True passthrough for exactly this. If you scrape pages carrying enormous inline JSON blobs, this is the bug you will otherwise spend a day on.

parsel gives you one selector object that speaks both CSS and XPath, and it is what Scrapy responses are made of:

from parsel import Selector

sel = Selector(text=html)
sel.css("h1::text").get()
sel.xpath("//a/@href").getall()

selectolax is the fastest thing on the list and the most limited. It wraps two C engines, and the choice between them is already made for you: the README names Lexbor the preferred backend as of 2024 and says "the underlying C library" behind the older Modest backend "is not maintained anymore". Our numbers agree. On the same benchmark Lexbor came in at 22 ms against Modest's 36 ms, which makes the legacy backend slower than plain lxml. Import selectolax.lexbor, not selectolax.parser, and accept the trade: CSS selectors only, no XPath.

Regular expressions get one honest paragraph. On the same benchmark, a single re.findall pulled all 4,700 hrefs in under 5 ms, four times faster than the fastest real parser. That is the entire case for regex, and it holds only while the markup stays as flat as you assumed. Use re for a phone number, a SKU, a value buried in a <script> block. Do not use it to walk nested structure. The first time a class attribute picks up a second value, the pattern keeps matching and starts returning the wrong thing.

Character encodings, and why your fix never fires

Garbled text is the classic Python scraping bug: café instead of café, or a Cyrillic page rendered as accented Latin gibberish. Almost every guide, this one included until now, explains it as "requests guessed the encoding wrong". That is not what happens, and the difference is why the standard fix often does nothing. The actual rule, read from requests 2.34.2's get_encoding_from_headers():

Any text/* type with no charset parameter gets ISO-8859-1, per the letter of the old HTTP spec. ISO-8859-1 is not None. And response.text only falls back to apparent_encoding when response.encoding is None. So for an HTML page served without a charset, the byte-sniffing detector that everybody recommends is never consulted. It fires for application/xml, which is not where your problem is.

response = requests.get(url)
response.encoding = response.apparent_encoding   # no-op on text/html
html = response.text

That is the fix half the internet prints, and on the exact case it is meant to solve it changes nothing.

What to do instead: hand the bytes to the parser.

from bs4 import BeautifulSoup

response = requests.get(url)
soup = BeautifulSoup(response.content, "lxml")   # .content, not .text

response.content is raw bytes. Both lxml and BeautifulSoup read the <meta charset> declaration inside the document and decode accordingly, which is what a browser does. If you must go through requests, clear the header guess first so the detector can run:

response.encoding = None
response.encoding = response.apparent_encoding   # now it actually runs

apparent_encoding runs charset_normalizer by default, a hard dependency of requests since 2.x. If the optional chardet extra is present, chardet wins: requests resolves the two in that order, so installing an unrelated package that pulls in chardet changes how your scraper decodes text.

Text you already mangled is repairable. If UTF-8 bytes were decoded as Latin-1, re-encode and decode again:

fixed = broken.encode("iso-8859-1").decode("utf-8")

One footnote that only bites on legacy sites. Python's iso-8859-1 codec is the real ISO standard. The web is not. The WHATWG Encoding Standard records that browsers map the latin1 and ascii labels to windows-1252, "which is a superset of both, and this choice was codified in this standard". The two disagree across bytes 0x80 to 0x9F. Decode 0x93 as iso-8859-1 in Python and you get an invisible control character; a browser gives you a left double quotation mark. On a Windows-authored page full of smart quotes, that is the difference between text and confetti. Guess with cp1252, not iso-8859-1.

XML plays by stricter rules, since the declaration inside the document wins over the transport header. The lxml guide covers what that means for non-UTF-8 documents.

Threads, processes and the free-threaded build

Fetching is I/O-bound, so threads help. While one thread waits on a socket, another runs. That much of the standard advice survives.

from concurrent.futures import ThreadPoolExecutor
import requests

urls = [f"https://example.com/page/{i}" for i in range(1, 101)]
session = requests.Session()
session.mount("https://", requests.adapters.HTTPAdapter(pool_maxsize=20))

def fetch(url):
    r = session.get(url, timeout=(5, 20))
    return url, r.status_code

with ThreadPoolExecutor(max_workers=10) as executor:
    for url, status in executor.map(fetch, urls):
        print(url, status)

The pool_maxsize line matters and is missing from every version of this snippet in circulation. Ten workers against the default pool of ten is already at the edge; twenty is throwing away connections.

Parsing is a different question, and the answer depends on the parser. A C extension only lets other threads run if it releases the GIL, and you can check whether yours does by reading its source. lxml wraps the libxml2 parse call in a with nogil block in parser.pxi, and its own FAQ states that lxml "frees the GIL ... internally when parsing from disk and memory", with the caveat that "if the majority of your processing is done in Python code (walking trees, modifying elements, etc.), your gain will be close to zero". selectolax does the same around Lexbor in lexbor.pyx. BeautifulSoup builds its tree in pure Python and holds the GIL for the whole of it.

So a thread pool that fetches with requests and parses with lxml can use more than one core. The same pool parsing with BeautifulSoup cannot, no matter how many workers you give it. That is the reason multiprocessing still exists in scraping code: it is not a general speedup, it is the escape hatch for pure-Python parsing.

The GIL is optional now, and the ecosystem mostly caught up

Python 3.13 shipped the free-threaded build as an experiment under PEP 703. That phase is over. PEP 779, Criteria for supported status for free-threaded Python, is Final and targets Python 3.14, which was released on 7 October 2025. Free-threaded CPython is now an officially supported build, not an experiment. The PEP puts the current cost at roughly 10% CPU overhead on Linux and Windows, around 3% on macOS, and 15 to 20% higher memory use.

Whether you can use it is decided by your dependencies, not by CPython. A free-threaded interpreter needs wheels built against the cp314t ABI, and there is no fallback: a package without them either builds from source or does not install. We checked the scraping stack against PyPI on 13 August 2026 by listing the wheel filenames on each project's current release.

Shipping free-threaded wheels: lxml, selectolax, aiohttp, pycurl, curl_cffi, charset-normalizer, regex, yarl, multidict, frozenlist, cryptography, numpy, pandas, Pillow. requests, Scrapy, Twisted, parsel, redis and protego are pure Python and need nothing.

Not shipping them: orjson, zstandard, brotli, psycopg2-binary. Fetch and parse are ready; serialization, compression and the Postgres driver are the holes, which is a strange place for them to be at the far end of a pipeline. Run the same check on your own requirements.txt, because the list moves monthly.

Temper the expectation for scraping specifically. Free-threading raises a ceiling on CPU-bound parsing, and your bottleneck is almost certainly the network and the target's tolerance. For thousands of concurrent connections the answer is still async, which holds tens of thousands of sockets in one thread with none of this complexity. See async web scraping in Python.

A word of caution: high concurrency is not a license to hammer someone else's server. Rate-limit yourself and respect robots.txt.

Reading the response properly

Parsing an error page as if it were data is the most common silent failure in production scrapers, because a 403 block page is valid HTML and your selectors just return empty lists.

response = requests.get(url)

print(response.status_code)      # 200, 404, 403, 500 ...
print(response.reason)           # 'OK', 'Not Found'
print(response.headers["Content-Type"])
print(response.url)              # final URL after redirects
print(response.elapsed)          # time to headers, not to last byte

response.elapsed gets logged as "response time" everywhere and is not one. It measures from sending the request to finishing the headers. With stream=True the body may still be arriving long afterwards.

Retry-After is not always a number

A bug this article shipped in its previous version, and that you will find in a hundred others:

wait = int(response.headers.get("Retry-After", 60))   # raises ValueError

RFC 9110 section 10.2.3 defines the field as Retry-After = HTTP-date / delay-seconds. A server is entirely within spec to answer Retry-After: Wed, 13 Aug 2026 14:00:00 GMT. int() on that raises ValueError, inside the handler you wrote for the case where the site is already angry at you, at the worst possible moment in the crawl.

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(response, default=60):
    raw = response.headers.get("Retry-After")
    if not raw:
        return default
    try:
        return int(raw)
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(raw)
        return max(0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return default

Let urllib3 do the retrying

Hand-rolled retry loops end up hammering. urllib3's Retry already understands backoff and Retry-After:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1,              # sleeps 1, 2, 4, 8, 16 seconds
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods={"GET", "HEAD", "POST"},
    respect_retry_after_header=True,
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=20))

Three defaults in that class are worth knowing before you rely on it. backoff_factor is 0 by default, so an unconfigured Retry retries immediately with no pause at all. allowed_methods excludes POST, on the correct assumption that it is not idempotent, so POST requests are not retried unless you say so. And urllib3 honours Retry-After only on statuses 413, 429 and 503, capping the value it will obey at six hours.

Statuses 403 and 503 usually mean an anti-bot layer rather than a broken server. Retrying the same request from the same address is the least useful reaction available. Rotate the exit address, change the client fingerprint, and slow down.

robots.txt, and the standard-library parser that under-blocks

The polite default for robots.txt in Python is urllib.robotparser, which ships with the interpreter. Its documentation points readers at RFC 9309 for the file format. The parser does not implement it.

RFC 9309 section 2.2.3 defines two special characters in rule paths: * designates "0 or more instances of any character" and $ designates "the end of the match pattern". Both appear in ordinary robots.txt files. Neither is handled by the standard library, which compares paths as literal prefixes.

We ran both parsers against the same file on 13 August 2026:

User-agent: *
Disallow: /*?sort=
Disallow: /private$
Allow: /public/

The stdlib parser waves through two paths the site explicitly disallowed. It fails open, silently, on exactly the syntax sites use to keep crawlers out of faceted search, which is the traffic pattern that gets scrapers banned in the first place. If your compliance story is "we use robotparser", it has a hole in it.

Use protego instead. It is the parser Scrapy uses, it handles wildcards, $, Crawl-delay and Sitemap, and 0.6.2 shipped 25 June 2026.

import requests
from protego import Protego

robots = Protego.parse(requests.get("https://example.com/robots.txt").text)
if robots.can_fetch("https://example.com/catalog?sort=price", "MyBot"):
    ...
delay = robots.crawl_delay("MyBot")

Two more clauses production crawlers get wrong. RFC 9309 section 2.4 says a cached robots.txt should not be used for more than 24 hours, so refetch once per host per day rather than once per process. Section 2.5 requires crawlers to parse at least 500 kibibytes, and large sites do ship files that big.

Proxies, with prices

Under sustained scraping a site bans your address on volume alone. The answer is a pool and rotation.

proxies = {
    "http":  "http://user:pass@123.45.67.89:8080",
    "https": "http://user:pass@123.45.67.89:8080",
}
response = requests.get(url, proxies=proxies, timeout=(5, 20))

import random
import requests

PROXIES = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]

def fetch_with_rotation(url):
    proxy = random.choice(PROXIES)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=(5, 20))

random.choice is fine for three proxies and wrong for three hundred. It will happily pick a dead one repeatedly, and it gives you no way to notice that forty of them now return 403 on this particular target. In production, keep health state per proxy: drop anything that times out or gets blocked, retry it after a cooling period, and track the block rate per target rather than globally, because a proxy burned on one site is usually fine on another.

What the three tiers cost

Datacenter addresses are cheap and easy to flag. Residential ones borrow real subscriber space and cost more. Mobile carrier addresses are the most trusted and the priciest. Prices read from each vendor's own pricing page on 13 August 2026, and note that the first two tiers are billed on different units: traffic for residential, the address itself for datacenter.

Webshare gives ten datacenter proxies free with no card, enough to test rotation logic before you spend anything.

Work the arithmetic before choosing a tier, because per-GB pricing punishes exactly the pages scrapers care about. A product page with images and analytics is 2 to 3 MB on the wire. At $2.50/GB, a hundred thousand of those is roughly $625 in bandwidth alone. The same hundred thousand fetched as JSON from the site's own API might be 40 KB each, or about $10. Request gzip and brotli, block images with a browser route rule, and prefer the JSON endpoint. The bandwidth bill is the scraping bill. A deeper walkthrough of pool design is in our guide to rotating proxies for scraping.

Scraping through Tor

Tor changes your exit address for free. It is slow, it is unsuitable for volume, and large sites know the exit-node list and either block it or challenge it. For low-rate work against targets that do not care, it costs nothing.

After installing Tor, a SOCKS5 proxy listens on 127.0.0.1:9050:

import requests

proxies = {
    "http":  "socks5h://127.0.0.1:9050",
    "https": "socks5h://127.0.0.1:9050",
}
# needs: pip install requests[socks]
r = requests.get("https://check.torproject.org/api/ip", proxies=proxies, timeout=30)
print(r.json())   # {"IsTor": true, "IP": "..."}

The h in socks5h is the whole point: DNS resolution happens at the proxy instead of locally, so your real resolver never sees the hostname. Drop the h and you have built an anonymity tool that leaks every domain you visit to your ISP.

Two corrections to the version of this section we published before. The first: it checked the exit address against the httpbin.org demo service, which answered 503 to every request we made on 13 August 2026. check.torproject.org/api/ip is the right endpoint anyway, because it answers the question you are actually asking, returning {"IsTor":true,"IP":"..."} rather than an address alone. For non-Tor work, api.ipify.org returns {"ip":"..."} and stays up.

The second is worse. The requests[socks] extra pulls in PySocks 1.7.1, released 20 September 2019, with no release in nearly seven years. httpx-socks and aiohttp-socks are both maintained and both shipped on 12 August 2026. None of this makes Tor a substitute for a rotating proxy pool at any volume worth the name.

Getting a new exit node

The documented way to force a new circuit is the NEWNYM signal on the control port, and every guide reaches for the stem library:

from stem import Signal
from stem.control import Controller

def renew_tor_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password="your_password")
        controller.signal(Signal.NEWNYM)

That code is correct and you may not be able to run it. stem 1.8.2 dates from 6 June 2023, PyPI carries a source distribution and no wheel, and its setup.py calls distutils.core.setup() directly. distutils left the standard library in Python 3.12. pip install stem failed on our machine on 13 August 2026 against a current setuptools, before any of your code ran.

Two ways out. Use a maintained controller library, txtorcon 26.6.0 on Twisted or aiostem 0.4.7 on asyncio. Or speak the control protocol yourself, since it is line-oriented text over a socket:

import socket

def renew_tor_ip(password, port=9051):
    with socket.create_connection(("127.0.0.1", port), timeout=10) as s:
        s.sendall(f'AUTHENTICATE "{password}"\r\n'.encode())
        assert s.recv(1024).startswith(b"250")
        s.sendall(b"SIGNAL NEWNYM\r\n")
        assert s.recv(1024).startswith(b"250")

Either way you have to enable the control port in torrc and set a password hash with tor --hash-password. Note also that Tor rate-limits NEWNYM to roughly one new circuit every ten seconds; calling it in a tight loop gets you the same exit node back.

HTTPS and certificates

requests verifies certificates against the certifi bundle, currently 2026.7.22. Most of the time there is nothing to configure. Problems appear on hosts with self-signed or expired certificates, and behind corporate TLS-inspecting proxies.

import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = requests.get(url, verify=False)   # NOT for production

Turning verification off is not a small compromise. It disables authentication of the endpoint entirely, so any device on the path can serve you whatever it wants and you will parse it as data. In a scraper that logs in, it also hands your credentials to whoever sits between you and the host. Fine while debugging, wrong in a scheduled job. For an internal CA or an inspecting proxy, trust that CA explicitly:

response = requests.get(url, verify="/path/to/custom-ca-bundle.crt")

The REQUESTS_CA_BUNDLE environment variable does the same process-wide, which is usually the right answer inside a container. If SSLCertVerificationError starts appearing on ordinary public sites, your bundle is stale rather than the site being wrong:

pip install --upgrade certifi

Cookies and sessions

Cookies carry sessions, authentication, and the token-and-redirect dance that interstitial protection pages use.

import requests

session = requests.Session()

# log in — the server returns a session cookie
session.post("https://example.com/login", data={"user": "u", "pass": "p"})

# subsequent requests are already authenticated
profile = session.get("https://example.com/profile")

Session stores and replays cookies, reuses TCP connections, and shares headers across requests. The connection reuse alone earns it: a fresh TLS handshake per request against an HTTPS host is a round trip you pay for on every page.

cookies = {"sessionid": "abc123", "csrftoken": "xyz789"}
response = requests.get(url, cookies=cookies)

for name, value in response.cookies.items():
    print(name, value)

Scraping behind a login changes your legal position, not just your code. That distinction is at the end of this article and it is the one that matters most.

The URL frontier, and the dedup bug you already have

Crawling a whole site means tracking what you have not visited and what you have. That structure is the frontier.

from collections import deque

to_visit = deque(["https://example.com"])
visited = set()

while to_visit:
    url = to_visit.popleft()
    if url in visited:
        continue
    visited.add(url)
    # ... download, parse, add new links to to_visit

This is correct and it is also where a slow leak starts. A set of URL strings deduplicates strings, and a site does not serve strings. Consider five links a crawler will genuinely find on real pages:

http://ex.com/q?id=111&cat=222
http://ex.com/q?cat=222&id=111
http://ex.com/q?id=111&cat=222#frag
http://EX.com/q?id=111&cat=222
http://ex.com/q?id=111&cat=222&utm_source=x

A set() keeps all five and fetches five pages. Run the same list through w3lib.url.canonicalize_url, which is what Scrapy hashes for its duplicate filter, and it collapses to two: query parameters get sorted, the fragment is dropped, the host is lower-cased. The fifth survives, because utm_source is a real query parameter and no general rule can know it does not change the response. That one you strip by hand, per site.

from w3lib.url import canonicalize_url

key = canonicalize_url(url)
if key not in seen:
    seen.add(key)
    to_visit.append(url)

Four-fifths waste on a five-URL example is contrived. A five-percent duplicate rate on a two-hundred-thousand-page crawl is not, and it is ten thousand pages of bandwidth you paid per gigabyte for.

When the frontier outgrows RAM

• Redis. A shared queue several workers pop from, surviving restarts. Lists for the queue, sets for the seen-keys.
• PostgreSQL or SQLite. A urls table with a status column: new / in_progress / done / failed. Slower per operation, and it lets you answer "how far in are we" with a query instead of a guess.
• A Bloom filter for the seen-set past a few million URLs. Fixed bits per key, tunable false-positive rate, and a false positive means you silently skip a page. rbloom (1.5.4, September 2025) is a maintained Rust-backed option; pybloom-live still works but last shipped in October 2022.

Scrapy has all of this built, which is most of the argument for using it. Its shipped defaults, read from scrapy/settings/default_settings.py in 2.17.0:

Read that table twice. Whether Scrapy respects robots.txt depends on whether you ran scrapy startproject or instantiated a crawler yourself, and the two answers are opposites. The generated project is also eight times politer per domain. And JOBDIR is unset in both, so a crawl killed at hour six restarts from nothing unless you set it. Details in the Scrapy tutorial.

What breaks between one page and ten thousand

The script that works is not a small version of the system that works. Five things change, and they change at different scales.

Everything transient becomes certain. At a 0.1% failure rate, one page in a thousand fails. Across a thousand pages you will never see it and will write no retry logic. Across two hundred thousand you get two hundred failures, and the question stops being whether to retry and becomes what to do with the ones that still fail after five attempts. Write those to a dead-letter table with the status code and the body. The pattern in that table is usually the actual finding.

Rate becomes the whole negotiation. One request per second is 86,400 pages a day and invisible in most access logs. Ten concurrent workers with no delay against a small site is a load test nobody ordered, and it produces a ban that outlives the crawl. Scrapy's startproject template picks one request per domain with a one-second delay for a reason. Start there and go up only while the error rate stays flat.

Memory stops being free. A set of two million canonical URLs is a few hundred megabytes of Python objects, and every response you keep just in case is on the heap. Parse, extract, discard the tree, write the row. Do not accumulate Response objects and process them at the end; a long crawl will find the ceiling for you.

Selectors rot measurably. A scraper returning zero rows is indistinguishable from a category that emptied out. Assert on shape rather than on the absence of exceptions: if a product page yields no price, or a listing yields fewer than ten links, raise instead of writing. A crawl that stops loudly on day four beats a table of nulls found in week three.

The bandwidth bill arrives, in dollars per gigabyte, and it is the line item people forget when they price a project by developer hours.

Fixing all five is what a crawl framework is for, which is why the recommendation at scale is Scrapy or a queue-plus-workers architecture rather than a bigger version of the ten-line script. Sites that fight back on top of all this are where a managed extraction service earns its keep, because the maintenance load is continuous and does not scale down.

Where Python stops being the right answer

Python is the default for scraping and the default is right most of the time. Low barrier, readable code, the widest library ecosystem for this task, and a direct path from scraped rows into pandas and a database. The failure modes deserve naming precisely rather than as a bullet list of cons.

Pure-Python processing is the ceiling, and it is lower than people expect. The parser benchmark above is the shape of it: the fast options are all C underneath, and the moment your hot loop is Python walking a tree you are back to one core doing interpreted work. Free-threading in 3.14 raises that ceiling and does not remove it.

A browser costs roughly a hundred times what an HTTP request costs. A Playwright page load holds a Chromium process, hundreds of megabytes of RAM and a second or more of wall clock. Ten thousand pages through a browser is a fleet and a bill; ten thousand through requests is a laptop and an afternoon. Establish that the data does not exist without JavaScript before you pay that.

Some ecosystems are ahead. Anti-detection tooling appears in Node first, because that is where the browser-automation work happens, and Python bindings follow. If your target sits behind a current bot-management platform and your team already writes TypeScript, insisting on Python costs months. The same reasoning applies to scrapers inside an existing JVM service, the subject of web scraping in Java.

Deployment is the tax nobody quotes. A scraper is a scheduled job with credentials, proxy configuration, browser binaries, a database and an alerting story. That work is identical in every language and it is most of the project.

Legal and ethical considerations

Scraping is a powerful tool, and it should be used responsibly:

• Respect robots.txt, with a parser that actually implements RFC 9309.
• Cache. Fetching the same page twice is your cost as well as theirs.
• Do not collect personal data without a lawful basis. GDPR and CCPA both reach data you found in public, and neither treats "it was on the open web" as a basis.
• Put a contact address in your User-Agent. Some fraction of the people who would otherwise block you will write to you first, and that conversation is usually cheaper than the ban.

The hiQ case does not say what it is quoted as saying. Almost every scraping article, this one's previous version included, cites hiQ Labs v. LinkedIn for the proposition that scraping publicly accessible data is permissible, and stops there. The Ninth Circuit rulings were preliminary-injunction decisions about likelihood of success, and they do stand for one useful point: the Computer Fraud and Abuse Act is a weak instrument against scraping of logged-out public pages. On 4 November 2022 the district court split the difference, siding with hiQ on CFAA access to public data while finding that hiQ had breached LinkedIn's User Agreement, partly through fake accounts. On 8 December 2022 the parties entered a consent judgment: $500,000 against hiQ, and a permanent injunction barring it from scraping LinkedIn. hiQ won the argument everyone cites and lost the case.

The operative lesson is the distinction the courts kept drawing. Logged-out pages and content behind an account are different legal objects, and the Supreme Court's 2021 decision in Van Buren v. United States framed CFAA liability as a gates-up-or-down question about entitlement to the area at all. Platforms win on contract law, not computer-fraud law. If your scraper creates an account, accepts terms, or logs in, you are on the part of the map where they win. In the EU, GDPR governs personal data whether or not it was public, and a "publicly available" defence has not worked. None of this is legal advice, jurisdictions differ sharply, and the position on AI training data has moved every few months since 2024.

The commercial calculation is simpler than the legal one. Most of the cost of scraping is not the first extraction, it is the second year: selectors that rot, proxies that burn, a target that adds a bot-management vendor in a quarter you had budgeted for something else. That is the arithmetic behind a data as a service pipeline rather than a repository of scripts, and it is worth doing on a spreadsheet before you pick either.

Start with requests and BeautifulSoup on one page. Add the retry adapter when the first crash annoys you, lxml when parsing shows up in a profile, curl_cffi when a site starts refusing Python, and Scrapy when you find yourself writing a queue. Every one of those steps is small, and none of them is the step you should take first.