Async Web Scraping in Python: aiohttp and asyncio

One thousand pages, half a second of server latency each. Fetched one after another, that run takes 502 seconds. Fetched through a single aiohttp session with asyncio.gather, it takes 5.21 seconds. Both numbers come from the same machine on the same afternoon, and the second one hides something worth knowing: nobody set a concurrency limit in that script. aiohttp set one anyway. Its connection pool stops at 100 simultaneous connections out of the box, and that default is doing most of the work people credit to their own code.

This article is an advanced follow-up to the overview guide on Python web scraping, where the libraries, selectors and encodings live in the general case. Here the subject is what happens when the page count goes up by three orders of magnitude. We rechecked every version, default and error message below on 10 August 2026, against Python 3.11.15 and aiohttp 3.14.3, released 22 July 2026 per the aiohttp changelog.

Where the speed actually comes from

Scraping is I/O-bound. For most of a request the process holds an open socket and waits for bytes that have not arrived. Synchronous code spends that wait doing nothing; async code spends it starting the next request. Here is one 1,000-page workload under four configurations, against a local server that sleeps 0.5 s before answering:

We measured this on a single 2-vCPU Linux container, Python 3.11.15, aiohttp 3.14.3, client and server on loopback so network jitter does not smear the result. The delay is artificial and uniform, which real sites never are. Your numbers will differ. The ratios will not.

The middle rows are the ones to stare at. Semaphore(20) protected you from nothing the defaults were not already handling, and it slowed the run fivefold, which is exactly what you asked for when you said twenty at a time. Lifting the connector limit to 1,000 did not deliver the 0.5 s naive arithmetic predicts, because the single-process server became the slow part.

Think in per-page units. One wasted second per page across a ten-thousand-page crawl is close to three hours. One wasted 100 ms is seventeen minutes. That unit is why the parsing section below runs longest.

The concurrency number you did not set

An earlier version of this article said that firing 10,000 requests at once means taking down the target server. That is not what happens, and the correction matters because the advice built on it stays right for a different reason.

ClientSession builds a TCPConnector if you do not pass one. Read from the client reference and confirmed by introspection on 10 August 2026:

• limit is 100, the ceiling on simultaneous connections for the whole session.
• limit_per_host is 0, meaning no limit. All 100 can land on one host.
• ttl_dns_cache is 10 seconds, use_dns_cache is on, happy_eyeballs_delay is 0.25.
• The default timeout is ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5).

So gather over 10,000 URLs does not open 10,000 sockets. It creates 10,000 Task objects that queue behind a pool of 100, each holding whatever its coroutine captured. Bounding the fan-out is a memory argument, not a politeness one.

Three levers get confused with each other constantly, so name them apart:

• TCPConnector(limit=...) bounds open sockets. Your resource ceiling.
• TCPConnector(limit_per_host=...) bounds sockets to one host. The default is unlimited, so this is what stops a broad crawl concentrating all its pressure on one unlucky domain.
• asyncio.Semaphore(n) bounds coroutines in flight, which is not a rate. Twenty concurrent requests against a server answering in 50 ms is 400 per second; against one answering in 2 s it is ten.

If you owe the site a rate rather than a concurrency, use a rate limiter. aiolimiter implements a leaky bucket over async with, and AsyncLimiter(100, 30) means at most 100 entries in any 30-second window whatever the responses do. Version 1.2.1, dated 8 December 2024, is current. And set ClientTimeout explicitly on every production run: at the 300-second default a hundred hung targets hold every connection slot for five minutes.

Fetching pages: one session, many requests

One ClientSession for the program, never one per request. The session owns the pool, and a fresh session per URL throws away keep-alive with it. The quickstart is blunt: "Don't create a session per request."

import asyncio
import aiohttp

TIMEOUT = aiohttp.ClientTimeout(total=15, connect=5, sock_read=10)

async def fetch(session, url):
    async with session.get(url, timeout=TIMEOUT) as resp:
        resp.raise_for_status()
        return await resp.text()

async def main(urls):
    connector = aiohttp.TCPConnector(limit=100, limit_per_host=8)
    async with aiohttp.ClientSession(connector=connector,
                                     headers={"User-Agent": "MyBot/1.0"}) as session:
        tasks = [fetch(session, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

results = asyncio.run(main([f"https://example.com/page/{i}" for i in range(1, 1001)]))

return_exceptions=True keeps one failed fetch from killing the batch. It returns the exception object in the results list instead of raising, so the list is now mixed and code that assumes every element is a string breaks on the first 404. Filter with isinstance(r, Exception) before touching anything.

gather is no longer the modern default. Python 3.11 added asyncio.TaskGroup, and the standard library states the difference plainly: if a subtask raises, "TaskGroup will, while gather will not, cancel the remaining scheduled tasks." For a scraper that trade runs backwards, since one dead URL in a thousand is normal and you want the other 999. Use TaskGroup when the tasks form one unit and a failure makes the rest pointless.

gather also materialises every result in one list: a thousand pages at 182 KB each is 182 MB resident before you parse a byte. Past a few thousand URLs, stream with asyncio.as_completed or the queue pattern below.

Threads against coroutines, measured

The claim that coroutines are cheaper than threads circulates without a number attached. A thousand pending asyncio.Task objects, measured with tracemalloc between two snapshots, allocate 1,379 KiB, about 1.4 KB each; ten thousand scale linearly to 13,886 KiB. A thousand parked threading.Thread objects, measured as the VmRSS delta in /proc/self/status, add 16,764 KiB resident, about 17 KB each, plus 8,983 MiB of virtual address space, because each thread reserves an 8 MiB stack under the default RLIMIT_STACK.

Twelve times the resident memory. The argument is smaller than folklore suggests: 17 KB per thread is survivable at a thousand threads and not at fifty thousand, which is where async stops being a preference. Thread pools and their trade-offs are covered in the Python web scraping hub.

Check ulimit -n before celebrating. Sockets are file descriptors. A soft limit of 1,024 caps a crawl at about a thousand concurrent connections whatever the connector says, and it shows up as a flood of OSError: [Errno 24] Too many open files partway through a long run.

Parsing is where async crawls die

This section changed most on rechecking, because the standard advice turned out weaker than its reputation.

HTML parsing is synchronous, CPU-bound work, and while it runs the event loop runs nothing else: not the other coroutines, not the socket reads, not the timeout callbacks. Here is what four parsers do to a 182,494-byte page, the PyPI project page for requests, extracting every href. Median of 20 runs after a warm-up.

Measured with bs4 4.14.3, lxml 6.1.0 and selectolax 0.4.11 on Python 3.11.15, 2 vCPUs.

Note rows two and four. Handing BeautifulSoup the lxml parser does not buy lxml speed. It buys lxml's tokenizer feeding BeautifulSoup's object tree, fifteen times slower than using lxml on its own. Write-ups that recommend BeautifulSoup(html, "lxml") as the fast option are making a smaller claim than it sounds: fastest among BeautifulSoup configurations.

The same parsers inside a real concurrent fetch, 200 pages from a local server with 0.5 s latency, the server in a separate process so client CPU load cannot slow it:

The process pool, the fix every tutorial reaches for, recovered about a quarter of the loss. It cannot do better, and the reason is arithmetic: 200 pages at 109 ms is 21.8 CPU-seconds, and two cores retire that in no less than 10.9 s whichever process runs it. Moving work off the loop does not make the work smaller. A 16-core box scales further. The ceiling is still your core count, and the queue behind it still grows at network speed.

The thread pool made things worse. BeautifulSoup is pure Python, so its threads fight over the GIL and add scheduling overhead to a job that was already serialised. Thread pools suit blocking C extensions that release the GIL, not pure-Python parsing.

What actually worked was changing the parser: inline lxml.html finished 200 pages in 1.71 s against a 1.17 s floor, with no executor and no extra processes.

import lxml.html

async def fetch_and_parse(session, url):
    async with session.get(url) as resp:
        body = await resp.read()          # bytes, not text: let lxml handle charset
    doc = lxml.html.fromstring(body)
    return doc.xpath("//h1/text()")

Offloading is still right for genuinely heavy work: full-text extraction, image decoding, anything numeric. Measure the parse first and offload second, and pass your own pool when you do. run_in_executor(None, ...) uses the default executor, a ThreadPoolExecutor sized min(32, cpu_count + 4), six threads on a two-core box, shared with everything else that touches it including DNS.

When the target speaks JSON, skip all of this. await resp.json() hands back a dict with no tree to build, and the techniques are collected in parsing JSON in Python.

Encodings: aiohttp does not guess

The old version of this article said aiohttp "tries to detect the encoding from the headers" and that on older sites this misfires. The first half is right and the second half is too gentle. aiohttp does not sniff bytes at all.

Reading ClientResponse.get_encoding() and checking it against a live server on 10 August 2026: if Content-Type carries a charset, aiohttp uses it; if the body is JSON, it assumes UTF-8; otherwise it calls fallback_charset_resolver, whose default in 3.14.3 is literally lambda r, b: "utf-8". Serve windows-1252 bytes with a bare Content-Type: text/html and await resp.text() raises:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 0: invalid continuation byte

A hard failure, which is the better of the two outcomes. requests would have guessed, sometimes wrongly, and handed you a plausible-looking string. The documented hook installs a real detector, and the alternatives are the same as in synchronous code:

from charset_normalizer import detect

session = aiohttp.ClientSession(
    fallback_charset_resolver=lambda r, b: detect(b)["encoding"] or "utf-8"
)

raw = await resp.read()
doc = lxml.html.fromstring(raw)           # best: the parser reads <meta charset> itself

html = await resp.text(encoding="utf-8")  # you know the site, you say so
html = raw.decode("windows-1252", errors="replace")   # last resort, lossy by design

Handing bytes to the parser also skips a decode-and-re-encode round trip per page. The general theory of broken charsets lives in the Python web scraping hub.

Status codes and the Retry-After trap

This block shipped in the earlier version of this article, and it has a bug:

if resp.status == 429:
    wait = int(resp.headers.get("Retry-After", 60))
    await asyncio.sleep(wait)

RFC 9110 defines the field as Retry-After = delay-seconds / HTTP-date and gives both forms: Retry-After: 120 and Retry-After: Fri, 31 Dec 1999 23:59:59 GMT. Feed the second to int() and you get ValueError: invalid literal for int() with base 10. Inside a gather with return_exceptions=True that failure lands in a results list nobody inspects, and the crawl keeps hammering a server that just asked, in a legal format, to be left alone.

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(value, default=60.0, cap=300.0):
    if not value:
        return default
    try:
        return min(float(int(value)), cap)
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(value)      # RFC 9110 HTTP-date form
    except (TypeError, ValueError):
        return default
    return max(0.0, min((when - datetime.now(timezone.utc)).total_seconds(), cap))

The cap is there because servers do send Retry-After: 86400, and a coroutine that sleeps for a day is a leak with good manners.

Sleeping one coroutine does not slow the crawl. That is the advantage over time.sleep, and it is also the trap. When worker 7 backs off on a 429, workers 1 through 100 keep firing at the same host and the site sees no reduction. Per-task backoff is not rate limiting. A 429 calls for a per-host pause every worker respects: a shared asyncio.Event, a shared limiter, or a circuit breaker keyed on the host. For the retry mechanics, tenacity 9.1.4, released 7 February 2026, understands coroutines directly and sleeps asynchronously between attempts.

Proxies, and what the bandwidth is worth

In aiohttp the proxy is a per-request parameter, not a session setting:

async with session.get(url, proxy="http://ip:port",
                       proxy_auth=aiohttp.BasicAuth("user", "pass")) as resp:
    html = await resp.text()

aiohttp ignores your environment proxies unless you ask it not to. ClientSession(trust_env=False) is the default, verified by introspection on 10 August 2026, so HTTP_PROXY and HTTPS_PROXY do nothing until you pass trust_env=True. requests honours them out of the box. Traffic that went through a proxy under the old synchronous script starts going direct after the rewrite, with no error and no log line.

Rotation by random.choice(PROXIES) is the version everyone writes first, and it stops being adequate the moment one endpoint dies, because a uniform picker keeps routing a third of the crawl into a hole. Count failures per endpoint and drop one out of the pool after n consecutive errors. Pool types and health checking are covered in rotating proxies for scraping.

Async makes it easy to spend bandwidth faster than you intended, and residential bandwidth is sold by the gigabyte. Read from Webshare's plans on 10 August 2026: rotating residential starts at $3.50/GB and falls to $1.40/GB at 3,000 GB, while datacenter endpoints run from $0.0299 per proxy. At the 182 KB page used above, ten thousand pages is 1.82 GB, so that run is about $6.37 through entry-tier residential and a rounding error through datacenter IPs. Managed APIs bill per request instead: ScrapingBee lists $49/month for 250,000 credits and $99/month for 1,000,000, both exclusive of VAT, on the same date. Which model wins depends on page weight, and page weight is the number nobody measures first.

Tor and SOCKS

aiohttp has no SOCKS support of its own; its documentation covers HTTP proxies only. Add aiohttp-socks, version 0.11.0, released 9 December 2025, requiring aiohttp 3.10 or newer:

# pip install aiohttp-socks
import aiohttp
from aiohttp_socks import ProxyConnector

async def main():
    connector = ProxyConnector.from_url("socks5://127.0.0.1:9050")
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get("https://httpbin.org/ip") as resp:
            print(await resp.json())   # the Tor exit node's IP

Rotating the exit node means sending NEWNYM over the control port, usually through stem, and the walkthrough is in the Python web scraping hub. Check the shipping date first: stem 1.8.2 is from 6 June 2023, and its own description still mentions supporting "both the python 2.x and 3.x series." Nothing is broken and nothing is moving.

Tor is slow, and under high concurrency it becomes the bottleneck rather than the target site. Stacking concurrency on a saturated circuit adds latency without throughput.

TLS: the handshake gives you away before the first header

By default aiohttp verifies certificates. Pass ssl=False for debugging, or ssl=ssl.create_default_context(cafile="/path/to/ca.crt") to supply your own trust store.

What has changed since this article was written is not the API but what the other end does with the handshake. TLS fingerprinting reads the cipher list, extension order and ALPN values your client offers, and Python's ssl module offers a combination no browser produces. That comparison happens before a single HTTP header is parsed, so the carefully chosen User-Agent above never gets a vote. aiohttp, httpx and requests share the exposure, because they share OpenSSL defaults.

The 2026 Thales/Imperva Bad Bot Report, published April 2026 on 2025 data, put automated traffic at 53% of everything observed, 40 points of that classed as bad bots, and reported AI-driven bot activity up 12.5x year over year. On 1 July 2025 Cloudflare changed the default for new domains to block AI crawlers unless they pay, so a growing share of the web refuses unfamiliar clients as policy rather than as abuse control.

The Python answer is curl_cffi, which binds curl-impersonate and reproduces browser TLS and HTTP/2 fingerprints rather than approximating them. Its AsyncSession(impersonate="chrome") takes the same gather shape as aiohttp, so migration is mechanical. Version 0.16.0 landed 1 August 2026, HTTP/3 arrived in 0.15.0 in April 2026.

Be honest about the boundary. Impersonation gets you past passive TLS checks and does nothing about behavioural scoring, JavaScript challenges or IP reputation. Against a target running all three the async question stops mattering, because the block rate sets your ceiling. Sites that fight back at that level are where a managed extraction service earns its keep.

Cookies, and one silent drop

ClientSession keeps a cookie jar across requests, the way requests.Session does, and the default jar discards cookies from IP-address hosts. CookieJar(unsafe=False) follows the rule that cookies belong to registered domain names. Tested on 10 August 2026: a Set-Cookie from http://127.0.0.1:8080/ leaves the default jar holding zero cookies and an unsafe=True jar holding one. Anyone debugging a login flow against a local server or a bare proxy IP hits this and blames their credentials.

Queues, workers and a clean stop

For crawling where links are discovered as you go, a fixed pool of workers pulling from an asyncio.Queue beats a growing list of tasks. The queue bounds memory and the worker count bounds concurrency, with no second mechanism.

import asyncio
import aiohttp

async def worker(name, queue, session, visited):
    while True:
        try:
            url = await queue.get()
        except asyncio.QueueShutDown:      # Python 3.13+
            return
        try:
            if url not in visited:
                visited.add(url)           # no await between check and add
                async with session.get(url) as resp:
                    html = await resp.text()
                # for link in extract_links(html): await queue.put(link)
        except Exception as exc:
            print(f"{name} error {url}: {exc}")
        finally:
            queue.task_done()

async def crawl(start_urls, num_workers=10):
    queue = asyncio.Queue(maxsize=10_000)
    visited = set()
    for url in start_urls:
        queue.put_nowait(url)

    async with aiohttp.ClientSession() as session:
        workers = [asyncio.create_task(worker(f"w{i}", queue, session, visited))
                   for i in range(num_workers)]
        await queue.join()
        for w in workers:
            w.cancel()
        await asyncio.gather(*workers, return_exceptions=True)

Three details there are load-bearing.

task_done() belongs in a finally. Miss it on any path and queue.join() waits forever, which presents as a crawl that finished its work and then hung with an idle CPU. Nothing is wrong except a counter, and counters do not appear in tracebacks.

The visited check is safe without a lock, and only because of where the awaits are. Between if url not in visited and visited.add(url) there is no await, so no other coroutine can run in between. Insert one there, an async dedup lookup for instance, and two workers will fetch the same page. That is asyncio's concurrency model in one line: you are interrupted only where you yield.

Fire-and-forget tasks vanish. The asyncio documentation is explicit: "The event loop only keeps weak references to tasks. A task that isn't referenced elsewhere may get garbage collected at any time, even before it's done." A crawler that calls asyncio.create_task(save(row)) and drops the handle loses rows under load. Keep a set of pending tasks and discard from it in add_done_callback. The final gather over the cancelled workers matters for the same reason: cancel() requests cancellation, it does not perform it, and closing the session under coroutines still unwinding produces "Unclosed connector" warnings and occasionally lost writes.

Python 3.13 added Queue.shutdown(), which unblocks waiting getters with QueueShutDown and retires the cancel-and-hope pattern above. A set for deduplication works until the URL count outgrows one process; move the frontier into Redis and the same loop becomes distributed. Scrapy has shipped this pattern for years, and release 2.17.0 from 7 July 2026 keeps AsyncioSelectorReactor as the default reactor, so a modern Scrapy project runs on your event loop even though the framework is built on Twisted.

httpx, curl_cffi, and how alive each of them is

httpx gets recommended as the modern alternative, one API across sync and async plus HTTP/2. Both halves are true. The maintenance picture is more mixed, so here is what the PyPI page said on 10 August 2026: the current release is 0.28.1, dated 6 December 2024, twenty months old, with classifiers stopping at Python 3.12. The GitHub API reports the repository as not archived, with commits as recently as 29 March 2026. Release cadence has stalled rather than the project, and whether a 1.0 lands is not knowable from outside.

The httpx snippet in the earlier version of this article also had a defect. AsyncClient(http2=True) does not work on a plain pip install httpx:

ImportError: Using http2=True, but the 'h2' package is not installed.
Make sure to install httpx using `pip install httpx[http2]`.

Reproduced locally on 0.28.1, and consistent with the project's own HTTP/2 page, which states that HTTP/2 "is not enabled by default, because HTTP/1.1 is a mature, battle-hardened transport layer." Enabling it guarantees nothing either: without h2 over ALPN you fall back to HTTP/1.1 silently. aiohttp has never supported HTTP/2 at all.

Pick aiohttp for volume against ordinary targets. Pick httpx if one API across sync and async is worth more than release frequency. Pick curl_cffi when the handshake is what gets you blocked.

Where async stops helping

CPU-bound work. The parsing section applies to every synchronous call in your coroutine: compression, image decoding, regex over megabytes, hashing.

Blocking libraries. A synchronous database driver, an SDK wrapping requests, an ORM call that touches the network. Each freezes the loop for its full duration, and run_in_executor is the only door out. Django has moved since this article was written: per the Django 6.1 documentation every QuerySet method that issues SQL has an a-prefixed variant and async for works on QuerySets. Transactions remain the exception, still to be written as one synchronous function called through sync_to_async().

DNS. aiohttp's DefaultResolver is ThreadedResolver unless aiodns is installed, confirmed by reading aiohttp/resolver.py on 10 August 2026. Threaded resolution runs through the default executor, six threads on a two-core box, so ten thousand cold hostnames is a queue you did not know you had. Install aiodns and pass AsyncResolver for broad crawls; a single-host crawl will not notice.

Speed-ups that lag the interpreter. uvloop swaps the event loop for libuv and wins on connection-heavy work, but its latest release is 0.22.1 from 16 October 2025 and its PyPI classifiers stop at Python 3.13, even though the 0.22.0 notes mention Python 3.14 fixes and free-threading support. Python 3.14 has been stable since 7 October 2025 per the Python developer guide. Check for a matching wheel, and note that uvloop has no Windows build at all.

Reading a stuck crawler

A crawl that has stopped making progress looks exactly like a crawl waiting on the network. Python 3.14 put the difference in reach: python -m asyncio ps PID prints every running task in a live process with its coroutine stack and awaiter chain, and python -m asyncio pstree PID renders it as a tree and reports cycles in the await graph. Details are in the 3.14 release notes.

On older interpreters, and alongside it, log the count of in-flight requests once a second. Pinned at your connector limit means a stalled loop, usually a blocking call in the coroutine rather than a slow target. Oscillating near zero means a rate limiter or a DNS queue. A run that finishes its work and hangs is a missing task_done(). Rows missing at random are fire-and-forget tasks collected by the garbage collector, or a gather results list whose exceptions were never inspected.

Choosing async, or not

Async pays when the page count is in the thousands and the per-page work is small. It does not pay for two hundred pages, where requests with a ThreadPoolExecutor reaches the same wall clock with a fraction of the reasoning, and the Python web scraping hub covers that route. It does not pay when parsing dominates, because the loop is not the constraint there. It pays enormously at fifty thousand concurrent connections, where the thread measurements above turn from awkward into impossible.

What you take on in exchange is a class of bug that is quiet, load-dependent and invisible in a traceback: the swallowed ValueError in a gather, the vanished fire-and-forget task, the dropped cookie from an IP host, the six-thread DNS queue. Each produces a scraper that runs and returns fewer rows than it should. Budget for observability from the first run, not the first incident.

For a whole site with link discovery, Scrapy has already solved the queue, the dedup and the retry policy. For a steady feed of structured records where the pipeline is not the point, data as a service moves proxies, retries and rate control to somebody who watches them. For throughput against cooperative endpoints, the code here holds at surprising volumes: one session, a bounded connector, an honest Retry-After parser and a fast tree builder.

Measure the parse before you tune the loop.