Async Web Scraping in Python: aiohttp and asyncio

When you need to download thousands or tens of thousands of pages, synchronous code hits a wall: every request waits for its response before the next one starts. Threads help, but they burn memory and run into overhead. Async web scraping in Python solves the problem more elegantly — a single thread holds thousands of simultaneous connections, switching between them while they wait on the network.

This article is an advanced follow-up to the overview guide on Python web scraping. The basics (libraries, encodings) are covered there; here we focus on scaling scraping with asyncio and aiohttp.

Why async is faster for scraping

Scraping is an I/O-bound task: 99% of the time the program is simply waiting for the server's response. In synchronous code, that wait sits idle. Async lets you launch hundreds of other requests while one is still waiting.

• Synchronous: 1,000 pages at 0.5s each = ~500 seconds.
• Async (100 at a time): the same 1,000 pages = ~5 seconds.

Unlike threads, coroutines cost almost nothing in memory — tens of thousands of concurrent tasks in a single thread are realistic. Compared with a multithreaded approach, async scales noticeably higher.

Fetching the page: aiohttp

aiohttp is the standard async HTTP client. The key principle: one ClientSession for the whole program (it reuses connections), and many concurrent requests through asyncio.gather.

import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
        return await resp.text()

async def main(urls):
    async with aiohttp.ClientSession(headers={"User-Agent": "MyBot/1.0"}) as session:
        tasks = [fetch(session, url) for url in urls]
        pages = await asyncio.gather(*tasks, return_exceptions=True)
        return pages

urls = [f"https://example.com/page/{i}" for i in range(1, 1001)]
results = asyncio.run(main(urls))

return_exceptions=True matters: one failed task won't take down the whole gather — it comes back as an exception object you can handle.

Parsing content in async code

An important nuance: the HTML parsing itself (BeautifulSoup, lxml) is a synchronous, CPU-bound operation. If the HTML is heavy, parsing blocks the event loop and cancels out the async win. Light pages can be parsed right in the coroutine:

from bs4 import BeautifulSoup

async def fetch_and_parse(session, url):
    async with session.get(url) as resp:
        html = await resp.text()
    soup = BeautifulSoup(html, "lxml")     # fine for light pages
    return soup.find("h1").get_text(strip=True)

If the parsing is heavy, offload it to a process pool so it doesn't block the loop:

import asyncio
from concurrent.futures import ProcessPoolExecutor

def heavy_parse(html):
    soup = BeautifulSoup(html, "lxml")
    return [a["href"] for a in soup.select("a")]

async def fetch_and_parse(session, url, pool):
    async with session.get(url) as resp:
        html = await resp.text()
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(pool, heavy_parse, html)

For the fastest parser under async load, reach for lxml.

Often you're hitting an API rather than HTML pages — then the response body is already structured, and instead of a parser a simple await resp.json() is enough. That's both faster and more robust than parsing markup; techniques for these responses are collected in parsing JSON in Python.

Character encodings in async scraping

On await resp.text(), aiohttp tries to detect the encoding from the headers. On older sites and non-UTF-8 content this misfires. The fixes are the same as in synchronous code:

# option 1: explicit encoding
html = await resp.text(encoding="utf-8")

# option 2: work with bytes and hand them to the parser
raw = await resp.read()
soup = BeautifulSoup(raw, "lxml")    # the parser reads <meta charset> itself

# option 3: manual decoding
html = raw.decode("windows-1252", errors="replace")

The full theory of the problem is in the Python web scraping hub.

Controlling concurrency: semaphores

Firing 10,000 requests at once means taking down the target server, your own network, and earning a ban. Limit concurrency with a semaphore:

import asyncio
import aiohttp

async def fetch(session, url, semaphore):
    async with semaphore:                      # no more than N at a time
        async with session.get(url) as resp:
            return await resp.text()

async def main(urls, concurrency=20):
    semaphore = asyncio.Semaphore(concurrency)
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url, semaphore) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

Semaphore(20) guarantees no more than 20 active requests at once. This is your main "politeness" lever: tune the value so you don't overload the target site. Add small random pauses (await asyncio.sleep(random.uniform(0.1, 0.5))) for a more natural rhythm.

Proxies

In aiohttp, the proxy is passed as a request parameter:

async with session.get(url, proxy="http://user:pass@ip:port") as resp:
    html = await resp.text()

Rotation is simply picking a random proxy per request:

import random

PROXIES = ["http://ip1:port", "http://ip2:port", "http://ip3:port"]

async def fetch(session, url):
    proxy = random.choice(PROXIES)
    async with session.get(url, proxy=proxy) as resp:
        return await resp.text()

The overall proxy strategy (types, weeding out dead ones) is covered in rotating proxies for scraping.

Scraping through Tor

aiohttp doesn't support SOCKS directly — you need the aiohttp-socks package:

# pip install aiohttp-socks
import aiohttp
from aiohttp_socks import ProxyConnector

async def main():
    connector = ProxyConnector.from_url("socks5://127.0.0.1:9050")
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get("https://httpbin.org/ip") as resp:
            print(await resp.json())   # the Tor exit node's IP

Rotating the exit node with the NEWNYM signal (the stem library) is described in the Python web scraping hub. Note: Tor is slow — under high concurrency it becomes the bottleneck.

HTTPS/SSL

By default aiohttp verifies certificates. Disable it (for debugging only) or supply your own context:

import ssl

# disabling verification — NOT for production
async with session.get(url, ssl=False) as resp:
    ...

# your own SSL context
ctx = ssl.create_default_context(cafile="/path/to/ca.crt")
async with session.get(url, ssl=ctx) as resp:
    ...

Working with cookies

ClientSession stores cookies between requests automatically — like requests.Session:

async with aiohttp.ClientSession() as session:
    # log in — the server sets a session cookie
    await session.post("https://example.com/login",
                       data={"user": "u", "pass": "p"})
    # subsequent requests are already authenticated
    async with session.get("https://example.com/profile") as resp:
        html = await resp.text()

You can pass cookies by hand via the cookies={...} parameter.

Response status and headers

async with session.get(url) as resp:
    print(resp.status)                       # 200, 404 ...
    print(resp.headers.get("Content-Type"))
    if resp.status == 429:
        wait = int(resp.headers.get("Retry-After", 60))
        await asyncio.sleep(wait)            # doesn't block other tasks!
    resp.raise_for_status()

The key advantage: await asyncio.sleep() on a 429 only puts this coroutine to sleep, while the rest keep working. In synchronous code, time.sleep() would freeze everything.

Queues: asyncio.Queue

For crawling as links are discovered, use asyncio.Queue and a pool of worker coroutines:

import asyncio
import aiohttp

async def worker(name, queue, session, visited):
    while True:
        url = await queue.get()
        if url not in visited:
            visited.add(url)
            try:
                async with session.get(url) as resp:
                    html = await resp.text()
                # ... find new links and push them onto the queue:
                # for link in extract_links(html):
                #     await queue.put(link)
            except Exception as exc:
                print(f"{name} error {url}: {exc}")
        queue.task_done()

async def crawl(start_urls, num_workers=10):
    queue = asyncio.Queue()
    visited = set()
    for url in start_urls:
        queue.put_nowait(url)

    async with aiohttp.ClientSession() as session:
        workers = [asyncio.create_task(worker(f"w{i}", queue, session, visited))
                   for i in range(num_workers)]
        await queue.join()          # wait until the queue is empty
        for w in workers:
            w.cancel()

A set for deduplication, a Queue to coordinate workers — the async analog of the frontier from the hub. For distributed crawling, move the queue into Redis. A production-grade implementation of this pattern is provided by Scrapy (which is async under the hood too).

httpx as an alternative

httpx is a modern client with the same API for synchronous and async code, plus HTTP/2 support:

import httpx
import asyncio

async def main(urls):
    async with httpx.AsyncClient(http2=True, timeout=15) as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks, return_exceptions=True)
        return responses

If you want to switch the same code between sync and async and need HTTP/2, httpx is more convenient than aiohttp. On raw speed at volume they're comparable.

Pros and cons of async scraping

Pros:

• Enormous concurrency with minimal memory.
• Order-of-magnitude speedups on I/O-bound tasks.
• Cheap "pauses": asyncio.sleep doesn't block other tasks.
• Fine-grained rate control through semaphores.

Cons:

• Harder to write and debug (async/await everywhere).
• CPU-bound parsing still blocks the loop — you need a process pool.
• You can't mix with blocking libraries without run_in_executor.
• Easy to overload the target site — semaphore discipline is required.

When to choose it: thousands of pages and speed matters. For a couple hundred pages, requests + ThreadPoolExecutor is simpler. For crawling an entire site, use Scrapy, where async and queues are already built in. And if the scraper lives inside a web app, note that Django's ORM is still largely synchronous and async code there needs care (sync_to_async).

FAQ

asyncio + aiohttp or httpx — which should I use? Both are excellent. Pick aiohttp for a mature, async-first client; pick httpx if you want one API across sync and async or need HTTP/2. Performance at scale is comparable.

How many concurrent requests is safe? There's no universal number — it depends on the target. Start with a semaphore around 10–20, watch for 429s and errors, and tune from there. Politeness beats raw speed.

Does async make parsing faster too? No. Async speeds up the network wait, not CPU-bound HTML parsing. Offload heavy parsing to a ProcessPoolExecutor so it doesn't block the event loop.

Running large concurrent crawls in production means owning proxies, retries, and rate control on top of the code. If you'd rather receive clean data on a schedule, scraping.pro offers data as a service — we operate the async pipelines and hand you structured results.