Python Web Scraping: The Complete Guide

Web scraping is the automated collection of data from web pages and its conversion into a structured form: tables, JSON, or a database. Python web scraping has become the de facto standard for this job, and for good reason: the language has clean syntax, a huge ecosystem of libraries, and a mature community that has already solved almost every problem you will hit.

This guide is a hub. We walk through the entire process, from requesting a page to managing URL queues, map out the key libraries and tools, and link off to focused deep-dive articles for the narrow topics. If you are new to scraping with Python, read it top to bottom; if you already scrape, use it as a reference.

What web scraping is and what it's made of

Every scraper, regardless of scale, comes down to four stages:

1. Fetching the page — an HTTP request to the server and the response it returns (HTML, XML, JSON).
2. Parsing the content — pulling the data you need out of the markup using selectors (CSS, XPath) or regular expressions.
3. Normalizing and storing — bringing the data to a single format and writing it to a file or database.
4. Managing the crawl — the URL queue, deduplication, rate limiting, and retries.

A simple script can fit all of this into ten lines. A production crawler splits each stage into its own layer with queues, proxies, and distributed workers.

Libraries and tools at a glance

To keep things clear, let's group the tools by the role they play.

Fetching pages (HTTP clients)

Parsing HTML/XML

Dynamic sites (JavaScript)

Frameworks and platforms

How to choose

• Simple page, no JavaScript, one-off task → requests + BeautifulSoup.
• Need speed at volume → httpx/aiohttp + lxml/selectolax.
• Hundreds of thousands of pages, crawling a whole site → Scrapy.
• Content rendered by JavaScript → Playwright/Selenium.
• Scraping embedded in a web service → Django.
• You need maximum concurrency → an async approach.

Fetching the page

A basic request with requests:

import requests

url = "https://example.com"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()   # raises on 4xx/5xx
html = response.text

Key points:

• Always set a User-Agent. Many sites block requests carrying the default python-requests string.
• Always set a timeout, or your script can hang forever.
• raise_for_status() saves you from checking the status code by hand.

If the page is rendered with JavaScript, requests returns an empty shell. Then you need a browser engine:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    page.wait_for_selector(".content")   # wait for the data to appear
    html = page.content()
    browser.close()

If what you actually get back is structured API data rather than HTML, see the dedicated guide on parsing JSON in Python.

Libraries for parsing content

BeautifulSoup — the entry point for beginners

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")   # the lxml parser is faster than html.parser

title = soup.find("h1").get_text(strip=True)
links = [a["href"] for a in soup.select("a.product-link")]
price = soup.select_one(".price").text

BeautifulSoup forgives broken markup and reads almost like plain English. For pulling tables out of a page, there's a separate walkthrough on scraping tables with BeautifulSoup.

lxml — speed and XPath

from lxml import html as lxml_html

tree = lxml_html.fromstring(html)
titles = tree.xpath('//h2[@class="title"]/text()')
prices = tree.xpath('//span[@class="price"]/text()')

lxml is written in C and runs many times faster on large volumes. Full XPath gives you flexibility that CSS selectors can't match. A deep dive lives in Python scraping with lxml.

parsel and selectolax

parsel (the core of Scrapy) unifies CSS and XPath:

from parsel import Selector

sel = Selector(text=html)
sel.css("h1::text").get()
sel.xpath("//a/@href").getall()

Reach for selectolax when you need to chew through hundreds of thousands of documents — it's noticeably faster than even lxml on CSS queries.

Regular expressions

re is only appropriate for simple, flat patterns (a phone number, an email, a SKU). Don't try to parse nested HTML with regexes — the structure breaks the moment the markup shifts.

Handling character encodings

A common headache, especially on older sites and non-English content, is garbled text — пÑивеÑ instead of readable characters, or caf� instead of café. The cause is an incorrectly detected response encoding.

Why it happens

requests guesses the encoding from the Content-Type header. If the server didn't send it, or sent it wrong, the text is decoded with the wrong codec (often ISO-8859-1 is assumed when the page is really UTF-8, Windows-1252, or a legacy regional charset like Shift_JIS or Windows-1251).

Fix 1: set the encoding manually

response = requests.get(url)
response.encoding = "utf-8"      # or the site's real charset
html = response.text

Fix 2: auto-detection

response = requests.get(url)
response.encoding = response.apparent_encoding   # detected from the bytes
html = response.text

apparent_encoding uses charset-normalizer (bundled with modern requests), which analyzes the bytes and picks a codec.

Fix 3: work with the raw bytes

The most reliable path is to hand the bytes to the parser and let it read the <meta charset> declaration:

from bs4 import BeautifulSoup

response = requests.get(url)
soup = BeautifulSoup(response.content, "lxml")   # .content, not .text

response.content is the raw bytes; lxml and BeautifulSoup will find the encoding declaration inside the HTML themselves.

Fix 4: manual decoding

html = response.content.decode("windows-1252", errors="replace")

The errors="replace" option swaps undecodable bytes for � instead of crashing your script. For XML documents with non-UTF-8 content, the nuances are covered in parsing XML in Python.

Using multithreading

Scraping is mostly waiting on the network (I/O-bound), so threads give a real speedup despite the GIL: while one thread waits for a response, another runs.

ThreadPoolExecutor — the simplest approach

from concurrent.futures import ThreadPoolExecutor
import requests

urls = [f"https://example.com/page/{i}" for i in range(1, 101)]

def fetch(url):
    r = requests.get(url, timeout=10)
    return url, r.status_code

with ThreadPoolExecutor(max_workers=10) as executor:
    for url, status in executor.map(fetch, urls):
        print(url, status)

When to reach for multiprocessing

If your bottleneck is heavy HTML parsing and processing (CPU-bound) rather than the network, threads hit the GIL wall. Then multiprocessing helps — several processes, each with its own interpreter.

Python 3.13 shipped an experimental free-threaded build (PEP 703) that removes the GIL. It's promising for CPU-bound work, but for scraping — which is I/O-bound — async and threads already do the job well.

The best alternative — async

For thousands of simultaneous requests, threads burn too much memory. Async (asyncio + aiohttp) holds tens of thousands of connections in a single thread. That's a big topic on its own — see async web scraping in Python.

A word of caution: high concurrency is not a license to hammer someone else's server. Rate-limit yourself and respect robots.txt.

Using proxies

Under heavy scraping, a site will ban your IP based on request volume. The answer is a proxy pool and rotation.

Simple proxy setup

proxies = {
    "http":  "http://user:pass@123.45.67.89:8080",
    "https": "http://user:pass@123.45.67.89:8080",
}
response = requests.get(url, proxies=proxies, timeout=15)

Proxy rotation

import random
import requests

PROXIES = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]

def fetch_with_rotation(url):
    proxy = random.choice(PROXIES)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

Proxy types

• Datacenter — cheap and fast, but easy to detect and ban.
• Residential — real ISP IPs, more expensive but less conspicuous.
• Mobile — carrier IPs, the most trusted and the priciest.

In production it pays to keep a list of "healthy" proxies: drop the ones that time out or return 403, and re-check them periodically. A deeper walkthrough is in our guide to rotating proxies for scraping.

Scraping through Tor

Tor is a free way to change your exit IP. It's slower than paid proxies and suits modest volumes, but it costs nothing.

Connecting

After installing Tor (the daemon or Tor Browser), it exposes a SOCKS5 proxy on 127.0.0.1:9050:

import requests

proxies = {
    "http":  "socks5h://127.0.0.1:9050",
    "https": "socks5h://127.0.0.1:9050",
}
# needs: pip install requests[socks]
r = requests.get("https://httpbin.org/ip", proxies=proxies)
print(r.json())   # you'll see the Tor exit node's IP, not yours

The socks5h scheme (with the h) matters: DNS resolution goes through Tor rather than locally, so your real DNS query doesn't leak.

Getting a new identity (a fresh IP)

To get a new exit node, send the NEWNYM signal over the control port (9051) with the stem library:

from stem import Signal
from stem.control import Controller

def renew_tor_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password="your_password")
        controller.signal(Signal.NEWNYM)

You need to enable the control port in torrc and set a password hash (tor --hash-password).

Keep in mind: many large sites know the list of Tor exit nodes and either block them or show a CAPTCHA. Tor is good for speed-tolerant, low-volume jobs.

Working with HTTPS/SSL

By default requests verifies SSL certificates via the certifi bundle. Most of the time you don't need to configure anything. Problems show up on sites with self-signed or expired certificates.

Disabling verification (debugging only)

import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = requests.get(url, verify=False)   # NOT for production

Turning off verification is unsafe — it opens the door to a man-in-the-middle attack. It's only acceptable locally, for debugging.

The right way — point to your own CA

response = requests.get(url, verify="/path/to/custom-ca-bundle.crt")

Updating root certificates

If you hit SSLCertVerificationError on normal sites, upgrade certifi:

pip install --upgrade certifi

Working with cookies

Cookies are needed for sessions, authentication, and getting past "protection" pages that set a token and redirect.

Session stores cookies automatically

import requests

session = requests.Session()

# log in — the server returns a session cookie
session.post("https://example.com/login", data={"user": "u", "pass": "p"})

# subsequent requests are already authenticated
profile = session.get("https://example.com/profile")

Session stores and re-sends cookies between requests, and it also reuses TCP connections (faster) and shared headers.

Passing cookies by hand

cookies = {"sessionid": "abc123", "csrftoken": "xyz789"}
response = requests.get(url, cookies=cookies)

Reading cookies you received

response = requests.get(url)
for name, value in response.cookies.items():
    print(name, value)

Response status and headers

You have to inspect the server's response — otherwise you'll happily parse an error page as if it were data.

response = requests.get(url)

print(response.status_code)      # 200, 404, 403, 500 ...
print(response.reason)           # 'OK', 'Not Found'
print(response.headers["Content-Type"])
print(response.headers.get("Server"))
print(response.url)              # final URL after redirects
print(response.elapsed)          # response time

Handling statuses properly

import time

if response.status_code == 200:
    parse(response.text)
elif response.status_code == 404:
    log("Page not found")
elif response.status_code == 429:
    # Too Many Requests — we're being throttled
    wait = int(response.headers.get("Retry-After", 60))
    time.sleep(wait)
elif response.status_code in (403, 503):
    rotate_proxy()   # likely a ban — change IP

The Retry-After header tells you how long to wait before retrying. Statuses 403/503 often signal anti-bot defenses: rotating the proxy, changing the User-Agent, and pausing usually help. Persistent walls may need CAPTCHA solving.

Storing URLs and queues (overview)

When a scraper crawls a whole site, you need somewhere to keep the "not yet visited" and "already visited" URLs. This is called the frontier.

The simplest option — in-memory sets

from collections import deque

to_visit = deque(["https://example.com"])
visited = set()

while to_visit:
    url = to_visit.popleft()
    if url in visited:
        continue
    visited.add(url)
    # ... download, parse, add new links to to_visit

A set gives instant deduplication; a deque works as a FIFO queue.

When there's a lot of data

• Redis — a shared queue for several workers that survives restarts. Redis lists and sets are ideal for distributed crawling.
• A database (PostgreSQL/SQLite) — a URL table with statuses new / in_progress / done / failed, handy for durability and analytics.
• Task queues (Celery + a broker, RabbitMQ) — when scraping is embedded in an app.
• A Bloom filter — saves memory at millions of URLs: a probabilistic "have we seen this URL?" check.

In Scrapy, queue management, deduplication, and priorities are built in — one of the main reasons to pick the framework on big projects. Details are in the Scrapy tutorial.

Pros and cons of building it in Python

Pros:

• Low barrier to entry, readable code, fast prototyping.
• The richest ecosystem: from requests to Scrapy and Playwright.
• A huge community — almost any problem is already solved.
• Easy integration with data analysis (pandas, numpy) and databases.

Cons:

• The GIL limits CPU-bound processing (worked around with async and multiprocessing).
• Pure Python is slower than compiled languages on heavy parsing (lxml/selectolax in C rescue you here).
• Dynamic JS sites require heavy browser engines.
• Fragility: when a site's markup changes, selectors break — it needs maintenance.

Legal and ethical considerations

Scraping is a powerful tool, and it should be used responsibly:

• Respect robots.txt and the site's terms of use.
• Don't create excessive load: add delays, cap concurrency.
• Don't collect personal data without a lawful basis (keep GDPR and CCPA in mind).
• Use an honest User-Agent where appropriate, and cache so you don't hit the server more than necessary.

On the legal question specifically, US case law such as hiQ Labs v. LinkedIn has generally treated scraping of publicly accessible data as permissible, while the CFAA and contract terms still matter — and in the EU, GDPR governs any personal data you touch. When in doubt, get advice for your jurisdiction.

FAQ

Which Python library should I start with? requests + BeautifulSoup. It covers the vast majority of static pages and is the gentlest on-ramp to Python web scraping.

How do I scrape a JavaScript-heavy site? Use a headless browser — Playwright is the current default, with Selenium as the classic alternative. Both execute the page's JavaScript so you can read the fully rendered DOM.

How do I avoid getting blocked? Set realistic headers, throttle your request rate, rotate proxies, and respect robots.txt. Aggressive scraping is what gets IPs banned.

Is web scraping legal? Scraping publicly available data is broadly permitted in the US and UK, but it depends on what you collect and how. Avoid personal data without a lawful basis, and read the target site's terms.

Need the data but not the maintenance? scraping.pro runs custom scrapers and delivers clean, structured data on a schedule as a data extraction service — so you get the results without babysitting selectors, proxies, and browser farms. For continuous feeds, our data as a service handles the whole pipeline end to end.