Scrapy Web Scraping: Python Framework Tutorial

When you need to crawl not one page but a whole site — thousands or millions of URLs — a hand-rolled script quickly turns into a mess of queues, retries, and rotating proxies. Scrapy solves that for you: it's a full-fledged asynchronous crawling framework with built-in queues, deduplication, processing pipelines, and a middleware system. For serious Scrapy web scraping, that machinery is the whole point.

This article builds on the overview in Web scraping with Python. If you just need a one-off pull from a couple of pages, requests + BeautifulSoup is enough; Scrapy shines at scale.

Contents

1. Scrapy architecture
2. Fetching a page: Spider and Request
3. Parsing content: selectors
4. Character encoding in Scrapy
5. Concurrency and speed
6. Proxies
7. Scraping through Tor
8. HTTPS/SSL
9. Working with cookies
10. Response status and headers
11. Queues and URL deduplication
12. Item Pipeline: storing data
13. Pros and cons

1. Scrapy architecture

Scrapy is built on an asynchronous engine (Twisted) and made of connected components:

• Spider — your class: which URLs to start from and how to parse responses.
• Scheduler — the request queue, with deduplication and priorities.
• Downloader — fetches pages asynchronously.
• Middlewares — interceptors for requests/responses (proxies, headers, retries).
• Item Pipeline — processing and storing the extracted data.

You write only the Spider and the Pipeline — the framework handles everything else.

Creating a project

pip install scrapy
scrapy startproject myscraper
cd myscraper
scrapy genspider example example.com

2. Fetching a page: Spider and Request

In Scrapy you don't write a request loop by hand — you yield Request objects, and the engine runs them asynchronously.

import scrapy

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    start_urls = ["https://example.com/catalog"]

    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (compatible; MyBot/1.0)",
        "DOWNLOAD_DELAY": 1.0,          # pause between requests
        "ROBOTSTXT_OBEY": True,         # respect robots.txt
    }

    def parse(self, response):
        # extract product cards
        for card in response.css(".product-card"):
            yield {
                "title": card.css(".title::text").get(),
                "price": card.css(".price::text").get(),
                "url": response.urljoin(card.css("a::attr(href)").get()),
            }

        # follow to the next page
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

response.follow automatically resolves the relative URL and queues the request, so pagination is a couple of lines. Run it:

scrapy crawl catalog -o products.json

3. Parsing content: selectors

Under the hood Scrapy uses the parsel library (built on lxml), which supports both CSS and XPath:

# CSS
response.css("h1::text").get()
response.css(".price::text").getall()
response.css("a::attr(href)").getall()

# XPath
response.xpath("//h1/text()").get()
response.xpath('//div[@class="price"]/text()').get()

# regex right inside the selector
response.css(".price::text").re_first(r"\d+")

get() returns the first result (or None), getall() returns a list. This is friendlier than "raw" lxml because it handles missing elements safely. More on XPath in the lxml tutorial.

If a page actually calls an API and returns JSON rather than HTML, Scrapy parses it directly through response.json() — no selectors needed. How to clean and reshape such responses is covered in detail in Parsing JSON in Python.

Items and ItemLoader

For structured projects it's convenient to describe data as an Item and populate it through an ItemLoader with cleaning processors (trimming whitespace, type coercion). For simple spiders, plain dictionaries as above are enough.

4. Character encoding in Scrapy

Most of the time Scrapy detects the encoding correctly from the headers and <meta charset> — non-ASCII text "just works." If you do get mojibake (an older site served as windows-1252 or ISO-8859-1, for example), you can set the encoding explicitly when building the response or decode the body manually:

def parse(self, response):
    # force re-read the body in a specific encoding
    text = response.body.decode("windows-1252", errors="replace")
    sel = scrapy.Selector(text=text)

The general theory of encoding problems is covered in Web scraping with Python. Also check FEED_EXPORT_ENCODING = "utf-8" in your settings so non-ASCII characters don't turn into \uXXXX escapes in the final JSON.

5. Concurrency and speed

Scrapy's headline advantage is out-of-the-box concurrency. Dozens of requests run at once without threads. It's tuned with settings:

# settings.py
CONCURRENT_REQUESTS = 16              # total concurrent requests
CONCURRENT_REQUESTS_PER_DOMAIN = 8    # per single domain
DOWNLOAD_DELAY = 0.5                  # base pause
AUTOTHROTTLE_ENABLED = True           # auto-tune the speed
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

AutoThrottle is the smart bit: Scrapy slows itself down when the server starts responding more slowly, balancing speed against politeness. It saves you from hand-tuning delays. Because everything is asynchronous, you don't need threads — compare that with the approach in asynchronous scraping in Python, which Scrapy implements "under the hood."

6. Proxies

The simplest approach is to set the proxy in the request meta:

yield scrapy.Request(url, meta={"proxy": "http://user:pass@ip:port"})

For rotating a pool, a ready package is more convenient:

pip install scrapy-rotating-proxies

# settings.py
ROTATING_PROXY_LIST = [
    "ip1:port",
    "ip2:port",
    "ip3:port",
]
DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

The middleware rotates proxies itself, tracks "banned" ones, and drops dead entries. The general theory is in our guide to proxies for scraping.

7. Scraping through Tor

Tor connects as a SOCKS5 proxy through meta:

yield scrapy.Request(url, meta={"proxy": "socks5h://127.0.0.1:9050"})

Rotating the exit node via stem (the NEWNYM signal) is covered in Web scraping with Python. In practice, for Scrapy people more often use paid rotating proxies — they're faster and get banned less than Tor exit nodes.

8. HTTPS/SSL

By default Scrapy verifies certificates. If you need to relax verification for a problematic site (only deliberately):

# settings.py
DOWNLOADER_CLIENT_TLS_METHOD = "TLS"
# for self-signed certificates you can configure a context factory

In most cases SSL "just works." The general principles of connection security are covered in Web scraping with Python.

9. Working with cookies

Cookies are enabled by default in Scrapy (COOKIES_ENABLED = True) — the engine keeps the session across requests automatically. To pass cookies manually:

yield scrapy.Request(url, cookies={"sessionid": "abc123"})

For authentication, FormRequest is handy:

def parse(self, response):
    return scrapy.FormRequest.from_response(
        response,
        formdata={"username": "user", "password": "pass"},
        callback=self.after_login,
    )

from_response picks up the form's hidden fields for you (including a CSRF token) — which removes a classic login headache.

10. Response status and headers

Access to status and headers is via the response object:

def parse(self, response):
    print(response.status)              # 200, 404 ...
    print(response.headers.get("Content-Type"))

By default Scrapy handles only 2xx and skips 4xx/5xx. Retries are managed by the built-in RetryMiddleware:

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 403]

Scrapy can respect a 429 (Too Many Requests) together with the Retry-After header. The overall logic of status codes is covered in Web scraping with Python.

11. Queues and URL deduplication

This is where Scrapy is especially strong — what you'd build by hand in a homemade scraper is built in here:

• The Scheduler holds a request queue with priorities.
• The Dupefilter automatically discards URLs it has already seen (by request fingerprint).
• The queue can be moved to disk (JOBDIR) so an interrupted crawl can resume:

scrapy crawl catalog -s JOBDIR=crawls/catalog-1

For distributed crawling across several machines there's scrapy-redis — a shared queue and dupefilter in Redis, so multiple workers can crawl one site together.

12. Item Pipeline: storing data

Extracted items pass through the pipeline, where they're validated, cleaned, and saved:

# pipelines.py
import pymongo

class MongoPipeline:
    def open_spider(self, spider):
        self.client = pymongo.MongoClient("mongodb://localhost:27017")
        self.db = self.client["scraping"]

    def process_item(self, item, spider):
        self.db["products"].update_one(
            {"url": item["url"]}, {"$set": dict(item)}, upsert=True
        )
        return item

    def close_spider(self, spider):
        self.client.close()

# settings.py
ITEM_PIPELINES = {"myscraper.pipelines.MongoPipeline": 300}

For a simple export you don't need a pipeline — the -o products.csv flag (or .json, .jsonl) saves the result directly.

13. Pros and cons of Scrapy

Pros:

• Concurrency, queues, deduplication, and retries — out of the box.
• High performance across large numbers of pages.
• Clean architecture: Spider, Middleware, and Pipeline are separated.
• Ready extensions: proxy rotation, scrapy-redis, auto-throttling.
• Resumable jobs and easy export to any format.

Cons:

• A steeper learning curve — you need to understand the architecture and Twisted.
• Overkill for a couple of pages (requests + BeautifulSoup is simpler there).
• JavaScript sites need integration (scrapy-playwright or Splash).
• The Twisted async model feels unfamiliar next to modern asyncio — though recent Scrapy runs on an asyncio reactor and supports async def callbacks, so the two worlds are converging.

For dynamic sites, add scrapy-playwright, which renders pages in a real browser. If the content is simple and static and you already have a Django stack, it's sometimes easier to scrape within Django.

Building a million-URL crawl and don't want to run the infrastructure yourself? scraping.pro operates large Scrapy pipelines as a managed data extraction service — proxies, retries, monitoring, and clean delivery included.