Scrapy Web Scraping: Python Framework Tutorial

A catalogue with 40,000 product URLs is not forty thousand times harder than one page. It is a different problem. One page needs an HTTP client and a selector. Forty thousand need a queue that survives a crash, a memory of every URL already fetched, a retry policy that tells a 503 apart from a 404, and a pool of rotating proxies for the day the site starts counting your IP. Build that by hand and you have rebuilt most of Scrapy, with fewer tests.

Scrapy hands you the machinery: an asynchronous engine, a scheduler that fingerprints requests so you never fetch the same URL twice, downloader middlewares, and item pipelines that push finished records into MongoDB or anywhere else. For Scrapy web scraping at any real size, that machinery is the whole reason to be here.

Everything below was rechecked against Scrapy 2.17.0, released 7 July 2026, on Python 3.11, on 13 August 2026. Every command and snippet here was executed, and three claims from the earlier version of this article did not survive: Scrapy does not verify TLS certificates by default, it cannot reach a SOCKS proxy through meta on the standard download handler, and it ignores Retry-After entirely. Each correction gets its own section.

This article builds on the overview in Web scraping with Python. For a one-off pull from two pages, requests plus BeautifulSoup is less work than a project skeleton.

1. What the crawl costs before you write a line of code

The framework is free. The traffic is not, and the traffic bill decides your architecture.

Take a million product pages at 200 KB of HTML each: 200 GB across the wire, once. Read from Webshare's pricing page on 13 August 2026, rotating residential bandwidth lists at $7.00/GB and falls to $1.40/GB at the 3,000 GB tier. That crawl costs $280 to $1,400 in proxy traffic alone, before a row lands in your database. Datacentre IPs start at $0.0299 per proxy per month at the 100-proxy tier and get blocked far sooner.

Now price it per request instead. Zyte's published pricing, read the same day, puts Zyte API pay-as-you-go at $0.13 to $1.27 per 1,000 requests for a plain HTTP response body, and $1.01 to $16.08 per 1,000 when the page has to be rendered in a browser. A million raw fetches is $130 to $1,270. A million rendered ones is $1,010 to $16,080.

Rendering costs roughly ten times what fetching costs. That ratio decides more of your design than any framework feature, and every hour spent finding the JSON endpoint behind a React page pays for itself several times over. Time behaves the same way: one wasted second per page across a million pages is 11.5 days of wall clock one request at a time, and about 17 hours at sixteen concurrent.

Whether you run this pipeline or buy the finished rows as a data feed is a budget question before it is a technical one.

2. Scrapy architecture

Scrapy is an asynchronous crawling framework built on Twisted. The pieces are worth naming, because you configure them by name.

• Spider: your class. Which URLs to start from, and how to turn a response into items or more requests.
• Downloader: fetches pages concurrently. Proxies, TLS, timeouts and DNS live here.
• Middlewares: interceptors on every request and response. Proxy assignment, retries, redirects, robots.txt and cookies are all middlewares you can reorder or replace.
• Scheduler: the part people underestimate. It holds a priority queue of pending requests plus the dupefilter, which drops any request whose fingerprint it has seen before. The fingerprint covers the canonicalised URL, the HTTP method and the body, so ?utm_source= variants collapse into one fetch while a POST and a GET to the same URL stay separate. Since 2.14.0 the default priority queue is DownloaderAwarePriorityQueue, which spreads work across domains instead of hammering whichever one sits at the front. Both queue and dupefilter can move to disk or to Redis, which is what turns a single-machine crawl into a distributed one.
• Item Pipeline: small classes that validate, clean, deduplicate and store what the spider yielded.

You write the Spider and the Pipeline. The framework owns the rest.

The engine changed underneath. Older write-ups describe Scrapy as a Twisted-only world sitting awkwardly next to asyncio. Since 2.13.0, released 8 May 2025, the asyncio reactor is the default, and every run says so:

[scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor

3. Creating a project, and what the template sets behind your back

pip install scrapy
scrapy startproject myscraper
cd myscraper
scrapy genspider example example.com

Those four commands work verbatim on 2.17.0. What almost nobody mentions is that startproject does not give you Scrapy's defaults. It writes a settings.py overriding four of them, and the overrides pull against the concurrency Scrapy is famous for.

Read from scrapy/settings/default_settings.py and scrapy/templates/project/module/settings.py.tmpl in the installed 2.17.0 package on 13 August 2026.

A fresh project therefore crawls a domain at one request per second, one at a time. That is a considerate default, nothing like the parallelism the feature list implies, and the answer to a lot of "why is Scrapy slow" questions.

The reverse trap is worse. Run a spider file with scrapy runspider, or drive CrawlerProcess from your own script, and none of the template applies: ROBOTSTXT_OBEY = False, eight concurrent requests per domain, no delay. The polite defaults live in a file you skipped.

robots.txt is not the formality it was. The Robots Exclusion Protocol became a standard in September 2022 as RFC 9309, and on 1 July 2025 Cloudflare changed its own default to blocking AI crawlers unless they pay for the content. Scrapy parses robots.txt with Protego. One RFC detail surprises people: when robots.txt returns 404 a crawler "MAY access any resources on the server", so a missing file means allow-all, and the run logs robotstxt/response_status_count/404 while fetching everything anyway.

4. Fetching a page: Spider and Request

You do not write a request loop. You yield Request objects and the engine schedules them.

import scrapy

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    start_urls = ["https://example.com/catalog"]

    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (compatible; MyBot/1.0)",
        "DOWNLOAD_DELAY": 1.0,          # pause between requests
        "ROBOTSTXT_OBEY": True,         # respect robots.txt
    }

    def parse(self, response):
        # extract product cards
        for card in response.css(".product-card"):
            yield {
                "title": card.css(".title::text").get(),
                "price": card.css(".price::text").get(),
                "url": response.urljoin(card.css("a::attr(href)").get()),
            }

        # follow to the next page
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

That spider runs unmodified on 2.17.0 with no deprecation warnings; we ran it against a local two-page catalogue and it scraped three items. response.follow resolves the relative URL and queues it, so pagination stays two lines. response.follow_all does the same for an iterable.

scrapy crawl catalog -o products.json

The entry point moved. The way to build start requests dynamically used to be start_requests(). It was deprecated in 2.13.0 and removed in 2.14.0 (5 January 2026); Scrapy no longer calls it. The replacement is an async generator:

class CatalogSpider(scrapy.Spider):
    name = "catalog"

    async def start(self):
        for page in range(1, 51):
            yield scrapy.Request(f"https://example.com/catalog?page={page}")

start_urls still works and is still the shortest path, because the default start() reads it for you. If you inherited a spider with start_requests(), it is now dead code and the crawl begins with zero requests. That failure is silent. Callbacks can be async def too, which pays off when a parse step has to await a database lookup or a captcha solver.

5. Parsing content: selectors

Under the hood Scrapy uses parsel (1.11.0 in the current dependency set), which is built on lxml and speaks both CSS and XPath:

# CSS
response.css("h1::text").get()
response.css(".price::text").getall()
response.css("a::attr(href)").getall()

# XPath
response.xpath("//h1/text()").get()
response.xpath('//div[@class="price"]/text()').get()

# regex right inside the selector
response.css(".price::text").re_first(r"\d+")

get() returns the first match or None, getall() returns a list. Missing elements come back as None instead of raising, which is why a spider degrades to empty fields rather than a stack trace when the site is redesigned. That is a mixed blessing: a silent None across 40,000 pages is silent data loss, so assert on the fields you cannot live without and let the spider fail loudly. More on XPath in the lxml tutorial.

When the page is a shell over an API, skip selectors entirely. response.json() parses the body directly, and cleaning the result is covered in Parsing JSON in Python.

Past a handful of fields, describe the record as an Item and populate it through an ItemLoader, so trimming and type coercion live in one place instead of being repeated in every callback. Plain dictionaries are fine for a spider you will read once.

6. Character encoding in Scrapy

Scrapy reads the encoding from the Content-Type header and the <meta charset> declaration, and non-ASCII text usually arrives intact. When an old site serves windows-1252 without saying so, decode the body yourself:

def parse(self, response):
    # force re-read the body in a specific encoding
    text = response.body.decode("windows-1252", errors="replace")
    sel = scrapy.Selector(text=text)

The theory behind these failures is in Web scraping with Python.

The export is a separate problem from the parse. Scrapy's own default for FEED_EXPORT_ENCODING is None, and the JSON exporter then escapes anything non-ASCII. Running the spider above against a page containing "Wîdget B" produced this in products.json:

{"title": "W\u00eedget B", "price": "24.50 EUR"}

Set FEED_EXPORT_ENCODING = "utf-8" and the same run writes Wîdget B. The project template already sets it, with the comment "Set settings whose default value is deprecated to a future-proof value". A spider run outside a project does not, which is why the escapes turn up in throwaway scripts and never in the project everyone tested with.

7. Concurrency, AutoThrottle, and the delay floor

Scrapy runs dozens of requests at once without threads. The knobs:

# settings.py
CONCURRENT_REQUESTS = 16              # total concurrent requests
CONCURRENT_REQUESTS_PER_DOMAIN = 8    # per single domain
DOWNLOAD_DELAY = 0.5                  # base pause
AUTOTHROTTLE_ENABLED = True           # auto-tune the speed
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

AutoThrottle is the part worth understanding, because it interacts with DOWNLOAD_DELAY in a way that quietly caps you. Per Scrapy's AutoThrottle documentation: on each response the target delay is latency / N, where N is AUTOTHROTTLE_TARGET_CONCURRENCY, and the next delay is the average of the current one and that target. Then comes the clamp: the "download delay can't become less than DOWNLOAD_DELAY or greater than AUTOTHROTTLE_MAX_DELAY".

Work the arithmetic on the block above. Target concurrency 4 against a server with 1-second latency wants a 0.25-second delay. The floor is 0.5. So you settle at 0.5 seconds, an effective concurrency of two, and the 4.0 you wrote never happens. Nothing warns you. Enable AutoThrottle inside a generated project and the floor becomes the template's DOWNLOAD_DELAY = 1, capping you at one request per second per domain whatever else you set.

Two more details catch people. Latency "is measured as the time elapsed between establishing the TCP connection and receiving the HTTP headers", so a slow-rendering page that streams its body fast looks fast here. And "latencies of non-200 responses are not allowed to decrease the delay", so a wall of 429s can only slow you down.

The default AUTOTHROTTLE_TARGET_CONCURRENCY is 1.0, not 4. All of this is what a hand-rolled crawler reaches for with asyncio, already wired up.

8. Proxies

The one-line version sets the proxy per request:

yield scrapy.Request(url, meta={"proxy": "http://user:pass@ip:port"})

For a pool, most tutorials reach for the same package:

pip install scrapy-rotating-proxies

# settings.py
ROTATING_PROXY_LIST = [
    "ip1:port",
    "ip2:port",
    "ip3:port",
]
DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

Check the date on that package before you build on it. scrapy-rotating-proxies last shipped 0.6.2 on 25 May 2019, seven years ago, and its PyPI page still classifies it as "3 - Alpha". We installed it alongside Scrapy 2.17.0 and it imports and builds through from_crawler without error. This is not a broken dependency, it is a frozen one. Nothing is moving, and nobody will fix it when a middleware API changes. Fine for a crawl you run twice; for anything you maintain, the middleware you actually need is about forty lines.

Two maintained alternatives. scrapy-zyte-api (0.35.0, June 2026) moves proxying, retries and browser rendering behind a paid API. scrapy-impersonate (1.7.0, May 2026) swaps in a download handler built on curl_cffi, so your TLS and JA3 fingerprint matches a real browser. That matters on targets that block you before your proxy pool gets a chance. General theory is in our guide to proxies for scraping.

9. Scraping through Tor, and the snippet that does not run

This is the correction that matters most, because the snippet below is copied across dozens of tutorials, the earlier version of this one included:

# This does NOT work with Scrapy's standard download handler.
yield scrapy.Request(url, meta={"proxy": "socks5h://127.0.0.1:9050"})

We ran exactly that against a local target on Scrapy 2.17.0. The result:

twisted.web.error.SchemeNotSupported: Unsupported scheme: b'socks5h'

HttpProxyMiddleware only understands http and https proxy schemes, and the Twisted-based downloader underneath has no SOCKS client. The request fails every attempt, three times over if retries are on.

Two routes work. The classic one puts an HTTP-to-SOCKS bridge such as Privoxy in front of Tor, then points Scrapy at the bridge with meta={"proxy": "http://127.0.0.1:8118"}. The second arrived in 2.17.0, whose release notes list HTTP/2 and SOCKS proxy support for the new httpx-based download handler:

pip install "httpx[socks]"

custom_settings = {
    "DOWNLOAD_HANDLERS": {
        "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
        "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    },
}

With that in place the same socks5h:// request went through: we pointed it at a minimal SOCKS5 server on localhost, watched the server log the CONNECT, and the spider logged a 200. Note the underscore in _httpx. Scrapy 2.16.0 made several download handler classes private, so this import path is experimental and can move. Treat it as a way to unblock one job, not as a foundation.

None of this makes Tor a good crawling transport. Exit nodes are few, slow and widely blocklisted. Rotating the exit with stem and the NEWNYM signal is covered in Web scraping with Python; paid rotating proxies win on every axis except price.

10. HTTPS and certificates: the default is the opposite of what you think

Nearly every Scrapy write-up, this one included until now, says the framework verifies TLS certificates and then shows you how to relax that. Both halves are wrong.

In the installed 2.17.0 package, default_settings.py reads DOWNLOAD_VERIFY_CERTIFICATES = False, and the context factory acts on it: verification off means check_hostname = False and verify_mode = ssl.CERT_NONE. We served a self-signed certificate on localhost and pointed a spider at it. Scrapy fetched the page with status 200 and logged a warning:

[scrapy.core.downloader.tls] WARNING: Remote certificate is not valid for hostname
"127.0.0.1"; Certificate does not contain any `subjectAltName`s.

Turn verification on and it behaves the way you assumed it already did:

# settings.py
DOWNLOAD_VERIFY_CERTIFICATES = True

The same request then failed with OpenSSL.SSL.Error: certificate verify failed, and the retry middleware gave up after three attempts. Presence of HTTPS is not proof of identity here. If you are scraping a login flow, posting credentials, or crawling anything where a hijacked endpoint would matter, turn this on deliberately.

The older advice to set DOWNLOADER_CLIENT_TLS_METHOD is worse than useless. Setting it to "TLS" changes nothing, since that is already the default, and 2.17.0 deprecated the setting outright: the code emits "Setting DOWNLOADER_CLIENT_TLS_METHOD to a non-default value is deprecated, please use DOWNLOAD_TLS_MIN_VERSION and/or DOWNLOAD_TLS_MAX_VERSION instead." The BrowserLikeContextFactory recipe alongside it is deprecated too, its warning pointing at DOWNLOAD_VERIFY_CERTIFICATES. Connection security in general is covered in Web scraping with Python.

11. Working with cookies and logging in

Cookies are on by default (COOKIES_ENABLED = True) and the engine keeps a session across requests. To set one by hand:

yield scrapy.Request(url, cookies={"sessionid": "abc123"})

For authentication, FormRequest.from_response reads the form out of the page and fills in what you did not supply:

def parse(self, response):
    return scrapy.FormRequest.from_response(
        response,
        formdata={"username": "user", "password": "pass"},
        callback=self.after_login,
    )

Hidden fields come along automatically, CSRF tokens included, which removes the classic login headache of a token that changes on every page load.

One session is the default, and that is a trap at scale. All requests share a single cookie jar unless you say otherwise, so 16 concurrent requests behind one logged-in account look like one impossibly fast user. Give each identity its own jar with meta={"cookiejar": worker_id}, and keep in mind that a jar is only coherent if it keeps the same exit IP.

12. Response status, headers and retries

def parse(self, response):
    print(response.status)              # 200, 404 ...
    print(response.headers.get("Content-Type"))

Scrapy passes 2xx to your callback and HttpErrorMiddleware drops the rest before you see it. To inspect an error page, allow it with HTTPERROR_ALLOWED_CODES = [404, 410], or per request with meta={"handle_httpstatus_list": [404]}. Reading a 404 body sounds pointless until the site starts serving soft 404s with a 200 status. Retries are handled by RetryMiddleware:

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 403]

Three notes on that block, all checked against the 2.17.0 source and docs.

RETRY_TIMES defaults to 2, not 3, and the comment in the source spells out what that means: "initial response + 2 retries = 3 requests". Setting it to 3 gives four attempts per URL, a 33% increase in traffic against a failing host.

429 is already in the defaults. The stock RETRY_HTTP_CODES is [500, 502, 503, 504, 522, 524, 408, 429]. Adding 429 changes nothing. Adding 403 does change something, usually for the worse: a 403 is a decision about your identity, not a transient fault, so retrying it triples the evidence against the IP you are about to lose. Rotate the proxy or fix the fingerprint instead.

Scrapy does not honour Retry-After. The earlier version of this article said it could. It cannot: grep the whole installed package for the header and you get zero matches, and the downloader middleware documentation never mentions it. Scrapy retries a 429 on its own schedule with a priority penalty, ignoring the interval the server asked for. Against a rate-limited API that matters, so write a middleware that reads the header in process_response and defers the retry, or let AutoThrottle absorb the pressure. Status codes in general are covered in Web scraping with Python.

13. Queues and deduplication: what actually breaks at a million URLs

Everything above behaves the same at 100 URLs and at 100 million. The scheduler is where the difference lives. It holds a priority queue of pending requests; RFPDupeFilter drops requests whose fingerprint it has already seen; and JOBDIR moves both to disk so an interrupted crawl resumes.

scrapy crawl catalog -s JOBDIR=crawls/catalog-1

Both structures live in RAM, and their cost is measurable. RFPDupeFilter keeps a Python set of hex-encoded SHA-1 fingerprints, 40 characters each. We built a set of one million such fingerprints under tracemalloc and it occupied 116.9 MiB, about 123 bytes per URL. Pending requests are heavier: 200,000 scrapy.Request objects measured 88.9 MiB, roughly 466 bytes each, putting a million queued requests near 444 MiB. Measured on one Linux machine with Scrapy 2.17.0 and CPython 3.11.15; your absolute numbers will differ with URL length and interpreter version, the ratio will not.

Add those and a breadth-first crawl that discovers ten million URLs before finishing the first million is not slow, it is dead. The fix is ordering, not hardware: use DEPTH_PRIORITY = 1 where you actually need breadth, cap DEPTH_LIMIT, and let JOBDIR push the queue to disk.

JOBDIR has sharp edges the jobs documentation states outright. The directory "must not be shared by different spiders, or even different jobs of the same spider". Requests "must be serializable with pickle", so a lambda or closure callback will not survive a pause. Resuming is "only supported when the spider is paused by stopping it cleanly", and a kill -9 can corrupt the queue. Then the practical one: "Cookies may expire", so a job paused on Friday and resumed on Monday wakes up logged out. On disk the dupefilter is plain text, requests.seen, one fingerprint per line, about 39 MiB per million URLs.

For crawling across machines, scrapy-redis puts the queue and dupefilter in Redis so several workers share one frontier. It sits at 0.9.1, released 6 July 2024, and it still runs on 2.17.0: we drove a RedisSpider against a local target with a stubbed Redis and it pulled its start URL and parsed the page. Its start_requests() is dead code under Scrapy 2.14 and later; the URLs now arrive through the spider_idle signal instead.

14. Item Pipeline: storing data, and the export flag that bites

Items pass through the pipeline, where they are validated, cleaned and saved:

# pipelines.py
import pymongo

class MongoPipeline:
    def open_spider(self, spider):
        self.client = pymongo.MongoClient("mongodb://localhost:27017")
        self.db = self.client["scraping"]

    def process_item(self, item, spider):
        self.db["products"].update_one(
            {"url": item["url"]}, {"$set": dict(item)}, upsert=True
        )
        return item

    def close_spider(self, spider):
        self.client.close()

# settings.py
ITEM_PIPELINES = {"myscraper.pipelines.MongoPipeline": 300}

The upsert=True is doing real work: it makes the crawl idempotent, so a resumed job overwrites rather than duplicates. Read the connection string from settings via from_crawler instead of hard-coding it, and raise DropItem on records that fail validation so bad rows never reach storage.

For a plain export you need no pipeline at all, and this is where a five-second mistake costs an afternoon:

scrapy crawl catalog -o products.json     # APPENDS to the file
scrapy crawl catalog -O products.json     # overwrites it

Lowercase -o is documented as "append scraped items to the end of FILE". Run it twice into a JSON file and you get two arrays concatenated, ][ sitting in the middle of the document. We did exactly that, and json.load reported Extra data: line 5 column 2. Your file looks plausible in an editor and fails in the loader. Use -O for a fresh dump, or export to .jsonl, where appending is legal and each line stands alone.

15. Where Scrapy stops being the right tool

JavaScript-rendered pages. Scrapy fetches HTML; it does not run scripts. The standard advice names two add-ons and only one is still alive. Splash last released 3.5 on 16 June 2020, and its README still carries Travis CI badges and a Qt 5.14 pin. Do not start a 2026 project on it. scrapy-playwright is the maintained answer, at 0.0.48 as of 10 July 2026, swapping in a download handler that drives a real browser. Budget for it: rendering is the order-of-magnitude multiplier from the first section, and the project warns that mismanaged browser contexts can "block the whole crawl". Before you render anything, open the network tab and look for the JSON endpoint the page is calling.

Sites that block on fingerprint rather than volume. When the block lands on the TLS handshake, more proxies will not help, because Scrapy's handshake does not look like Chrome's. That is the gap scrapy-impersonate and the commercial unblocking APIs fill.

Teams that live in asyncio. Scrapy has a serious competitor it did not have when this article was first written. Crawlee for Python, from Apify, reached 1.0.0 on 29 September 2025 and is at 1.9.x in August 2026. It is asyncio-native rather than Twisted-derived, autoscales concurrency against available system resources, and uses one API for HTTP and headless crawling. Scrapy still wins on ecosystem and on the number of edge cases already solved inside its middlewares; Zyte maintains it, with over 500 contributors behind it.

16. Pros and cons of Scrapy

What you get:

• Queue, deduplication, retries, redirects, cookies and concurrency, all present before you write a line. Rebuilding that is a month you do not get back, and your version will not have the edge cases.
• Resumable jobs through JOBDIR, a distributed frontier through scrapy-redis, and export to JSON, JSON Lines, CSV or XML from a command-line flag.
• A middleware system where a proxy policy, a header policy and a ban detector are three small classes you can test in isolation.

What it costs:

• A real learning curve. You are learning an architecture, not an API, and you will meet Twisted eventually.
• Overkill below a few hundred pages, and no JavaScript execution without an add-on and a browser.
• Defaults that are not what the feature summaries imply: TLS unverified, no Retry-After, no SOCKS on the standard handler, and a project template that throttles you to one request per second.

The last line here used to say Twisted "feels unfamiliar next to modern asyncio, though recent Scrapy runs on an asyncio reactor". That was understated. The asyncio reactor has been the default since 2.13.0, async def callbacks are normal, and 2.15.0 added an experimental mode that installs no reactor at all.

For static content on a stack you already own, scraping inside Django is sometimes less machinery than a separate project. At the other end, once a crawl needs proxy rotation, ban detection, monitoring and a delivery format someone else depends on, the work stops being about Scrapy and becomes an operations problem, which is where a managed extraction service starts to look like the cheaper column in the spreadsheet.

Read the defaults before you read the tutorials. This one included.