Web Crawling vs Web Scraping vs Parsing: Key Differences

Ask three developers what they are building and all three might answer "a parser." The first is writing something that follows links across an entire catalogue. The second is writing a price extractor for twelve URLs that already sit in a spreadsheet. The third is turning a JSON file on disk into database rows and never touches the network at all. One word covers all three in conversation, which costs nothing over coffee and quite a lot in a project spec.

Three numbers from one machine on 10 August 2026. Fetching a single 144 KB product page across the public internet took a median of 48 milliseconds. Parsing that same page and running one CSS selector over it took 2.5 milliseconds with lxml and 97 milliseconds with the parser that ships in Python's standard library. Deciding which page to fetch next took no time at all, because nothing in that test was crawling. Three jobs, three cost curves, three separate ways to end the night with an empty file.

If you want it in one line each:

• Crawling finds pages. It follows links and decides what to visit next.
• Scraping fetches a page and takes the fields you want off it.
• Parsing breaks down text or markup into a structure. It downloads nothing at all.

Only one of the three is governed by robots.txt. Only one of them gets harder as the job gets bigger. Only one of them is defined line by line in a published standard, and it is not the one most people worry about.

Why the distinction pays for itself

Permission attaches to fetching. The Robots Exclusion Protocol became a Standards Track document, RFC 9309, in September 2022, after roughly twenty-eight years as a convention. It standardises three fields, user-agent, allow and disallow, requires crawlers to parse at least 500 kibibytes of the file, and says a cached copy should not be used for more than 24 hours. What it governs is which URLs a client may request. It says nothing about what you do with bytes you already hold, and nothing about a file on your own disk. Parsing sits entirely outside its scope. So does most of what people mean when they say a site "blocked their parser."

The bill attaches to fetching too. At 48 milliseconds per page and one connection, a million pages is thirteen hours of pure network time before a single field gets extracted. Parsing those same million pages with a fast library is about forty minutes of CPU, and that work parallelises across cores on hardware you already own. Getting the two confused is how a project ends up optimising the cheap half.

The failure modes look identical from outside. An empty output file means the crawler found no links, or the fetch returned a challenge page instead of the product, or the selector matched nothing because the parser built a different tree than your browser did. All three produce zero rows and no exception. Naming the stages is the difference between a debugging session and a guessing session.

Parsing

Parsing is the act of taking data apart according to defined rules and turning it from a "raw" form into a structured one.

The word comes from computer science, where a parser reads a sequence of characters (program source, JSON, XML, HTML) and builds a machine-friendly structure out of it: a tree, an object, a set of fields.

A simple example. We have a string of HTML:

<div class="price">$1,299</div>

The parser turns this into a tree of elements, a DOM, that you can walk and query: give me the contents of the element with class price. What comes back is no longer a run of characters but a value, $1,299, from which you can pull the number 1299.

Parsing on its own downloads nothing. It works with text you already have. You can parse a local file, an API response, or a string somebody pasted into a chat. Parsing is about structure, not source. That single property is what separates it from the other two words, and it is the property that gets lost first in casual speech.

What the earlier version of this article got wrong

This page used to say that a parser "has to guess at the structure somehow" when the markup is broken. That was true in 2005 and it is wrong now, and the correction matters because it moves the blame.

Since HTML5, error recovery is not a matter of taste. The WHATWG HTML Standard specifies the parsing algorithm as a state machine with 84 tokenizer states and 21 insertion modes, and it defines what happens to invalid input at every one of them. In the spec's own words, "the error handling for parse errors is well-defined (that's the processing rules described throughout this specification)." Two conforming parsers handed the same broken page must build the same tree. There is no guessing left in the algorithm.

What varies is whether the library you imported implements that algorithm. Many of the ones in daily use predate it or deliberately skip it for speed, and that is where the surprises actually come from.

Three parsers, one tag, three answers

Nine characters of misnested markup are enough to demonstrate it. Run this with Python 3.11, beautifulsoup4 4.14.3, lxml 6.1.0 and html5lib 1.1:

from bs4 import BeautifulSoup

markup = "<b>bold<p>para</b>tail</p>"
for parser in ("html.parser", "lxml", "html5lib"):
    soup = BeautifulSoup(markup, parser)
    print(parser, [b.get_text() for b in soup.find_all("b")])

html.parser ['boldpara']
lxml ['bold']
html5lib ['bold', 'para']

One bold element containing both words, one bold element containing the first word, or two separate bold elements. The same find_all("b") returns a different number of elements with different text depending on a keyword argument that most tutorials never mention. The html5lib answer is the one your browser gives, because html5lib implements the spec algorithm above, including the adoption agency procedure that re-opens <b> inside the new paragraph. The Beautiful Soup documentation states it plainly: html5lib "parses pages the same way a web browser does."

Unclosed list items split the same way:

BeautifulSoup("<ul><li>a<li>b</ul>", parser).select("ul > li")

lxml and html5lib both return two items, a and b. html.parser nests the second <li> inside the first and returns one item whose text is ab. If your inventory count comes from len(soup.select("ul > li")), your inventory is now wrong by a factor that depends on your import line.

The Beautiful Soup documentation has documented this divergence for years with its own example, <a></p>, which lxml renders as an empty <a>, html5lib as <a><p></p></a>, and html.parser as a bare <a></a>. Nobody is broken. They implement different rules.

Pick your parser once, on purpose, and write it down. Changing it later silently changes your data.

Scraping

Scraping (web scraping) is the extraction of specific data from web pages.

A scraper does two things:

1. Fetches the page, meaning it sends an HTTP request and gets a response body back.
2. Pulls out the fields you want: prices, titles, contacts, reviews.

On that second step the scraper uses parsing as a tool. To extract a price from a page, you first have to parse the page. Parsing is a stage inside scraping, not an alternative to it.

Scraping usually targets known data on known pages. The classic jobs: collect competitors' prices, export a product catalogue, gather listings, monitor a news feed. The list of URLs may come from a sitemap, a database, a client's spreadsheet, or a crawler. Where it comes from is a separate question, which is the whole point of the third word.

The fetch step is also where you stop being anonymous, and it produces a failure that no parser can help you with. While measuring the timings quoted at the top of this article, we sent seven requests in quick succession to one page on pypi.org, the same page used for the benchmark below. The next request returned HTTP 200, 3,038 bytes, and this title:

<title>Client Challenge</title>

The body held a <noscript> block reading "JavaScript is disabled in your browser" and a loader for two scripts. A polite single request with a descriptive user agent thirty seconds later got the same 3 KB challenge. Nothing errored. response.status_code was 200, len(response.text) was a plausible number, the parser parsed it happily, and every selector returned an empty list. A scraper that checks only the status code writes an empty row and moves on, and the log looks clean.

Check the shape of what came back, not just the status. A page that suddenly weighs 2% of its usual size is a challenge, a login wall, or an error page dressed as a success.

Crawling

Crawling is the systematic traversal of pages by following links.

A crawler (also called a spider or a bot) starts from one or several seed URLs, extracts every link it finds, queues them, follows them, extracts more links, and repeats. Its goal is not one field on one page but coverage: discover and traverse as many relevant pages as possible, and know when to stop.

Search engines are the best-known operators. Google now documents its fleet in three groups: common crawlers, special-case crawlers and user-triggered fetchers. Of the first group it says the crawlers "always obey robots.txt rules when crawling automatically." The split exists because the groups behave differently. Write a rule that assumes one Googlebot and you have written it for a third of the fleet.

Scale is the part that is hard to picture. Common Crawl's July 2026 archive, CC-MAIN-2026-30, holds 2.14 billion pages collected between 7 and 25 July, 364.01 TiB of uncompressed content, and 603 million URLs never seen in any previous crawl. That is roughly 1,300 pages a second sustained for nineteen days. The 603 million is the figure to sit with: more than a quarter of what a monthly crawl turns up has never been seen before.

Most projects never need discovery on that scale, because the site tells you where its pages are. The sitemaps protocol caps one file at 50,000 URLs and 50 MB uncompressed, with sitemap index files carrying the same limits, so a two-million-page catalogue arrives as forty files and an index. Reading those is not crawling. It is a download followed by parsing, and it is faster, kinder and less likely to get you blocked than walking the link graph. Try the sitemap first.

In real projects the three combine: a crawler walks a catalogue page by page, a scraper handles each product page it turns up, and inside the scraper a parser converts markup into fields.

How it all fits together

The easiest way to picture it is as a pipeline: crawling finds the pages, scraping fetches them and lifts out the fields, parsing turns the markup into a tree the extraction step can query.

• Crawling answers "which pages do we process?"
• Scraping answers "what do we take from those pages?"
• Parsing answers "how do we turn markup into data?"

You can parse without scraping: break down a file you already have. You can scrape without crawling: the list of pages is already known, from a sitemap or a database. Crawling without scraping is rarer and does exist, since a link-graph audit or a broken-link check needs discovery and nothing else, but most crawls end in an extraction step.

There is a fourth stage that picture leaves out, and it is the one that decides whether the project survives contact with next month. Something has to decide what to re-visit and when. Prices move hourly, catalogues move daily, company addresses move once a decade. A pipeline that re-crawls everything at the same interval is either wasting most of its requests or serving stale data, and usually both at once for different parts of the same site.

Terminology: why "parsing," "scraping," and "crawling" get mixed up

Here is where most of the confusion lives.

In careful usage the split is clean: collecting data from websites is web scraping, while parsing is specifically the technical business of breaking down markup. In everyday developer talk the lines blur. People say "I'll just parse that site," "let me write a parser for Amazon," or "we're crawling the marketplace," and they almost always mean the same broad thing. You will also hear "screen scraping," "web harvesting" and "data mining" thrown at the same task. The last one means something else entirely in its home field. Data mining operates on data you already hold. It is the step after everything discussed here.

Why does it happen? These words entered common use through forums, job posts and blog articles, where precision matters less than being understood. The narrow technical term ended up standing in for the whole job. "Parser" in particular sounds concrete and familiar, so it stuck to the entire task.

There is a second reason, and it is a genuine mismatch. In compiler theory a parser is defined against a grammar, and the grammar is the specification. HTML has no such grammar. Its parsing rules are the state machine described earlier, 84 tokenizer states of accumulated browser behaviour, written down after the fact so that everyone would break identically. Calling that component a "parser" is accurate but slightly misleading, since it does far more than a textbook parser: it repairs, re-opens and relocates elements as it goes.

Strictly speaking, "parsing a site" is not quite right. Parsing is only one stage of scraping, the breakdown of markup already downloaded. When someone says they are parsing a site, what they are actually doing is:

1. finding the pages (crawling, or reading a sitemap),
2. fetching them and pulling out the wanted values (scraping),
3. and only inside that second step actually parsing the HTML.

The vendors have quietly settled on the strict definitions, which is a decent argument for using them. Firecrawl exposes scrape as a single-URL operation and crawl as a separate one that "recursively gather[s] content from entire sites." Crawlee describes itself as a library that "handles blocking, crawling, proxies, and browsers for you." Scrapy, tagline "the world's most-used open source data extraction framework," has kept crawling and item extraction as separate concepts for its entire history. If you write "parse the catalogue" into a spec and hand it to any of these ecosystems, somebody has to guess which half of the job you meant, and the guess is billed either way.

None of this makes casual usage wrong. Language does what it does, and everyone in the room will understand you. In precise documentation, a project brief or a tool comparison, the distinction earns its keep: parsing is not scraping, and crawling is the discovery step, not the extraction step.

What each step actually costs

Descriptions are cheap. Here are measurements.

We fetched https://pypi.org/project/scrapy/ seven times over one keep-alive session and timed each stage separately on 10 August 2026, on a single Linux container with Python 3.11.15, requests 2.33.1, lxml 6.1.0, beautifulsoup4 4.14.3, html5lib 1.1 and selectolax 0.4.11. The page was 144,306 bytes on the wire. Parse timings are the median of repeated runs over the same in-memory string, each including one CSS query for h1, so the numbers cover tree construction plus a trivial selection.

Two things in that table are worth arguing with.

The tree wrapper costs more than the parse. Beautiful Soup on top of lxml takes 52.7 ms where bare lxml takes 2.5 ms. The actual parsing is identical; the difference is building a Python object for every node. Beautiful Soup is a pleasant API and worth paying for on a hundred pages. On a hundred thousand it turns four minutes of parsing into eighty-eight.

Parsing stops being free before the network does. The common advice is that the network dominates and parser choice does not matter. On this page it is the other way round: the default html.parser costs twice the fetch. The gap between the fastest and slowest options here is about 95 ms per page, which is sixteen minutes across ten thousand pages and two hours forty minutes across a hundred thousand.

The selectolax project publishes its own benchmark across 754 top domains: 2.39 seconds for lexbor against 9.09 for lxml and 61.02 for Beautiful Soup. Our ordering of the top two is reversed, most likely because one 144 KB page with one selector is a different workload than 754 varied documents. Both runs agree where it counts: the Python-object layer costs an order of magnitude, the C parsers all land in single-digit milliseconds, and html5lib's spec conformance is paid for at roughly 40× the time.

Measured on one machine, on one 144,306-byte page, over a warm keep-alive connection to a CDN-hosted host with no proxy in the path. Add a proxy, a TLS handshake per request, and a slower origin and the fetch column grows. The parse column does not. Your numbers will differ; the shape will not.

Problems with parsing

Parsing looks easy right up until the markup gets messy. In practice, these get in the way.

• Broken HTML. Browsers render pages with unclosed tags and misnested elements without complaining, and so does every serious parser. The risk is not that parsing fails; it is that it silently succeeds differently, as the <b>bold<p>para</b>tail</p> example above shows. If you need the tree your browser shows in DevTools, use a spec-conforming parser and accept the cost.
• Shifting structure. You wrote the rule "take the price from the element with class price," and a month later the site renames the class. Selectors tied to presentational markup are the most brittle thing in the pipeline, and build-time hashed class names have made them worse. Anchor on things the site cannot change without breaking itself: JSON-LD blocks, ARIA roles, visible label text, the JSON payload the page's own front end already fetches.
• Dynamic content. Many sites load data after the page loads, via JavaScript. The values are simply absent from the initial HTML, so there is nothing to parse until the page is rendered in a real browser. Before reaching for one, open the network tab: if a JSON endpoint is populating the page, calling that endpoint directly is faster, more stable and easier to parse than any DOM.
• Encodings. A misdetected character encoding turns text into mojibake, and it does it quietly, one field at a time. The WHATWG Encoding Standard is worth knowing here for one non-obvious rule: the labels iso-8859-1, latin1, ascii and us-ascii are all mapped to windows-1252, because that is what browsers have always done. A parser that honours a declared iso-8859-1 literally will disagree with every browser on bytes 0x80 to 0x9F, which is exactly where curly quotes, dashes and the euro sign live. The standard's instruction to authors is blunt: "Authors must use the UTF-8 encoding."
• Ambiguity. Sometimes data with different meaning is marked up identically. Two <span> elements, one holding a list price and one a sale price, distinguished only by their order on the page. No parser resolves that. Rules do.

Problems with scraping

The parsing problems all still apply, plus the complications of reaching out over the network to a machine that owes you nothing.

• Anti-bot defences. Sites use CAPTCHAs, rate limits, address blocking, TLS fingerprinting and behavioural scoring. An over-eager scraper gets throttled quickly, and often invisibly, as the Client Challenge above shows. Getting through means rotating proxies and solving CAPTCHAs, both of which turn a free operation into a metered one.
• JavaScript sites. If scripts build the content, a plain request is not enough and you need a headless browser that actually runs them. Budget for it honestly: you are now paying for a full page load with all its subresources, plus a browser process per worker, in place of one HTTP request.
• Brittleness and maintenance. The site changes its layout and the scraper breaks. Every scraper needs upkeep, and the only real question is whether you find out from your monitoring or from whoever consumes the data.
• Silent breakage beats loud breakage. A scraper that crashes gets fixed the same day. A scraper that returns None for one of nine fields runs for four months. Assert on shape, not just on success: expected row counts, non-null rates per field, price ranges that make sense.
• Honeypots. Some pages carry links or form fields invisible to humans. A person never sees them; a bot follows or fills them and identifies itself in the process.
• Legal and ethical questions. Terms of service, copyright, personal data, and the load you place on someone else's servers. What is technically possible is far from always permissible, and the case law is less settled than the confident summaries suggest. The most-cited example is the clearest warning. hiQ Labs is routinely described as having beaten LinkedIn. hiQ lost on the merits in 2022, accepted a $500,000 judgment, and agreed to a permanent injunction barring it from scraping LinkedIn at all. Check your own situation rather than inheriting a slogan.
• robots.txt. The file where a site declares what its paths permit. Since RFC 9309 it is a standard rather than a courtesy, and Google enforces a 500 KiB parse limit and caches for up to 24 hours. Two things it does not do: it does not control indexing, and it does not license anything. A page you were allowed to fetch is not a page you were granted rights to republish.

Problems with crawling

Crawling has its own difficulties, and they exist precisely because the bot decides for itself which links to follow while the scale grows underneath it.

Loops and "crawler traps"

This is the most characteristic problem of crawling.

• Link cycles. Page A links to B, B links to C, C links back to A. A crawler with no memory loops forever. The basic fix is a set of already-visited URLs, checked before every queue insert.
• Infinite traps. Worse than a cycle is a site that generates infinitely many new URLs. A calendar widget with a "next month" link leads politely into the year 2099. Filter and sort parameters multiply: ?sort=price&color=red&page=2 is a new URL for every combination, and the combinations are astronomically many for a shop with a dozen facets. Google's URL Parameters tool, which used to let site owners declare which parameters to ignore, was retired in March 2022, so nobody is filtering these for you.
• Session IDs in the URL. If the site puts a unique session identifier into the path or query, every page looks new on every visit, and a visited-set does not help, because the URLs genuinely are different.
• Duplicate pages. The same content reachable with and without a trailing slash, with and without www, with parameters in a different order, over both HTTP and HTTPS. Without normalisation the crawler pays repeatedly for one document. Google's own crawl budget guidance puts it in one line: "eliminate duplicate content to focus crawling on unique content rather than unique URLs."

The fixes: URL normalisation, depth limits, a cap on pages per domain, cutting off sections that visibly multiply, and recognising traps by shape. Normalisation has a specification, RFC 3986 section 6, covering case normalisation of scheme and host, percent-encoding normalisation and removal of dot-segments. It deliberately stops short of the site-specific part, because only you know whether ?utm_source= changes the page. Scrapy handles this layer with a request fingerprint, scrapy.dupefilters.RFPDupeFilter, which hashes method, canonicalised URL and body rather than comparing strings.

Scale and politeness

• Volume. The web is large and so is one big retailer. Prioritisation is the real work: which section first, how deep, how much per domain. Google's own crawl budget documentation says the question only becomes pressing for sites above roughly a million pages that change weekly, or above ten thousand pages that change daily. Below that, the crawler is rarely your bottleneck.
• Server load. Requests that are too frequent degrade or take down somebody's site. This is the part where politeness is not decoration. crawl-delay never made it into RFC 9309 and Google states plainly that it does not support the directive, so the limit has to live in your own configuration. Scrapy ships defaults on that side of the argument: CONCURRENT_REQUESTS_PER_DOMAIN of 1 and DOWNLOAD_DELAY of 1 second in current versions, with the older values of 8 and 0 kept only as fallbacks for existing projects, alongside ROBOTSTXT_OBEY defaulting to true.
• Data freshness. Pages change, and re-crawl scheduling is a whole subsystem once the catalogue is large. Cheap signals help: Last-Modified and ETag with conditional requests, sitemap lastmod values where the site maintains them honestly, and a per-section change rate you measure rather than assume.
• Invisible pages. Content behind forms, behind logins, or reachable only by search is not reachable by following links. A crawler will not find it, and a login is the point at which the legal picture changes materially rather than gradually.

What breaks when the same script meets ten thousand pages

Every part of this works on a laptop with fifty pages. The following start failing somewhere between one thousand and one hundred thousand, and they fail in an order that surprises people.

The frontier stops fitting in memory. A Python set of visited URLs costs about 161 bytes per 70-character URL on CPython 3.11, measured the same afternoon as the timings above. That is 0.2 GB at a million URLs and 8 GB at fifty million, for a structure whose only job is answering "seen it?". That is when crawlers move to on-disk queues, database-backed frontiers or Bloom filters, and each of those brings its own wrong answer: a Bloom filter can claim you have already seen a URL you have not.

Retries become a data-quality problem, not a networking one. At scale a fixed percentage of requests will fail. If the failures are randomly distributed you lose coverage evenly; if they correlate with a slow section of the site, you have quietly under-sampled exactly the pages that were interesting.

One wasted second per page turns into three hours per ten thousand. This is why the parser table above matters, and it is also why an unconditional re-fetch of unchanged pages is expensive rather than merely inelegant.

Schema drift outruns your attention. With one site you notice a layout change immediately. With four hundred sites you notice it when someone asks why the March numbers look odd. The fix is monitoring on the output: null rates per field, row counts per source, value distributions, all compared against last week.

The politeness budget becomes a real constraint. One request per second per domain is a hard ceiling of 86,400 pages per day per site, whatever your hardware is doing. Crawls get planned around that number, or they get planned around a proxy bill.

Teams usually meet this boundary twice. Once when they build the pipeline, and once when they decide that maintaining scrapers for four hundred sites is not their product and hand it to a managed extraction service instead. Both are reasonable. Confusing the two decisions is what produces a half-maintained crawler with nobody's name on it.

Where the three words stop being useful

The distinction is a tool, and tools have edges.

When the data arrives as JSON, parsing gets trivial and the interesting work moves. A site whose front end calls its own API hands you structured values directly. There is no markup to break down, and json.loads is not what anyone means by a parser problem. The work becomes authentication, pagination and rate limits, all of which are fetch-side concerns.

A headless browser collapses three stages into one call. page.goto() fetches, executes, renders and exposes a live DOM. Fetching and parsing stop being separable in your code even though they are still separable in your bill, and the bill is what tells you which one to optimise.

LLM-based extractors move the boundary again. Tools that return model-ready markdown or a filled JSON schema from a URL do crawl, fetch and extract in one product, and the selector step leaves your codebase without leaving the world. What survives is the vocabulary: those products still ship scrape and crawl as separate endpoints, because the two remain separate operations with separate costs.

Nobody has renamed the failure modes. Whatever the tool, the questions hold. Did we find the page, did we get the page, did we read it correctly. An afternoon script and four hundred sites answer the same three, at different volume.

Quick recap

Which of the three you need decides the shape of the work, and the shapes are not degrees of each other. A known list of URLs and a stable page is a scraper, and it is an afternoon. A catalogue you have to discover, at a size where the frontier stops fitting in memory, is a crawler, and it is an operation with a re-crawl schedule and a politeness budget. A file already sitting on your disk is a parser, and nothing about proxies, robots.txt or crawl budgets touches you at all. Buying the finished output, as a one-off data extraction export or a running data-as-a-service feed, is the same three-way decision made about somebody else's pipeline.

Three words. Three bills.