Web Crawling vs Web Scraping in E-commerce

Open the robots.txt of a large retailer and count the lines that begin with Sitemap:. Walmart publishes thirty-nine of them. The same file disallows /search. The store is handing automated clients a machine-readable inventory of the pages it wants found, while shutting the corridor most people would have walked down to find them.

That one file holds the argument of this article. Finding pages and taking data off them are separate jobs, with different costs, different failure modes and different answers from the site you point them at. Collapsing both into one word is how a project ends up with a directory of links and no prices in it.

Everything quoted below was read from the source on 10 August 2026: four retailer robots.txt files, the sitemap and robots specifications, Google's crawling documentation, three platform APIs, five price lists. Numbers move, and the file you fetch tomorrow is the one that counts.

The one-sentence difference

• Crawling finds pages. A crawler follows links to discover which URLs exist.
• Scraping takes data. A scraper opens a known page and extracts the fields you defined.

In store terms: crawling walks the aisles and writes down where everything is; scraping opens each product page and lifts the price, title and stock line. Parsing is a third verb hiding inside the second, and it has its own treatment in web crawling vs web scraping vs parsing.

What a web crawler does

A crawler starts from seed URLs, reads the links on each page, adds the new ones to a queue, and repeats. That queue has a name in the literature, the frontier, and managing it is most of the engineering. Its goal is not a field. Its goal is coverage: the URLs themselves, the link graph, which page points at which.

Bookkeeping is the whole trick. A crawler that does not remember what it fetched is an infinite loop with a user-agent string. Real ones normalise URLs before comparing them, strip tracking parameters, and keep a seen-set that survives a restart.

Scale arrives faster than people expect, and faceted navigation is why. Google's guidance on faceted navigation says crawlers "typically access a very large number of faceted navigation URLs", because there is no way to tell from outside whether a filter combination is useful without fetching it. Take one category with five filters, four values each, and two sort orders. That is 1,024 filter combinations doubled by sort order, so 2,048 addressable variants of one page. Two hundred categories gives 409,600 URLs resolving to the same few thousand products. Google draws its crawl-budget line at one million pages changing weekly, or ten thousand changing daily. A mid-size catalogue with unrestrained filters clears that bar without owning a million products.

E-commerce crawling jobs:

• Walk a marketplace category page by page to enumerate every product URL, following pagination to the end rather than stopping at the first screen.
• Read the store's sitemap.xml instead of brute-forcing links, which is the sanctioned door and usually the fastest (see crawling a sitemap).
• Diff today's URL list against yesterday's to catch new and delisted products.
• Map internal linking, or find pages nothing links to.

What a web scraper does

A scraper is surgical. It goes to a known URL and collects the fields you named. Two steps, always: it downloads the page, then extracts values from what came back.

On a modern retail page, the second step often should not touch the HTML. Product pages carry structured data because retailers want the rich result in search, and Google's merchant listing documentation sets out what goes in it: name, image and a nested offers, plus price and priceCurrency in three-letter ISO 4217. Recommended alongside are availability, sku, gtin, mpn and priceValidUntil. A JSON-LD block a merchandising team maintains for Google is a steadier target than a CSS class a build step renames.

The vocabulary has edges. schema.org lists twelve ItemAvailability values; Google accepts ten, leaving out MadeToOrder and Reserved. PreOrder, BackOrder, LimitedAvailability, InStoreOnly and Discontinued are all things a page can legitimately say, and a pipeline with a boolean in_stock column has collapsed twelve states into two.

Price has its own trap, and schema.org spells it out: use priceCurrency rather than "ambiguous symbols such as '$'", use "'.' (Unicode 'FULL STOP' (U+002E)) rather than ','" for the decimal point, and neither as a readability separator. That is why the visible price and the marked-up price disagree on European pages. The customer sees 1.299,00 €, the JSON-LD says 1299.00, and a parser that treats the dot as a decimal point books 1,299 euros as 1.30.

From one product page a scraper can lift the name and canonical URL, description, brand, category, images, ratings and review counts, seller and shipping terms, the SKU or GTIN or ASIN, and the two fields the exercise exists for: price and stock.

One correction to the earlier version of this guide. It closed that list with "anything publicly rendered on the page is fair game". That is wrong twice over. Review text carries personal data and falls under the GDPR whether or not the page is public. Product photography and long descriptions are somebody's copyright, publicly displayed and still owned. Public means reachable, not unencumbered.

Web crawler vs web scraper: side by side

What four retailers actually allow

Advice about crawling tends to be written in the abstract. These files are public, so here is what they said on 10 August 2026.

Walmart disallows /search and leaves item pages under /ip/ open, with thirty-nine Sitemap: lines and a single Crawl-delay: 5 aimed only at Yahoo's Slurp. Etsy blocks /search?*q= across every locale it serves, allows /listing/, and publishes no Sitemap: line at all. eBay disallows /sch/ with narrow exceptions for advanced search and the category index, lists seven sitemaps, and has no blanket rule against /itm/. Its nineteen /itm/ rules are all suffixed, /itm/*_nkw and /itm/*_pgn= among them, so item pages are crawlable and their parameterised variants are not. eBay repeats that rule set verbatim for bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User.

Gymshark runs Shopify's default file, which is the most instructive of the four because thousands of stores ship it unchanged:

Disallow: /search
Disallow: /collections/*sort_by*
Disallow: /collections/*/products*
Crawl-delay: 10        # AhrefsBot, AhrefsSiteAudit, MJ12bot
Crawl-delay: 1         # Pinterest
Sitemap: https://www.gymshark.com/sitemap.xml

That third line is the useful one. Shopify serves the same product at /products/{handle} and at /collections/{collection}/products/{handle}, and the default file blocks the second form. The store tells you which of the two it treats as the duplicate, before you have deduplicated anything.

The pattern across all four runs against the intuition most guides start from. The paths a crawler wants are the restricted ones. The paths a scraper wants are open. Search results, sorted collections and filtered listings get blocked, because they cost the site money to render and produce nothing it wants indexed. Product pages stay open, because the store wants them in search. Discovery by link-following is what sites push back on. Discovery by sitemap is what they pay a CDN to serve you.

Two details about the file get misread routinely.

Crawl-delay is not a general speed limit. Google states that "other fields such as crawl-delay aren't supported" in its robots.txt specification. It is absent from RFC 9309 too, which standardised the protocol in 2022 and defines only user-agent groups, Allow and Disallow, resolving conflicts by longest match with Allow winning a tie. When Walmart writes Crawl-delay: 5, that line reaches Slurp and nothing else.

Disallow is not a noindex. Google's own introduction says robots.txt "is not a mechanism for keeping a web page out of Google". A blocked URL that other sites link to can still surface in results, described from the anchor text alone.

Discovery without crawling

Most e-commerce discovery does not need link traversal, because the store already publishes the list. There are two places to look.

Sitemaps. The protocol caps one file at "no more than 50,000 URLs" and 50MB uncompressed, with an index listing up to 50,000 sitemaps under the same cap. Gymshark's index points at nine files, split by pages, collections, products and blog, and again per locale. Two optional tags are not worth reading: Google ignores <priority> and <changefreq> entirely, and uses <lastmod> only when it is "consistently and verifiably accurate". Check it against a few known changes before building incremental logic on it.

Platform JSON endpoints. Documented, supported, and far cheaper than parsing HTML.

• Shopify's Ajax API serves GET /{locale}/products/{product-handle}.js, returning the product object with variants, prices, SKUs and availability. The documented ceiling is 250 variants per response.
• WooCommerce ships the Store API at /wp-json/wc/store/v1/products. It is explicitly "an unauthenticated API. It does not require API keys or authentication tokens for access." Default page size 10, maximum 100, with X-WP-Total and X-WP-TotalPages headers giving the collection size before you page through it.
• Adobe Commerce exposes a GraphQL products query taking search, filter, pageSize and currentPage, defaulting to 20 items. Adobe's newer Cloud Service drops it for a separate Catalog Service query, so a 2021 store and a 2026 one differ here.

And one piece of folklore. Roundups repeat that every Shopify store exposes /products.json with the whole catalogue, 250 per page. We requested it on Gymshark on 10 August 2026 and got HTTP 403, from a host whose robots.txt and sitemap.xml had answered the same client normally minutes earlier. Whether that is the endpoint being switched off, a plan restriction or an edge rule blocking an unfamiliar client is not knowable from outside. The practical point survives the ambiguity: it is not in Shopify's documented API, and a pipeline built on it needs a fallback that is.

Ask the store for the list before going looking for it.

How they work together in a pipeline

In practice you rarely pick one instead of the other. A product-data job chains them:

1. Crawl the categories, or read the sitemap, to build the master list of product URLs.
2. Scrape each URL for price, stock, title and specs.
3. Store every run as a timestamped snapshot, raw response included.
4. Diff snapshots to surface price moves, new arrivals and stock-outs.

Crawling answers which pages you process. Scraping answers what you take from them. Skip the crawl and new products stay invisible until someone notices. Skip the scrape and you have a link directory.

Step two is where conditional requests earn their keep. RFC 9110 defines If-Modified-Since and If-None-Match, and a server that agrees your copy is current answers 304 Not Modified with no body. On a catalogue where most descriptions and images never change, storing the previous run's ETag turns a large share of fetches into empty responses. Prices are the exception, which argues for splitting the job by volatility.

One shortcut is worth naming. When the list of pages is already fixed, say a spreadsheet of 5,000 ASINs where you only need /dp/{ASIN} for each, discovery is done and no crawl is needed. Crawling earns its keep when the catalogue is large, moves, or was never handed to you.

What breaks at ten thousand pages a night

A script that pulls forty competitor products works on the first try. The same logic pointed at 200,000 URLs fails in ways that have nothing to do with parsing.

Arithmetic first. One request per second over 200,000 URLs is 55.6 hours. A nightly feed has roughly eight usable hours, so about seven requests a second, sustained, for the whole window. One wasted second per URL costs another 55 hours you do not have. That is why discovery gets pushed onto sitemaps and JSON endpoints. Not elegance. The clock.

A failed fetch is not a zero. The most expensive bug in retail scraping is also the quietest. A 403, a timeout or a page that never finished rendering produces a missing price, and a careless diff records missing as "removed" and empty stock as "out of stock". One bad night generates thousands of fictitious stock-outs, and if that feed drives repricing, you have moved real money on invented data. Distinguish fetched and absent from not fetched, keep the HTTP status in the row, and refuse to diff a run whose success rate fell below a threshold set in advance.

Rate limiting is a protocol, not an accident. RFC 6585 defines 429 Too Many Requests and says the response "MAY include a Retry-After header indicating how long to wait". Honour it. A retry loop that ignores Retry-After and backs off on a fixed timer turns a temporary throttle into a permanent block.

One product is not one URL. The same item lives at a canonical path, a collection-scoped path and three tracking-parameter versions. Colour and size variants may be separate URLs or one page with 250 variants in a single JSON blob. Deduplicate on GTIN or manufacturer part number where the page publishes one, on a normalised URL where it does not, and keep both keys. Fields also arrive inconsistent across sites, which is why collected data needs normalization and deduplication before anyone can analyse it.

Layouts drift under you. Rarely all at once. A class name changes on one template, the field goes null for 4% of the catalogue, and nobody notices for a fortnight because the row count looks fine. Alert on field-level fill rates per template, not on job success.

Sites that fight back are a separate budget line. Rendering, proxy rotation and challenge handling are recurring cost with a maintenance tail, which is where a managed web data extraction service stops being an outsourcing decision and becomes an arithmetic one.

What each step costs

Prices below were read from each vendor's own pricing page on 10 August 2026, before negotiation, with several promotions running that day.

Put one catalogue through them. A single pass over 200,000 product pages costs $26 to $254 on Zyte's HTTP tiers and $202 to $3,216 rendered. That spread, eight to thirteen times at equivalent complexity, is the price of a browser and the biggest single lever in the budget. ScrapingBee's credits have the same shape: one rendered pass is 1,000,000 credits, the entire monthly allowance of the $99 plan, and premium proxies take it to 5,000,000, past the 3,000,000 on the $249 tier. Bandwidth meters separately. At $8 per GB and an assumed megabyte per page, a number to measure rather than trust, 200,000 pages is 200 GB and $1,600.

Every figure above collapses if discovery comes from a sitemap and extraction from a JSON endpoint. Read the store's documentation before writing a selector.

When to use which

• The catalogue is large and changes often → crawl first, then scrape. New and delisted products only surface if you re-discover URLs on a schedule. This is why the two words get confused: you are doing both, nightly, forever.
• You already hold the exact URLs, ASINs or SKUs → scrape only.
• A handful of competitor products → scrape only, and a cron job will do.
• Auditing links or building a URL inventory → pure crawling, no extraction.
• A daily price and stock feed → the full chain, with the diff step treated as seriously as the fetch. This is the backbone of competitor price monitoring.

Defences, and which step they land on

Both halves got harder than a decade ago, when a retail page was static HTML anyone could parse in an afternoon. The defences do not land on both halves equally.

• Anti-bot systems hit discovery hardest. Listing, search and pagination endpoints are expensive to render and worthless to index, so they get rate-limited and challenged first. Sustained collection needs rotating proxies and, where challenges appear, CAPTCHA solving. Product pages, wanted in search results, are the softer target.
• JavaScript-rendered prices. Price and stock are often injected after load, so they are absent from the raw HTML. Either call the API the page itself uses or render in a headless browser. The first is eight to thirteen times cheaper.
• Crawler traps. Infinite filter and sort combinations, session IDs in paths, calendars with no end. Normalise URLs, cap depth, prefer the sitemap.
• Politeness. Cap concurrency per host, and back off on 429 and 503 rather than through them.

Crawler tooling has stayed conservative. Scrapy shipped 2.17.0 on 7 July 2026 and still handles the frontier, deduplication and throttling that people otherwise rewrite badly; Crawlee covers the same ground for JavaScript and Python. Crawl4AI and hosted equivalents return model-ready markdown from a URL, a different trade: less selector maintenance, less control over what got dropped. Playwright is where rendering ends up, with 1.62 at the top of its release notes on the day we checked.

Where the legal line sits

Public data is not unowned data, and price monitoring runs into rules written for the merchant rather than for you.

Prices in the EU have a legal definition. Article 6a of Directive 98/6/EC, inserted by Directive (EU) 2019/2161, says an announcement of a price reduction must state the prior price, and defines it: "the lowest price applied by the trader during a period of time not shorter than 30 days prior to the application of the price reduction." Member States had to adopt the measures by 28 November 2021 and apply them from 29 November 2021. The crossed-out figure on a European page is therefore a defined quantity with a 30-day lookback behind it, and putting it in the same column as a US "compare at" price stacks two different objects.

Personal data does not become impersonal because it is public. Review text, reviewer names and seller details fall under the GDPR whichever way the page is served. The safe default in a price feed is not to carry them.

Terms and robots.txt are evidence, not code. RFC 9309 gave the file a specification, not an enforcement mechanism. What it gives you is a record of what the operator permitted, which is what every later argument turns on. None of this is legal advice, and jurisdictions diverge sharply.

Where this leaves you

Crawling is discovery. It builds and refreshes the list of product pages. It is the half sites restrict, the half faceted navigation inflates, and the half a sitemap usually makes unnecessary.

Scraping is extraction. It turns each page into a row of price, stock and specs. It is the half where structured data beats selectors, where a failed fetch must never be recorded as a zero, and where rendering decides the bill.

You need discovery when the catalogue moves and you do not hold every URL. You need extraction to get anything at all. For a scheduled feed you need both, plus a diff step that knows a vanished product from a failed request.

Standing that up and keeping it alive against changing layouts and hardening defences is the actual work, and it recurs rather than finishes. Teams that would rather receive a clean structured product feed than maintain crawlers can buy the output instead, from a one-off catalogue export through a web data extraction service to an ongoing data-as-a-service feed. Either way the first question is the one robots.txt already answers: which pages, and who says so.