Web Scraping Methods & Tools: Every Way to Extract Web Data

Zyte publishes two price ranges for the same task. Pulling a page over plain HTTP starts at $0.13 per 1,000 requests. Rendering that page in a browser against a well-defended site runs up to $16.08 per 1,000. Same URL, same field, roughly a hundredfold spread in what the bytes cost. ScraperAPI says it in another currency: one credit for an ordinary page, 5 for Amazon, 25 for Google, 30 for LinkedIn, and 10 more on top when the target sits behind Cloudflare, DataDome or PerimeterX. Both price lists were read on 13 August 2026.

That spread is the real subject of this guide. Picking a scraping method is not a taste question about Python versus a point-and-click interface. It is a decision about who absorbs four costs and in what currency: your engineers' hours, a vendor's invoice, or a dataset that is free but three months stale.

What follows is a map, ordered from the methods where somebody else has already done the work to the ones where you do all of it. Each entry gets a verified price or version, the condition that makes it right, and the wall it hits. The deep dives live in their own articles, and this page routes you to them. If the vocabulary is fuzzy, what web scraping is and crawler vs scraper vs parser come first.

The four costs every method has to cover

Acquisition. Getting the bytes: requests, retries, proxies, sessions, rate control.

Extraction. Turning a page into fields: selectors, XPath, JSON paths, schema validation.

Defence. Getting past whatever the site runs to keep you out. This is the cost with the widest variance, and the one that turns a weekend script into a standing job.

Maintenance. Keeping the first three working after the site redesigns, the endpoint moves, the token format changes or the plan you were on gets repriced.

Every method below pays all four. The only variable is who holds the bill.

0. Check whether the data has already been collected

The cheapest scraper is the one you do not run. Common Crawl publishes a monthly archive of the open web as WARC, WAT and WET files on s3://commoncrawl/ in AWS us-east-1, free to anyone, no account needed. The August 2025 archive held 2.44 billion pages and 424 TiB of uncompressed content, and the release list now runs through CC-MAIN-2026-30. WET files are plain text already extracted, so a language survey or a link graph can be built without fetching a single page yourself.

Large sites often publish their own bulk data too: database dumps, sitemaps, RSS, open-data portals, government registries. Checking takes ten minutes. Building a crawler takes a week.

Where it stops. A broad crawl is wide, not deep. Your forty competitor product pages may be absent or six months old, and no coverage guarantee exists for any specific host. Prices, stock and ratings are exactly what a monthly corpus cannot give you. Grinding hundreds of terabytes also costs compute, so the data is free the way a quarry is free. If the answer has to be current, start with a sitemap crawl to size the job and read on.

1. Official APIs, and what they actually cost

Before scraping HTML, look for an endpoint. An API gives you typed fields, pagination, versioning and a contract instead of a parser that breaks on a redesign. The earlier version of this guide called that route "cleaner, more stable and less legally fraught." Two of those three have held up. Stability has not.

X publishes its tiers openly: a free tier capped at 100 reads a month, Basic at $200 a month for 10,000 posts read at app level, Pro at $5,000 a month for a million, and Enterprise on request (developer.x.com, read 13 August 2026). Google's Custom Search JSON API still documents 100 free queries a day and $5 per 1,000 after that, up to 10,000 a day, under a banner that reads "The Custom Search JSON API is closed to new customers" with existing customers given "until January 1, 2027 to transition to an alternative solution." A pipeline built on that endpoint in 2024 now has a migration deadline.

So treat an official API as a supplier relationship rather than a fixture. Ask what happens at ten times your volume, whether every field you need is present, and what the terms say about storage and redistribution.

The other API worth knowing about is the one the site never advertised. Most modern pages fetch their own data as JSON from an internal endpoint, and reading that endpoint beats parsing the DOM around it. That is a method in itself, covered in scraping vs. using an API and in the private-API half of scraping dynamic content.

2. Spreadsheets: the ten-minute method

Google Sheets ships a scraper in a formula. IMPORTXML takes IMPORTXML(url, xpath_query, locale) and drops the result into cells, refreshing on its own schedule, at no cost. For a price on one page, a list of links, or a table you want to watch for a fortnight, nothing beats it on time-to-answer. Excel does the equivalent through Power Query and VBA.

Where it stops. The formula fetches HTML and evaluates XPath. It does not run JavaScript, so any page that renders client-side returns an empty cell. Volume is capped by Google's own limits, error handling is a red cell, and a sheet full of import formulas turns into an unmaintainable dependency graph. Details and working formulas are in scraping with Google Sheets and Excel and VBA scraping.

3. Browser extensions

An extension turns the page in front of you into a selector picker: click the fields, click next-page, export CSV. The reference point is Web Scraper, whose Chrome listing showed 800,000 users, 4.1 stars from 1.1K ratings, and version 1.111.13 updated on 15 July 2026. It is free and runs locally, which is the point: it inherits your logged-in session and your fingerprint, so a site that trusts you as a reader usually trusts the extraction too.

Scheduling is where it stops being free. webscraper.io prices its cloud at $50 a month for 5,000 URL credits, $100 for 20,000, and from $200 for unlimited credits priced per concurrent scraper, with residential proxies as a $2.5/GB add-on.

Where it stops. Your laptop is the runtime. Close the tab and the job dies. No retry policy, no rotation, no alerting, and nothing to stop a defended site serving a challenge instead of a page. Current options are compared in the scraping browser extensions overview.

4. No-code and visual platforms

These are extensions with a server behind them: point and click to define the extraction, then run it in someone's cloud on a schedule, with proxy rotation and CAPTCHA handling bundled in. Analysts and marketers ship real projects this way without an engineer.

The pricing is not trivial, and roundups that call this tier "free" are describing the trial. Octoparse gives a free plan of 10 tasks, 50,000 exported rows a month, two local concurrent runs and no cloud extraction. Standard is $69 a month, or $58 billed annually, for 100 tasks and up to 3 cloud runs. Professional is $249 a month, or $209 annually, for 250 tasks and up to 20 cloud runs.

One correction to the lists this article sits alongside, its own earlier version included. ParseHub is still named in most no-code roundups with prices attributed to it. parsehub.com still answers, and its title still advertises "the most powerful web scraper", but the site renders entirely client-side and gave up no readable pricing to repeated fetches on 13 August 2026. Treat any figure quoted for it as unverified until you see it in your own browser.

Where it stops. Selector drift after a redesign, multi-step flows behind logins, exports past a few hundred thousand rows, and targets running commercial bot management. The wider survey lives in the web scraping tools roundup, with a closer look in the Octoparse review.

5. Writing it yourself with code libraries

The default engineering answer: an HTTP client fetches, a parser extracts, your code decides everything else. It has no per-row cost, no ceiling you did not build yourself, and it lives inside your repository with your tests.

We rechecked the load-bearing packages on PyPI and npm on 13 August 2026:

Two things read off that table. The old stack is not old: requests and Beautiful Soup both shipped this year, and the requests you install now needs Python 3.10 or newer. And httpx, recommended everywhere as the modern async client, last published 0.28.1 in December 2024. Nothing is broken and nothing is moving, which is worth knowing before you standardise on it.

Two packages that keep appearing in method roundups have not shipped in years. requests-html sits at 0.10.0 from 17 February 2019, pyspider at 0.3.10 from 18 April 2018. Neither belongs in a 2026 recommendation.

Language matters less than which one your team already ships. Python is the default, with a worked Python scraper and the library comparison to start from, and JavaScript and Node are natural on script-heavy targets. There are full guides for PHP, Java, C#, Go, Rust and Django projects, and the trade-offs sit in which language to use. One correction while we are here: an earlier version of this page pointed the word "Go" at the Django article.

Where it stops. You inherit the whole maintenance surface: markup changes, proxies, CAPTCHAs, rate limits, retries, storage and dynamic pages that need a real browser. For one site that is a weekend. For forty sites it is a role.

6. Frameworks, when one script becomes forty

A framework is what you reach for when the problem stops being extraction and starts being orchestration: scheduling, deduplication, retry policy, concurrency limits, item pipelines, storage.

Scrapy is the Python standard, at 2.17.0 released 7 July 2026, described by its own site as "fast, asynchronous, extensible, and BSD-licensed" and maintained by Zyte with several hundred contributors. Our Scrapy series is the practical entry point. Crawlee covers the same ground for JavaScript and Python, in its own words handling "blocking, crawling, proxies, and browsers for you", published by Apify, with the Node package at 3.18.1 on 12 August 2026. When a crawl needs a browser for some pages and not others, scrapy-playwright bridges the two and shipped 0.0.48 on 10 July 2026.

Where it stops. A framework organises work. It does not get you past a bot manager, and a Scrapy project pointed at a defended target fails the same way a bare script does, only with better logging. It is also genuine overhead for a job that is one page a day.

7. Headless browsers

When the data only exists after JavaScript runs, one option is to run the JavaScript. Playwright, Puppeteer and Selenium drive a real browser, wait for elements, click, scroll and read the rendered DOM.

The cost is not theoretical, and a vendor price list is the cleanest way to see it. Zyte charges $0.13 to $1.27 per 1,000 requests for an HTTP response, and $1.01 to $16.08 per 1,000 for browser-rendered content. That gap is the browser tax: memory, CPU, and a page load that pulls every image and tracker instead of the one JSON payload you wanted.

Which is why the first move on a dynamic page is to look for that payload. Scraping dynamic content works through both routes in code, and the private-API route wins on cost more often than teams expect.

Stealth tooling deserves a version check before you trust it. puppeteer-extra-plugin-stealth last published 2.11.2 on 1 March 2023, and undetected-chromedriver last published 3.5.5 on 17 February 2024, while nodriver reached 0.50.3 on 13 May 2026. A stack pinned to the first two is running patches written for browsers nobody ships now. What each defence actually detects is catalogued in how websites protect themselves and how it is bypassed, with proxies and CAPTCHA services as the two costs attached to it.

8. Unblocking and scraping APIs

This category did not exist in the first version of this guide, and it is now where most serious in-house scraping actually runs. You send a URL, the vendor handles proxies, browser rendering, fingerprints and challenges, and you get HTML or JSON back. Your code keeps the extraction logic and hands over the defence problem.

Published prices on 13 August 2026:

• ScrapingBee starts at $49 a month for 250,000 API credits and 50 concurrent requests, rising through $99 for a million and $249 for three million to $599 for eight million. New accounts get 1,000 free credits without a card.
• ScraperAPI starts at $49 for 100,000 credits and 20 threads, with $149 for a million and $299 for three million. The credit multipliers are the interesting part: 1 for a standard page, 5 for Amazon, 25 for Google or Bing, 30 for LinkedIn, plus 10 for a bot-protected target. Ten thousand LinkedIn pages is 300,000 credits, about $45 of a $149 plan.
• Zyte API is pay-as-you-go with commitment tiers at $100, $200 and $500 a month, and $5 of free credit to test with.
• Bright Data lists Web Unlocker and SERP API from $1 per 1,000 requests, residential proxies from $2.5/GB at the discount showing on the page, and datacenter proxies from $0.9 per IP.

Where it stops. The cost is linear and stays linear, so the crossover against a managed contract arrives quietly. You still own extraction, schema changes and validation, and "success" as the vendor bills it is a 200 response, not a correct field. Meter both.

9. LLM extractors and browser agents

The newest method class, and the one that changes the maintenance arithmetic rather than the fetching arithmetic. Instead of writing selectors, you describe the fields and let a model read the rendered page.

Firecrawl returns model-ready markdown from a URL and prices at 1,000 free credits a month, then $16 a month for 5,000 pages, $83 for 100,000, $333 for 500,000 and $599 for a million, on annual billing, where one credit is one page. Its Python client reached 4.35.0 on 12 August 2026. The open-source side is moving at a similar pace: crawl4ai 0.9.2 on 15 July 2026, scrapling 0.4.14 on 10 August 2026, browser-use 0.13.7 on 27 July 2026, Stagehand 4.0.0 on npm on 10 August 2026, and Microsoft's @playwright/mcp at 0.0.79 on 6 August 2026, which lets a model drive a browser through the accessibility tree.

What you buy is resilience to redesigns, since nothing is anchored to a CSS path. What you pay is per-page inference cost and non-determinism: the same page can yield a differently shaped record twice in a row.

Where the knowledge ends. We looked for an independent, reproducible accuracy benchmark for these extractors and found none. Every accuracy figure in circulation traces back to a vendor's own evaluation. Treat the tier as unmeasured: pin an output schema, validate every record against it, and diff a sample against hand-checked ground truth before trusting a run of ten thousand.

10. Data feeds and Data-as-a-Service

Sometimes the dataset already exists as a product. Instead of commissioning collection, you subscribe to a maintained feed and query or ingest it: product catalogues, company records, job postings, domain data. Bright Data lists datasets from $250 per 100,000 records, which is a quarter of a cent per record, and the buying question becomes whether the same fields can be collected and maintained in-house for less. DaaS and data by API explains how the delivery side works.

Where it stops. You get the categories somebody already maintains, with their schema, their refresh cadence and their coverage decisions. Anything niche, anything your competitors have not already made a market for, and anything where you need a field the provider does not collect falls outside it.

11. Managed collection

The method with no build step. A specialist team scopes the sources, writes and maintains the scrapers, runs the proxy and challenge infrastructure, monitors for breakage, and delivers a validated dataset as CSV, a database, a feed or an API on your schedule. That is the model behind managed extraction, and it wins in a specific shape of situation rather than universally.

• Maintenance is the whole cost. Anti-bot vendors retrain continuously and sites redesign on their own calendar. Somebody has to notice at 3am that a field went null, and with a service that somebody is on staff elsewhere. This is the largest line item that in-house estimates leave out.
• The defence stack comes assembled. Residential rotation, browser farms and challenge handling are already running, already amortised across clients.
• A data sample precedes the commitment. Fields, coverage and quality are visible before a contract exists, which is what separates a supplier from a promise.
• Cost scales with sources, not headcount.

Where it stops. For two hundred rows once, this is the wrong instrument. For a dataset your product depends on, price it against the fully loaded alternative, unblocking bill and engineer-hours included. The questions to put to a provider are in choosing a web scraping service, and a maintained feed via data-as-a-service is often the cheaper half of the same conversation.

What breaks at ten thousand pages a night

Every method above works on page one. The differences appear at volume, and they are the same differences regardless of which tool you picked.

Bandwidth becomes a line item. Ten thousand HTML-only pages at roughly 500 KB each is about 4.8 GB. At the residential prices above, that is around $12 a night at $2.5/GB and about $38 at $8/GB, before a single image loads. Run those pages through a real browser and the figure multiplies, because the browser fetches everything the page references.

Wasted time compounds. One unnecessary second per page across ten thousand pages is close to three hours of wall clock. A fixed two-second wait instead of a proper condition check is not a style preference at that scale.

Failures stop being exceptional. At ten thousand requests a night, a 1% failure rate is a hundred incidents. Without idempotent retries, a dead-letter queue and a rule for partial records, the dataset quietly loses its hardest pages.

Silent field loss is the expensive bug. A parser that finds nothing when a class name changes writes a null and exits zero. Nothing alerts. Monitor the shape of the output rather than the exit code: fill rates per field, row counts against yesterday, value distributions. A price column that is 4% null today and 40% null tomorrow is a redesign you have not noticed.

Identity and deduplication. Same product, three URLs, two currencies, one tracking parameter. Deciding what makes a record unique is upstream work, and data normalisation is what cleans up after skipping it.

Politeness is self-interest. Concurrency caps, backoff on 429 and an honest user agent keep you out of the logs that trigger a block. Knowing how sites detect scraping helps before your traffic pattern teaches them.

Knowing you picked the wrong method

Switching costs are lowest before the pipeline has consumers. Four symptoms say the method has been outgrown:

1. You are scheduling a browser extension by hand. Anything you run on a calendar belongs in a cloud platform, a framework or a service.
2. Your no-code platform started returning empty runs. Empty results on a site that works in your browser usually means challenges, not configuration. That is a defence problem, and no-code tools are not sold on the strength of their bypass.
3. Your engineers spend more time on proxies than on parsing. Move acquisition to an unblocking API and keep the extraction. The bill is legible and the maintenance drops.
4. Your unblocking bill approaches a managed quote. Per-request pricing loses to a contract somewhere in the tens of millions of requests. Compare totals, including the hours nobody currently invoices.

Is web scraping legal

The method changes the legal picture more than the topic does. Public pages, an official API and a login-gated area are three different positions, and the third is the one that has produced most of the litigation.

One claim needs correcting because it travels through nearly every methods guide. hiQ Labs v. LinkedIn is cited as the case that made scraping public data lawful. Both appellate rulings were preliminary, the 2019 opinion was vacated and remanded, and on the merits hiQ lost: LinkedIn won summary judgment on breach of contract in November 2022, and in December 2022 hiQ accepted a $500,000 judgment and a permanent injunction barring it from scraping LinkedIn at all. The narrower true statement is that the US Computer Fraud and Abuse Act is a weak instrument against logged-out scraping, which is why platforms now litigate contract, copyright and anti-circumvention instead.

Personal data is a separate track with its own rules regardless of how public the page is, which is the subject of GDPR and web scraping. The full picture, and the questions to put to your own counsel, are in is web scraping legal. None of this is legal advice.

Choosing, in order

Work down the list and stop at the first yes.

1. Has someone already published it? Common Crawl, a bulk dump, an open-data portal. Ten minutes of looking against a week of building.
2. Is there an official API with your fields? Use it, and price the tier you will actually need. A free tier of 100 reads a month is a demo, not a plan.
3. Is this a one-off question? A spreadsheet formula or a browser extension answers it this afternoon, and there is nothing to maintain afterwards.
4. Recurring, small, no developers? A no-code platform at $69 to $249 a month, as long as the target is not defended.
5. You have engineers and the pipeline is a product? Libraries for one or two sites, a framework past that, and an unblocking API the moment defence starts eating the sprint. Budget for maintenance as a standing cost rather than a project, because that is the line item that decides this build-versus-buy question and not the code.
6. The data is a maintained category? A feed or DaaS subscription beats collecting it yourself at a quarter of a cent per record.
7. You need the data, not the plumbing? Managed collection puts the maintenance and the defence stack on someone else's payroll, and the sample arrives before the commitment.

The methods are not ranked. A spreadsheet formula that answers today's question beats an elegant Scrapy project that ships in three weeks, and a $5,000 monthly API tier can be cheaper than the engineer who would replace it. What kills projects is picking a method for a volume you no longer have: an extension run by hand at ten thousand pages, or a bespoke crawler built for four hundred rows a month.

So price the method before you install it. Fetching, extraction, defence and maintenance are the four numbers, and every method on this page is a different arrangement of who pays them. Get that arrangement right and the tool barely matters. Get it wrong and no tool saves you.