Best Free Web Scrapers and Tools for Developers

A 1.44 MB product listing page takes 40 milliseconds to parse. The full job, parse the page and pull 5,000 prices out of it, takes anywhere from 44 milliseconds to 13.5 seconds on the same machine, depending on one character in your selector. Nothing about that gap is written on any tool's home page. The free toolkit is easy to assemble and easy to assemble wrong.

Almost every piece of a scraper is free, open source, or has a usable free tier. What changes from year to year is which pieces are still alive. We rechecked each tool below against its own release page, repository, store listing or pricing page on 13 August 2026. Four entries moved since the last revision: one extension changed owner, one HTTP debugger turned out to be non-commercial-only and in maintenance mode, one framework has not tagged a release since March 2024, and one online utility this article recommended by name spent five years publishing whatever users pasted into it.

The stack that works, in one screen

1. Fetch with requests 2.34.2 or httpx 0.28.1. Switch to curl_cffi the moment a site returns 403 to code that works fine in a browser, because by then the problem is your TLS handshake and no header will fix it.
2. Parse with lxml, or selectolax when the pages are large. BeautifulSoup stays the right answer for messy markup and for code other people have to read.
3. Crawl with Scrapy 2.17.0 in Python, Crawlee 3.18.1 in Node.
4. Render with Playwright, only for pages that genuinely need it.
5. Find what to request in the DevTools Network tab. Still the highest-value fifteen minutes in the process.
6. Wrangle with CyberChef and jq.

Everything after this is why, plus the parts that fail.

Free is a licence, not a budget

Three costs never appear in a comparison table, and all three are larger than the zero on the price tag.

Wall-clock cost of a wrong choice. The measurements below show a 235-fold spread between two selector styles that return identical results. On a ten-thousand-page crawl that is thirty-seven hours against ten minutes. Nobody notices on the test page.

Free-tier arithmetic. A managed API that gives you 1,000 credits a month is giving you 33 requests a day. Enough to build and debug an integration. Not enough to keep a price monitor running, and the gap between those two is where the invoice starts.

Maintenance half-life. Scraping libraries rot faster than most software, because their environment is adversarial and changes without notice. Of the tools on the original version of this list, one has an untouched store entry from 2022, one has no tagged release since 2024, and one is labelled by its own vendor as not in active development. None are broken. Knowing which is which is the actual skill.

Open-source web scraping libraries

If you write code, this is where you live. Versions below were read from PyPI and the npm registry on 13 August 2026.

Python

• requests 2.34.2 (14 May 2026): the synchronous workhorse, shipping again after a long quiet period. Four releases landed in eight days in May 2026.
• httpx 0.28.1 (6 December 2024): HTTP/2 and native async, which is why it wins for concurrent work. Read that date again. The stable line has not moved in twenty months and the 1.0 previews stopped at 1.0.dev3 in September 2025. Nothing is broken. Nothing is moving either. HTTP/2 also needs the extra, pip install httpx[http2], and without it http2=True raises rather than downgrading quietly.
• BeautifulSoup 4.15.0 (7 June 2026): the friendliest HTML parser and the most forgiving with broken markup. Pair it with the lxml backend, which more than halves its runtime on a large page.
• lxml 6.1.1 (18 May 2026): C-based parsing with real XPath 1.0. Fast, and capable of being catastrophically slow on the wrong XPath. See the benchmark below before assuming speed comes for free.
• selectolax 0.4.11 (15 July 2026): the one most roundups miss. A binding to the lexbor engine with a native CSS selector engine, no XPath, and a much smaller API. It was the fastest thing measured below, on the selector that costs lxml the most.
• parsel 1.11.0 (29 January 2026): Scrapy's selector layer as a standalone package. XPath, CSS and regex on one object, without the framework.
• Scrapy 2.17.0 (7 July 2026): a crawling framework, not a library. Scheduler, concurrency, retries, item pipelines, exporters. Still the right answer past a few thousand pages. More in our Scrapy walkthrough and our roundup of the best Python web scraping libraries.
• Playwright 1.62.0 (31 July 2026) and Selenium 4.47.0 (10 August 2026): real browsers for JavaScript-rendered pages. Playwright is the modern default: auto-waiting, one API across Chromium, Firefox and WebKit. Selenium is the veteran with the deepest ecosystem and it ships monthly. See Selenium web scraping and scraping dynamic content.
• curl_cffi 0.16.0 (1 August 2026): the entry that did not exist when lists like this one were first written, and the one that changes outcomes most often. More below.

Scrapy's release notes are worth a skim if you are on an older version. 2.14 dropped Python 3.9. 2.15 introduced an experimental mode that runs Scrapy without a Twisted reactor at all. 2.16 added official Python 3.14 support and deprecated scrapy.FormRequest. 2.17 carries a security fix making S3 requests default to HTTPS, plus HTTP/2 and SOCKS proxy support in the experimental HTTPX download handler.

JavaScript and Node.js

• Cheerio 1.2.0 (23 January 2026): server-side jQuery-style parsing. Small, fast, no JavaScript execution. Correct when you already have the HTML.
• Puppeteer 25.6.0 (11 August 2026): Chrome and Chromium, maintained by the Chrome team. It follows Chrome's own release cadence, so a pinned old version drifts away from the browser it drives faster than most dependencies do.
• Playwright 1.62.1 (30 July 2026): cross-browser, same API surface as the Python package, which matters on a mixed team.
• Crawlee 3.18.1 (12 August 2026): Apify's crawling framework, wrapping Cheerio and Playwright/Puppeteer with request queues, autoscaling, session and proxy rotation. The closest thing Node has to Scrapy. The Python port is at 1.9.1 (6 August 2026), younger and moving fast.

Go, Ruby, PHP, and the rest

• Colly (Go): a clean callback API, excellent throughput, 25.3k stars, and here is the part the lists leave out. The latest tagged release is v2.2.0, 27 March 2024. The repository is not archived and master keeps moving, but pinning to tags pins you to something over two years old. Go users routinely pin to a commit hash instead, which makes this less alarming than it would be in Python. Check the branch before you assume.
• Nokogiri (Ruby) 1.19.4, 18 June 2026: the definitive Ruby HTML and XML parser, CSS and XPath, actively maintained. Version 1.19.0 added native gem support for Ruby 4.0 and ended support for Ruby 3.1.
• Symfony DomCrawler 8.1.1 with BrowserKit (PHP): the modern PHP parsing stack. Pair with Guzzle or symfony/http-client. CSS selectors come from the separate symfony/css-selector package, easy to forget until a selector silently fails to resolve.

Your bottleneck is the selector, not the parser

The reasonable assumption is that parser choice decides speed, so you pick the fast C library and move on. That assumption is wrong by two orders of magnitude in a case you will hit on your first real catalogue page, and no roundup we could find measures it. So we did.

Method. We generated a synthetic catalogue page of 5,000 product cards, 1,442,228 bytes, each card a nested structure of links, images, headings, a price span and a two-item tag list. Then we timed parse-plus-extract of all 5,000 prices, median of three runs, Python 3.11.15 on a single Intel Xeon core at 2.10 GHz. The generator is below, so the result is reproducible.

def build(n=5000):
    p = ["<!doctype html><html><head><title>Catalogue</title></head><body><div id='grid'>"]
    for i in range(n):
        p.append(f"<div class='card' data-sku='SKU{i:06d}'>"
                 f"<a href='/p/{i}'><img src='/img/{i}.jpg' alt='Item {i}'></a>"
                 f"<div class='body'><h3 class='name'>Item {i}</h3>"
                 f"<span class='price' data-cur='USD'>${(i%900)+9}.99</span>"
                 f"<span class='stock'>{'in' if i%3 else 'out'}</span>"
                 f"<ul class='tags'><li>tag{i%7}</li><li>tag{i%11}</li></ul>"
                 f"</div></div>")
    p.append("</div></body></html>")
    return "".join(p)

Two selectors were compared for every library. The descendant form, div.card span.price, is what everyone writes. The child form, div.card > div.body > span.price, is the same query with the path spelled out.

Measured on a single machine, on a 1,442,228-byte page, median of three runs. Your numbers will differ; the shape will not. Note that selectolax's child-form total sits below its own parse-only figure, which is run-to-run noise rather than magic: treat everything under about 60 ms here as the same number.

Four things fall out of that table.

Parsing is free. Selecting is not. Building the tree costs 40 to 52 milliseconds either way. Every millisecond above that is the query. When someone tells you lxml is fast, they are describing the bottom of that table and ignoring the top.

The // in the middle of an XPath is the expensive character. //div[@class='card']//span[@class='price'] costs 13.5 seconds. Replacing that inner // with /div/ costs 57.7 milliseconds, a factor of 235 for a change that does not alter the result set. A descendant step re-walks the subtree under every matched card, so the cost grows with the product of the two node counts rather than the sum.

lxml's CSS support is XPath in disguise, and it inherits the problem. cssselect compiles div.card span.price into an XPath with exactly that inner descendant step, which is why it lands at 4.6 seconds while the child form lands at 175 milliseconds. BeautifulSoup has no such failure mode, because soupsieve is a native CSS engine rather than a translator, and its two forms differ by only 30 percent.

selectolax is the outlier that earns a line in your requirements file. It ran the naive descendant selector in 58.5 milliseconds: 79 times faster than lxml on the identical query, 68 times faster than BeautifulSoup's default parser. No XPath and a smaller API, which for a fixed extraction job is not much of a loss.

On ten thousand pages of this size, the bad XPath costs thirty-seven hours and the good one costs ten minutes. The difference is one character. For more on the selectors themselves, see our guide to CSS selectors for web scraping.

Free web scraping APIs and what their free tiers actually are

Managed scraping APIs put rotating proxies, headless browsers and CAPTCHA solving behind a single HTTP call. Most are commercial, and several ship a free allowance that is real rather than decorative. Every figure below was read from the vendor's own pricing page on 13 August 2026.

Two corrections against older roundups, this article's previous version included.

Bright Data is not an enterprise-only option. The earlier version of this article filed Bright Data and Oxylabs under "free trials, overkill for hobby use". Bright Data's Web Unlocker free tier is 5,000 requests a month with no card, five times ScraperAPI's recurring allowance and the largest permanent free tier of the seven. Whether you want a data broker in your stack is a separate question.

ScrapingBee's 1,000 credits are one-off, not monthly. The page reads, verbatim, "Get access to 1,000 free scraper API credits, no credit card required." There is no permanently free tier. Treat it as a trial and budget accordingly.

Where the free tiers break. Free plans cap concurrency before they cap volume, and concurrency is what you actually need. Firecrawl allows two simultaneous requests on free, ScraperAPI five. At two concurrent requests and one second per page, 1,000 credits are gone in under nine minutes of wall-clock time and then you wait a month. Fine for building an integration, useless for running one.

Also watch the unit. Zyte and Oxylabs price per successful response, Apify prices compute units, Firecrawl and ScraperAPI price credits, and a JavaScript-rendered request costs more than one credit almost everywhere. Comparing headline numbers without reading each vendor's definition of a unit produces confident nonsense.

Free browser-extension scrapers

Point-and-click extensions are the fastest path from "I want that table" to a CSV. They also sit on a platform that just moved under them.

The Manifest V2 deadline is real and it is this month. Google's own MV2 deprecation timeline records two dates. On 24 July 2025, with Chrome 138, "Manifest V2 is disabled everywhere" for all users on all channels. On 31 August 2026, "All remaining Manifest V2 extensions removed from the Chrome Web Store." A listing that still loads today is therefore not evidence that the extension runs. Store figures below were read on 13 August 2026.

Instant Data Scraper changed hands, and almost nobody has noticed. One click auto-detects the main table or list on a page and exports to CSV or Excel, and on listing pages it is startlingly good. Every article on the subject, this one's earlier version included, credits it to Web Robots. Web Robots' own page now says, verbatim: "Instant Data is no longer owned, developed or supported by Web Robots", and points users at a Facebook group for help. The store lists the publisher as Flavr Technology, LP, and the extension is still updated, most recently 16 July 2026. So this is a transfer, not an abandonment. It still matters: a free extension with a million installs and a new owner reads every page you point it at.

Web Scraper (webscraper.io) remains the strongest free pick for structured, recurring, no-code jobs. You build a sitemap by clicking elements, and it follows pagination and nested pages. Local scraping is free without limit. Cloud plans start at $50 a month for 5,000 URL credits and two concurrent scrapers, rising to $100 for 20,000. See our Chrome scraping extensions roundup for the wider field.

Data Miner works from recipes, with a large public library for popular sites. Its free plan is 500 pages a month, and the wording deserves quoting: "If you do exceed the 500 page scrapes in a given month your account will be automatically locked indefinitely." Not throttled. Locked. Paid plans run $19.99, $49, $99 and $200 a month.

SelectorGadget is not a scraper. Click what you want, click what you do not, and it produces a CSS selector, which makes it useful when writing code scrapers rather than only when avoiding them. Its store entry has not been touched since 3 August 2022, normally a death sentence in a Manifest V2 world. Not here: the extension manifest for that same version 1.2.0 declares "manifest_version": 3, so it survives the August 2026 purge. The project site also still offers the original bookmarklet, which depends on no store policy at all. Four years without an update, and still the right recommendation.

Extensions run in your browser, at your IP address, at human speed, and they break when a site changes its markup. Use them for reconnaissance and one-off pulls. Graduate to code for anything that has to run twice.

Developer helper tools: selectors, sniffers, and data wrangling

Half of scraping is the small stuff at the edges: working out what to request, and cleaning up what comes back.

Find and test selectors

• Browser DevTools (F12): free, built in, and still the single most valuable tool here. The Elements panel copies CSS selectors and XPath. The Network tab, filtered to Fetch/XHR, exposes the JSON endpoints that make the rest of the job trivial. Fifteen minutes there routinely removes the need for a headless browser. Our guide to scraping hidden JSON APIs covers what to do once you find one.
• regex101: live testing with match highlighting, a step-by-step debugger and a plain-English explanation of the pattern. It covers PCRE2, JavaScript, Python, Go, Java, .NET, Rust, POSIX ERE and POSIX BRE, which matters because a pattern that works in one flavour can silently mean something else in another. RegExr is the friendlier alternative, with two flavours only, JavaScript and PHP/PCRE. Full field in our best online regex testers.
• The DevTools console: $$("css") and $x("//xpath") run your selector against the live DOM in one line. Test there first, because the DOM you get in Python is the DOM before JavaScript ran, and the two disagree more often than people expect.

Watch the traffic

Modern pages fetch their data in the background. A sniffer shows you exactly what was requested so you can replay it directly.

• mitmproxy 12.2.3 (12 May 2026): free, MIT-licensed, scriptable, and the one to learn. Three tools in the box: mitmproxy on the terminal, mitmweb in a browser, mitmdump for scripted capture. Its real advantage is mobile. Point a phone at it, install the certificate, and an app's private API is suddenly readable.
• Fiddler Classic: still a free download, and the earlier version of this article stopped there. Telerik's own page is blunter: "Non-commercial use only | Maintenance mode | Community support | Free", and "Fiddler Classic is not in active development and offers no commitments for releases, patches or tech support." If you are debugging a scraper you get paid for, the licence does not cover you.
• Charles Proxy: version 5.1, released 21 April 2026, with a 5.2.1 build on the download page. Not free: licences are $50 each for 1 to 4 users, $40 at 5 or more, $30 at 10 or more, with a $400 site licence and a $700 multi-site licence. The trial runs 30 days. Our Charles review covers the workflow.

Ranked honestly for scraping work: mitmproxy first, Charles second if you want the GUI and will pay $50, Fiddler Classic third and only for personal use.

Wrangle data formats: JSON, XML, CSV

A few tools have absorbed the old generation of single-purpose web utilities.

• CyberChef 11.3.0 (24 July 2026): GCHQ's "Cyber Swiss Army Knife", free, open source and actively developed, with four releases since April 2026. It chains operations into a visual recipe: pretty-print or minify JSON, swap CSV delimiters, parse XML, decode URLs, and several hundred more. Everything runs in your browser tab, which is the property that matters below.
• jq 1.8.2 (20 June 2026): the standard command-line JSON processor, and the project is alive again after moving to the jqlang organisation following a five-year silence. Update yours. 1.8.2 is a security release, and jq is exactly what you pipe untrusted scraped input through.
• jsonrepair 3.15.0 (3 July 2026): a library and CLI that fixes JSON that is nearly JSON. Trailing commas, unquoted keys, single quotes, comments, truncated documents, Python's None/True/False. Inline page state is a JavaScript object literal far more often than it is valid JSON, and json.loads refuses all of it.
• JSON viewers: for deeply nested payloads a graph view beats an indented tree. Our free online JSON tools and the full JSON viewer roundup cover the field, including which ones process data locally.
• CSV and XML repair: the recurring headache is a CSV whose delimiter collides with a locale that uses commas as decimal separators. CyberChef or a csvkit one-liner swaps delimiters cleanly. For XML, the W3C Markup Validator is still the reference, because it shows the offending snippet in context rather than printing an error code.

A correction that matters more than a version bump. Earlier versions of this article named jsonformatter.org as a formatter worth using. Do not. In November 2025 the security firm watchTowr Labs reported that jsonformatter.org and codebeautify.org had been exposing user-submitted content through their save-and-share features for roughly five years: around 80,000 documents, some 5 GB, containing Active Directory credentials, database connection strings, cloud API keys and internal bank data. Nothing was breached in the usual sense. People pasted secrets, clicked save, and the resulting links were public with guessable identifiers.

The lesson is narrower than "never use online tools". Formatting in a browser tab is normally harmless. The dangerous feature is persistence: any button labelled save, share link or cloud document puts a copy of your payload on someone else's disk behind a URL you do not control. Scraped data very often contains personal data, which makes that a transfer to a third party rather than merely a bad habit.

Encode, decode, and hash

Scrapers constantly meet encoded values: a Base64 blob in a payload, a URL-encoded parameter, a hash used as a cache key or signature. Two tools cover essentially all of it. CyberChef handles Base64 and Base32, URL encoding, base conversion, the hash suite from MD5 through SHA-512 to RIPEMD and CRC32, and ciphers, all chainable in one recipe. DevToys 2.0.9.0 (8 January 2025) does the same offline: about thirty utilities on Windows, macOS and Linux since the 2.0 line, including JWT decoding and JSON to YAML, with no round trip to any website. Its release cadence has been quiet since early 2025, but it is a utility box, not a browser engine, and there is little in it to rot.

Between them they killed the whole genre of single-purpose conversion sites.

What changed since this list was first written

Three shifts that a 2024-vintage roundup will not mention, and all three change what you should install.

Your TLS handshake is a fingerprint. A plain requests or httpx call is distinguishable from a browser before a single byte of HTML is exchanged, because Python's TLS ClientHello, cipher ordering and HTTP/2 SETTINGS frame do not look like Chrome's. That is why code copying every browser header still gets a 403. curl_cffi exists for exactly this: it binds to a patched libcurl and impersonates browser TLS/JA3 and Akamai HTTP/2 fingerprints, pseudo-header order and settings frames included. Its own documentation is clear about the ceiling: "curl_cffi is a python binding to a C library, with no browser or JavaScript runtime under the hood." It fixes the handshake. It cannot answer a JavaScript challenge. A headless browser is the step after that.

A whole category appeared for LLM pipelines. Crawl4AI 0.9.2 (15 July 2026) is open source and self-hostable, built to turn pages into clean Markdown for retrieval pipelines, with CSS, XPath and LLM-based extraction strategies. Firecrawl covers similar ground as a managed API. Whether you need either depends on where the data lands: a database schema is better and cheaper served by a normal scraper, while a vector store makes Markdown-first tools worth the dependency.

Extension platforms consolidated. Manifest V2 is gone from Chrome and its remnants leave the store this month, thinning the long tail of point-and-click scrapers. The survivors have a company behind them. Mixed blessing: fewer abandoned extensions, more extensions whose ownership can change without a version bump.

Where the free stack stops

Free tools cover more ground than people expect and less than paid vendors claim. Four places where the boundary is real.

Sites that fight back. Once a target runs a modern anti-bot service, the open-source stack stops being a tooling question and becomes a proxy and fingerprint question. Residential IPs cost money. So does solving challenges. The honest options are to pay a managed API per request, to buy proxies and do the fingerprint work yourself, or to hand the pipeline to a managed extraction service and receive the rows.

Volume with a deadline. A single machine running Scrapy will happily do a few hundred thousand pages. The trouble starts when you need them by a fixed hour, because that becomes distributed scheduling, deduplication across workers, retry budgets and monitoring. All solvable with free software. None of it free of your time.

Data you have to keep fresh. A scraper is not a project, it is a subscription to someone else's markup. Every redesign is an unplanned afternoon. This is where the free stack quietly becomes the expensive one, and where teams that need the rows rather than the pipeline move to data as a service instead.

Anything involving personal data. The tools are legal. What you collect and how you use it is the regulated part. Respect robots.txt and rate limits, read the Terms of Service, and treat GDPR and CCPA/CPRA as binding on public data too, because publicly visible is not the same as freely processable. Our overview of web scraping legality is the place to start before collecting anything about people.

Comparison table

Libraries and utilities only. The managed APIs and the extensions have their own tables above.

How to choose your free toolkit

1. Look before you fetch. Open DevTools, filter the Network tab to Fetch/XHR, and check whether the page loads its own data from an endpoint you can call directly. Half the jobs people plan as headless-browser work are one JSON request.
2. Fetch. requests or httpx for static pages. curl_cffi when working code starts returning 403. Playwright only when content genuinely needs a rendering engine, because a browser costs orders of magnitude more time and memory per page.
3. Parse. lxml by default, selectolax on large pages, BeautifulSoup when the markup is broken or someone else has to maintain the code. Write child selectors, not descendant ones.
4. Crawl. Scrapy or Crawlee once you need retries, scheduling, deduplication and pipelines. That point arrives earlier than people expect, usually the first time a run dies at 80 percent.
5. Clean. CyberChef or DevToys for encoding chores, jq for pipelines, jsonrepair in front of anything scraped out of a page rather than out of an API.
6. Skip the code when the job is one table on one page. Instant Data Scraper or Web Scraper will finish before your virtualenv does.

Reach for a managed API's free tier when anti-bot overhead costs more of your time than the data is worth. Rough marker: if you have spent two days on fingerprints and are still blocked, the $49 plan is cheaper than the third day.

Dead ends and stale recommendations

• jsonformatter.org and codebeautify.org save-and-share links. Five years of public documents containing credentials. Named in this article's earlier version; withdrawn.
• undetected-chromedriver. Last PyPI release 3.5.5, 17 February 2024, and still installed by thousands of tutorials. Two and a half years is a long time in anti-bot terms.
• puppeteer-extra-plugin-stealth. Last published 2.11.2 on 1 March 2023, alongside puppeteer-extra 3.3.6 the same day. Still beats naive checks. Has tracked nothing since.
• html5lib. Version 1.1, June 2020. The most spec-accurate Python HTML parser and the slowest by a wide margin. Use it deliberately, not by default.
• Fiddler Classic for paid work. Free, but non-commercial-only and in maintenance mode by the vendor's own description.
• Colly release tags. The branch is active. The tags are not.
• Instant Data Scraper as a Web Robots product. The store publisher is Flavr Technology, LP.
• Any Chrome extension whose store entry predates mid-2024. Manifest V2 stopped running in July 2025 and the last MV2 listings go on 31 August 2026. SelectorGadget is the exception: old listing, MV3 manifest, still fine.

Changelog

August 2026. Rechecked every tool against its own release page, repository, store listing or pricing page. Added the measured selector benchmark, the free-tier price table with dollar figures, selectolax, parsel, curl_cffi, Crawl4AI, Firecrawl and Bright Data's free allowance. Corrected Fiddler Classic's licence and maintenance status, ScrapingBee's one-off rather than monthly free credits, and the claim that Bright Data and Oxylabs are enterprise-only. Flagged the Instant Data Scraper ownership change, Colly's stale release tags and httpx's twenty-month gap. Added the Manifest V2 removal deadline. Withdrew the jsonformatter.org recommendation. Dropped the FAQ section and folded its content into the body.