SEO Scraping: How Web Scraping Is Used in SEO

Search Console impressions collapsed on most sites in September 2025, and almost none of those sites had done anything. Tyler Gargula of LOCOMOTIVE Agency looked at 319 properties and found impressions down on 87.7% of them and unique ranking terms down on 77.6%, with average position improving at the same time. Google had stopped honouring the &num=100 parameter. Rank trackers that had been pulling a hundred results per query, and generating an impression for every site sitting on pages two through ten, could no longer do it in one request.

That is the shortest available proof of how much search marketing runs on scraping: the industry's headline metric moved several points because somebody else's scrapers broke, on sites that never scraped anything. The mechanics are worked through in what actually changed about scraping Google; what matters here is the map. Every price, quota and version below was read from the vendor's own pricing page, documentation or release history on 13 August 2026.

Why this got harder, in numbers

SEO scraping used to be about efficiency: more phrases, more URLs, more positions checked. The reason has shifted. The results page now consumes your data instead of handing it back, so collecting it yourself is the only way to see it.

SparkToro, working from Similarweb clickstream data, put US Google searches ending without a click at 68.01% for January through April 2026, up from 60.45% in 2024. The same analysis found AI Overviews on more than 20% of searches, with click-through falling by close to 60% where they appear. Pew Research Center reached the same place by a different route in July 2025, tracking 900 US adults across 68,879 Google searches: users clicked a traditional result on 8% of visits where an AI summary appeared, against 15% where none did, and clicked a link inside the summary on 1%.

Read those together and the job changes shape. Your analytics see a shrinking share of the demand you compete for, and the rest lives in the SERP.

The six directions, and which one you need

Almost every SEO task that involves scraping falls into one of these six. They share a word and nothing else.

Keyword research

The largest and most tool-fragmented of them. Demand data comes from several places, each needing its own collector.

• Google Keyword Planner scraping - volume, seasonality and geography, straight from Google Ads. Google's own help page states that "your search volume statistics are rounded" and averaged over twelve months by default. Remember that before comparing two months of Keyword Planner figures and calling the difference a trend.
• Google autocomplete scraping - suggestions from the search box, still the cheapest source of real phrasing because the endpoint is light and public.
• People Also Ask scraping - questions and related searches, useful for intent, unreliable as a fixed inventory because the block appears and disappears by query.
• Keyword research at scale - combining sources, deduplicating, and clustering by SERP overlap rather than by string similarity.

Your own site

Reading your site the way a crawler does, and the one direction where nobody can block you. Site structure crawling walks every URL and produces the section tree, the duplicates, the redirect chains and the depth problems. Scraping titles and meta tags bulk-exports Title, Description and H1-H6, so you can find the empty ones, the duplicates and the ones Google rewrites in the results.

Rank tracking

Regular position capture for target keywords, by location and device. The September 2025 change hit this direction hardest, and hosted tracker versus own scraper is a purely arithmetic decision here.

The SERP and competitors

Collecting the results page as an object: who ranks, how snippets are built, which features occupy the space above the links. Relevance maps, competitor copy analysis and content briefs are downstream of it, and it is available as Google SERP scraping if you would rather receive the rows than produce them.

Images

Narrow in SEO, substantial elsewhere. Harvesting images from the SERP and from sites is mostly a dataset-building task now.

AI answers

This one did not exist when the first version of this article was written, and every suite has now shipped it. Instead of a position you get a share of citations across a sampled set of prompts. Semrush meters it as "AI prompts tracked daily", 50 on Starter and 200 on Advanced; SE Ranking bundles 100 with its Core plan; Ahrefs sells it separately as Brand Radar from $199 a month. Note the unit. A prompt, sampled, is not a query universe, and there is no index to page through.

The official search APIs are closing

The earlier version of this article presented official APIs and direct SERP collection as comparable options. That has stopped being true, and the correction is worth naming rather than editing away.

Bing went first. Microsoft's documentation now sits under previous-versions and says it in one line: "Public Bing Search and Bing Custom Search APIs were retired on 11th August 2025." The archived doc set is what remains.

Google is on the same path. The Custom Search JSON API carries a notice reading "The Custom Search JSON API is closed to new customers," with Vertex AI Search named as the alternative "for searching up to 50 domains" and existing customers given "until January 1, 2027 to transition to an alternative solution." Its terms while it lasted: 100 queries a day free, then $5 per 1,000, capped at 10,000 a day. It never returned the live web SERP anyway.

What survives is official data about you, not about the results page. The Google Ads API reached v25 on 22 July 2026 and remains the supported path to volumes. The Search Console API remains the supported path to your own clicks and impressions. Neither tells you who outranks you.

One line Google has published for years completes the picture. google.com/robots.txt reads Disallow: /search for every user agent, with exceptions only for /search/about, /search/howsearchworks, and a carve-out granting Allow: /search to facebookexternalhit and Twitterbot. Any position data you hold was gathered against that file.

What you scrape with, with prices read on 13 August 2026

Two names from the earlier version resisted checking. Serpstat renders its plan table client-side and returned no figures; Moz answered our automated requests with HTTP 403. Both sites are up. Neither price is quoted here, because we did not read it.

What a rank check costs, worked from the table. SE Ranking Core is $129 a month for 2,000 keywords checked daily: 60,000 checks over thirty days, or $0.0022 each. Ahrefs Lite, same $129 for 750 daily keywords, works out at $0.0057. Google's own Custom Search API charged $5 per 1,000, or $0.005, capped at 10,000 a day. The arithmetic assumes every plan runs at its ceiling every day; run at half and every figure doubles. Hosted trackers are priced below what Google charged for a worse version of the same data.

For custom work the one library worth naming is advertools, at 0.18.0 on 17 June 2026: a Scrapy-based crawler that extracts SEO elements by default, with sitemap and robots.txt parsers alongside. The rest of the stack is general-purpose data extraction work.

Where the desktop tools stop

The old claim that desktop crawlers have "no subscription caps" is misleading. The cap is your hardware, and Screaming Frog documents it precisely. In memory storage mode the application allocates a maximum of 2GB on 64-bit machines, "which enables it to crawl typically between 10k-100k URLs of a site." An 8GB machine with the allocation raised gets you "a couple of hundred thousand URLs." Switch to database storage mode with an SSD and 4GB allocated and the same tool handles "approx. 2 million URLs." The user guide rules out spinning disks in that mode, because read and write speed becomes the bottleneck.

That maps onto Google's own sizing. Its crawl budget guide says the subject concerns "large sites (1 million+ unique pages) with content that changes moderately often (once a week)" or "medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)." Under those thresholds, one laptop and £199 covers the whole direction.

Version 24.0, released 19 May 2026, added something more interesting: an MCP server that lets an AI assistant run crawls and export the data through node.js. The current build is 24.3, from 29 June 2026.

What breaks at scale

Index status is the wall nobody expects. The Search Console URL Inspection API is capped at 2,000 queries per day per property and 600 per minute. At the ceiling, checking a 250,000-URL site once takes 125 days. There is no tier above it. Past a few thousand URLs, index coverage is something you sample, not something you verify.

Your keyword export is not all your keywords. Search Analytics allows a rowLimit of 25,000 per request against a default of 1,000, so pagination is mandatory and easy to forget. Rare queries are withheld for privacy regardless, so the long tail you are measuring is the part most likely to be missing.

Politeness is your problem alone. Google's robots.txt specification supports user-agent, allow, disallow and sitemap, and states that "other fields such as crawl-delay aren't supported." That line stops nobody, Googlebot included. Pacing has to live in your crawler.

Scale changes the failure mode. A crawl that dies at 80% costs an afternoon. A nightly job that silently drops one locale for three weeks costs a decision. Record row counts per run and alert on the delta, not on the exception.

Rendering is where the budget goes. A static fetch is cheap. A headless browser is not, in wall time and in memory per instance, and on 200,000 URLs that gap decides the plan. Crawl statically first, re-crawl only the URLs whose text comes back thin, and expect anti-bot systems to notice either way.

Where SEO scraping stops working

A rank is a distribution, not a number. Two trackers checking the same keyword from the same country at the same hour disagree, because personalisation, location granularity and result mixing are all live. Treating a tracked position as a measurement rather than a sample is the most common misreading in the discipline, and it is unpacked in rank tracking.

Volumes are models, not counts. Keyword Planner numbers come from Google, rounded. Every third-party volume is a clickstream model calibrated against something else. On low-volume terms they disagree by multiples, and neither is wrong.

AI visibility is a survey. Prompt tracking samples a list somebody wrote. There is no ranked index behind an AI answer, no page two, nothing to enumerate. A complete picture of it is a sample with the error bars taken off.

The law is neither settled nor uniform. Public data is not free data in the EU, and US terms of service bind users differently from logged-out visitors. The line that hiQ beat LinkedIn is the wrong lesson to carry: hiQ accepted a $500,000 judgment and a permanent injunction in December 2022. Where the boundaries sit is worth reading before a large collection programme.

Build it, buy the suite, or have it delivered

Two numbers mostly settle it: how many URLs, and how often.

Under roughly 100,000 URLs and a few hundred keywords, one desktop licence and one suite subscription cover everything above for under $2,000 a year. Nothing here justifies engineering time at that size.

Above it the arithmetic inverts. Suite plans meter keywords and audit pages, and multi-locale tracking multiplies both. Custom collection turns a subscription into a per-request cost: cheaper at volume, and considerably more work, once you add proxies, solving, pacing and parser maintenance on a layout that changes without notice.

The third route removes the infrastructure question. Somebody else absorbs the proxy bill, the solve rate and the parser breakage, and delivers rows in CSV, Excel or JSON: SERP collection as a recurring feed, or broader data as a service where SEO is one consumer among several.

Four questions decide it. How many URLs, how often, how many locations, and what happens if the collection silently returns half the rows for a week. The last one gets skipped, and the last one costs money.

The next article in this series takes the first direction apart in detail, starting with scraping Google autocomplete suggestions.