Web Scraping Keywords from Google for SEO

Ask Google's suggest endpoint what follows "web scraping" and it answers with nine phrases: python, meaning, ai, services, vs web crawling, legality, python libraries, definition, tools python. No key, no account, no cookie. That request went out on 13 August 2026 and came back in one round trip. Now ask how many people search each of those nine phrases per month, and the free part of the day is over. That number lives inside Google Ads, and Google Ads will not show it to you until you have entered a payment method.

That gap between the two halves of the job is the real subject here. Collecting phrases is easy and getting cheaper. Attaching a defensible number to each phrase is neither, and the ways of doing it changed more between September 2025 and now than in the previous five years. Web scraping keywords means building both halves at a scale where hand-work stops: the phrases, the demand behind them, and the queries that cluster around them.

This guide is about the sources and about the fact that they disagree with each other. The mechanics of each one are covered next door and not repeated here: autocomplete endpoints and their parameters, what a search-volume figure is and where to buy one, turning raw phrases into a structured keyword set, scraping the results page itself, and tracking positions once the pages are live.

What a keyword number means depends on where it came from

Four sources give you a number called "search volume" or something adjacent to it. The four numbers are not the same kind of thing, and putting two of them in one spreadsheet column is the most common way a keyword project goes quietly wrong.

Keyword Planner returns an average, for a group, from an ad system. The figure is an average of monthly searches over a trailing window, produced for advertisers who want to forecast spend. Plurals, close variants and common misspellings routinely come back with identical numbers, which is the visible edge of an internal grouping rule Google has never published. Practitioners also report that volumes arrive as broad ranges until an account is spending; Google documents neither the behavior nor the threshold, so treat every specific figure you have read about that threshold as folklore. What Google does document is the door: "You must complete your account setup by entering your billing information to access basic features like 'Get ideas for new keywords'." That sentence is on Google's own help page for the tool. Keyword Planner is free of subscription, not free of onboarding.

Trends returns a ratio of a sample, scaled to 100. Google's documentation on how Trends data is adjusted is unusually direct about it: "Each data point is divided by the total searches of the geography and time range it represents to compare relative popularity," and "the resulting numbers are then scaled on a range of 0 to 100 based on a topic's proportion to all searches on all topics." It is also a sample rather than a census, because "providing access to the entire data set would be too large to process quickly." A 100 in Trends and a 100 in Keyword Planner have no relationship whatsoever.

Search Console returns counted events for one property, with the tail removed. Impressions and clicks here are real, logged, and yours. They are also incomplete on purpose. Google's reference for the Search Console bulk export tables describes an is_anonymized_query field this way: "Rare queries (called anonymized queries) are marked with this bool," and "the query field will be null when it's true to protect the privacy of users making the query." Read that twice if you do long-tail work. The rarest queries, which are exactly the ones you scraped suggestions to discover, are the ones Search Console will count for you and refuse to name.

Third-party volumes are models. Ahrefs, Semrush and the rest blend clickstream panels with Planner data and their own indexes. Two vendors will give you two different numbers for the same string on the same day, and there is no arbiter, because none of them publishes panel composition per country in a form you could audit. That is not an accusation. It is a reason to treat vendor volume as an ordering signal rather than a measurement.

Four sources, four units. Never put two of them in the same column.

What changed between September 2025 and today

Five things moved, and four of them narrowed the options.

The &num=100 parameter went away in the middle of September 2025. One request no longer returns a hundred results, so a hundred-deep look at any query now costs ten requests instead of one. The arithmetic and its consequences for SERP collection are worked through in the Google scraping notes. The consequence specific to keyword work landed somewhere less obvious: Search Console. Rank trackers had been pulling hundred-result pages all day, and every one of those pages generated impressions for every site on it. When the parameter died, the phantom impressions died with it. Tyler Gargula of LOCOMOTIVE Agency looked at 319 properties and published the result through Search Engine Land on 18 September 2025: 87.7% of sites lost impressions in Search Console, and 77.6% lost unique ranking terms. Nothing happened to those sites. Their baseline was corrected. If your keyword prioritization compares numbers from before and after that week, it is comparing two different measuring systems.

Answers started eating clicks. The Pew Research Center tracked 900 US adults across 68,879 unique Google searches in March 2025 and found that users who saw an AI summary clicked a traditional result in 8% of visits, against 15% for those who did not. That is not a reason to abandon informational keywords. It is a reason to stop treating volume as a proxy for traffic on any query a summary box can finish in three sentences.

The Custom Search JSON API closed to new customers. It was never a keyword tool, but it was the standard answer to "how do I get Google results legitimately," and half the tutorials for this topic still recommend it. Google's overview page now carries both the closure and the deadline: existing customers "have until January 1, 2027 to transition to an alternative solution." For the record, the pricing that is going away is 100 queries per day free, then $5 per 1,000 queries up to a 10,000-per-day ceiling.

The Bing Search API is gone. The recommended fallback in every older comparison no longer exists as a product. Microsoft's documentation for the Bing Web Search API now sits under a previous-versions path and is flagged as retired and archived in its own metadata. If a guide tells you to route around Google by buying Bing results from Microsoft, that guide is out of date.

Google announced a Trends API in alpha. The Search Central blog carries a post titled "Introducing the Google Trends API (alpha): a new way to access Search Trends data," dated July 2025. We could not read the body of that post through automated fetching, and developers.google.com/trends returns 404, so anything specific you read about its quotas or history depth is worth checking against Google before you build on it. What is checkable today is that there is still no self-serve, publicly documented Trends API sitting where an API would sit.

The surfaces, and what each one costs to reach

The suggest endpoint is the one that still behaves like the open web. It answers without a key, returns clean JSON with client=firefox, and localizes on hl and gl. It is also undocumented, which means Google owes you nothing and can change the response shape on a Tuesday.

Related searches and People Also Ask are not endpoints. They are blocks inside a results page, so collecting them is SERP scraping with all of the cost that implies, and that cost went up tenfold in September 2025 for anything deeper than page one.

The Trends trending feed is the surface nobody in this genre mentions. Google serves a per-country RSS feed of rising searches. Fetched on 13 August 2026, https://trends.google.com/trending/rss?geo=US returned valid RSS with items carrying an approx_traffic value, and the top five that afternoon read "kuril islands", "dawn marie", "susan collins maine voter poll", "jennifer stone" and "clacton by election", at 100+, 100+, 100+, 100+ and 500+ respectively. Note the shape of those numbers. The traffic bands are bucketed, so this is a discovery feed for catching a term the day it moves, not a volume source. It costs nothing, needs no key, and updates faster than any keyword database.

Keyword Planner through the API has a rate limit that decides your architecture. Google publishes it plainly in the API quota documentation: GenerateKeywordIdeas, GenerateKeywordHistoricalMetrics and GenerateKeywordForecastMetrics are limited to 1 QPS per customer ID. The access-level page adds the daily ceiling: Basic access allows 15,000 operations per day, and Standard access lifts the daily cap entirely while leaving the per-second limit where it is. Fifteen thousand operations at one per second is about four hours and ten minutes of continuous polling. The scarce resource on this path is the account, not the address.

Search Console is the only surface that knows your site

Every other source tells you what the world searches. Search Console tells you what the world searched and then saw you. That makes it the calibration anchor for everything else you collect, and it is the least-used piece of the whole toolkit because it needs OAuth and most keyword tutorials stop at the point where a key is required.

The Search Analytics API is free, documented, and generous by the standards of this article. The query reference puts rowLimit at "1–25,000; Default is 1,000", pages with a zero-based startRow, and lets you choose dataState between final and all depending on whether you want settled numbers or fresh ones. The quota page gives 1,200 QPM per site and 1,200 QPM per user, inside a per-project ceiling of 40,000 QPM. Those numbers are far above what a keyword workflow needs, which is a pleasant change of subject.

Code below was written against Python 3.11.15 and requests 2.34.2, and expects an OAuth access token with the webmasters.readonly scope:

import requests

ENDPOINT = "https://searchconsole.googleapis.com/webmasters/v3/sites/{site}/searchAnalytics/query"
PAGE = 25000  # documented ceiling for rowLimit

def gsc_queries(site_url, token, start_date, end_date):
    """Page through every query row Search Console will hand over."""
    rows, start_row = [], 0
    url = ENDPOINT.format(site=requests.utils.quote(site_url, safe=""))
    while True:
        body = {
            "startDate": start_date,
            "endDate": end_date,
            "dimensions": ["query"],
            "rowLimit": PAGE,
            "startRow": start_row,
            "dataState": "final",
        }
        resp = requests.post(url,
                             headers={"Authorization": f"Bearer {token}"},
                             json=body, timeout=60)
        resp.raise_for_status()
        batch = resp.json().get("rows", [])
        rows.extend(batch)
        if len(batch) < PAGE:
            return rows
        start_row += PAGE

if __name__ == "__main__":
    data = gsc_queries("https://example.com/", "<access-token>",
                       "2026-05-01", "2026-07-31")
    print(f"{len(data)} query rows")
    for row in data[:5]:
        q = row["keys"][0]
        print(f"{q!r}: {row['impressions']} impressions, {row['clicks']} clicks")

Two things about that loop matter more than the code. The first is that the API "does not guarantee to return all data rows but rather top ones," in Google's own phrasing, so a short final page is a stopping condition and not a promise of completeness. The second is the anonymized tail described earlier. Rows exist for rare queries; the query text does not.

The calibration itself is four lines of arithmetic and it is worth more than most paid features. Take the queries where you already rank in the top three. For each one, you have a real impression count from Search Console and a modelled volume from whichever vendor you pay. Divide one by the other, take the median across fifty queries, and you have a correction factor for your niche, your country and your seasonality. Apply it to vendor volume on keywords you do not rank for yet, and your forecasts stop being somebody else's national averages. We are not publishing our own factor, because it is a property of our topics and it would mislead you on yours.

The caveats are real and short. Impressions are not searches, so the comparison only holds where you are visible for nearly every search of that term. Position one to three is the safe zone; below that the ratio measures your ranking, not the demand. And it tells you nothing about queries where you never appear at all, which is the entire reason the rest of this article exists.

Harvesting suggestions without pretending it is an API

Autocomplete stays the cheapest source of genuine phrasing because there is a lightweight endpoint that returns clean JSON:

https://suggestqueries.google.com/complete/search?client=firefox&q=YOUR+QUERY

With client=firefox the response is a two-element array: the echoed query, then the list of completions. To go wide, append each letter and digit to the seed so Google completes "web scraping a...", "web scraping b..." and so on. The Python below iterates seeds and expansions, backs off when Google says no, dedupes, and writes CSV.

import csv
import time
import random
import string
import requests

SUGGEST_URL = "https://suggestqueries.google.com/complete/search"
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}

def get_suggestions(query, lang="en", country="us", attempts=4):
    params = {"client": "firefox", "q": query, "hl": lang, "gl": country}
    delay = 5.0
    for attempt in range(attempts):
        resp = requests.get(SUGGEST_URL, params=params,
                            headers=HEADERS, timeout=10)
        if resp.status_code in (429, 503):
            time.sleep(delay)          # back off, do not retry immediately
            delay *= 2
            continue
        resp.raise_for_status()
        return resp.json()[1]
    raise RuntimeError(f"throttled on {query!r} after {attempts} attempts")

def expand(seed):
    """Yield the seed plus 'seed a' ... 'seed z' and 'seed 0' ... 'seed 9'."""
    yield seed
    for ch in string.ascii_lowercase + string.digits:
        yield f"{seed} {ch}"

def harvest(seeds, lang="en", country="us"):
    found = set()
    for seed in seeds:
        for query in expand(seed):
            try:
                for suggestion in get_suggestions(query, lang, country):
                    found.add(suggestion.lower().strip())
            except (requests.RequestException, RuntimeError) as e:
                print(f"skip '{query}': {e}")
            time.sleep(random.uniform(1.0, 3.0))  # jitter between calls
    return sorted(found)

if __name__ == "__main__":
    seeds = ["web scraping", "keyword research", "price monitoring"]
    keywords = harvest(seeds)
    with open("google_suggestions.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["keyword"])
        for kw in keywords:
            writer.writerow([kw])
    print(f"Saved {len(keywords)} unique suggestions")

Three notes before you run it at volume. The endpoint is undocumented, so treat it as best-effort rather than a supported service; it answered normally on 13 August 2026, and that is the strongest statement anyone can honestly make about it. The hl and gl parameters change the results, not just the language of them, Point the same seed at another country and the list comes back different. And the jitter earns its place. Firing a tight, evenly spaced loop from one address is the fastest way to be told to stop.

If the results should land somewhere a team can use the same afternoon, swapping the CSV writer for a push straight into Google Sheets is a small change. The endpoint parameters, the alternative hosts, and what each response field carries are pulled apart in more depth in the autocomplete scraper walkthrough.

Collecting in rounds, and knowing when to stop

One habit separates a thorough keyword set from a shallow one: collect in rounds. Pull an initial batch, sort by whatever demand signal you trust, promote the strongest terms to seeds, and run them back through the harvester. Each pass reaches phrases the previous pass could not, because the seed itself has changed.

The part everyone skips is the stopping rule, and it is arithmetic rather than judgement. Every seed costs exactly 37 requests in the script above: the bare seed, twenty-six letters, ten digits. Three hundred seeds is 11,100 requests, and at an average two-second gap that is a little over six hours of wall time. Round two, with a fresh three hundred seeds promoted from round one, costs the same again while returning fewer new uniques, because the sets overlap more each time.

So measure the overlap. Record new uniques per round as a share of the set so far. When a round costs six hours and adds two percent, the next round is not a discovery decision, it is a budget decision, and you should make it as one. Two or three rounds is where most projects land, not because three is a magic number but because that is where the yield curve flattens for a normal commercial niche.

Region targeting is the other multiplier, and it multiplies in the wrong direction. Every additional hl and gl pair reruns the whole tree. Twelve country-language pairs against three hundred seeds is 133,200 requests, which is a different class of job than the one you started. Pick the markets you actually sell in before you pick the seeds.

Getting past Google's defenses

Google fights automated collection, and the fight is not uniform across surfaces. The suggest endpoint is tolerant, cheap and rarely challenges a well-paced client. The results page is where the money goes, and since the hundred-result parameter died, it goes ten times faster than the figures in older guides suggest.

• Rotating proxies. Spread requests across many addresses so no single one looks like a loop. Rotating residential proxies are the standard instrument; datacenter ranges get flagged by ASN long before they get flagged by behavior.
• User-Agent rotation. Vary the client string instead of repeating one, and keep it coherent with everything else you send. A user-agent claiming the newest Chrome, attached to a TLS handshake no Chrome build has ever produced, is not a disguise.
• Delays and jitter. Randomize the gap, as the harvester above does. Mechanical timing is the signal that survives every other precaution.
• Back off rather than retry. A 429 answered instantly with the same request is a confession. Double the wait, as in get_suggestions, and stop after a few attempts. When a challenge does appear, solving CAPTCHAs programmatically works and adds both cost and fragility.

For anything past small autocomplete pulls, the honest comparison is between engineering time and a SERP API: a service that hides proxies, headless browsers and challenges behind one call and returns structured JSON. It costs money per request and it removes the entire anti-block treadmill, including the part where a markup change breaks your parser at two in the morning. Prices for the current field, read the same week as everything else here, are laid out in the Google results scraping guide.

Tools, with prices read on 13 August 2026

Every price below came off the vendor's own pricing page on 13 August 2026, monthly billing unless stated.

Two cells say "not read" instead of carrying a number. Both pages build their prices in the browser and return nothing useful to a plain fetch, and copying a figure out of somebody's review post and presenting it as read is how wrong prices propagate for years. If you need those two, open them yourself.

Three things in that table contradict the way this comparison usually gets written, including in the earlier version of this article. Ahrefs and Semrush are filed under "premium" everywhere, and both now have an entry tier under $140 that did not exist when most of those comparisons were written. AlsoAsked and AnswerThePublic get named in the same breath as People Also Ask tools, and only one of them is: AlsoAsked's own site describes what it sells as "live 'People Also Asked' data", while AnswerThePublic's homepage today describes questions "across different platforms" and no longer states a source at all. And the cheapest programmatic route to Planner numbers is not Google. DataForSEO's $0.06 per thousand keywords works out to $60 per million in the standard queue, $90 in live mode, with no developer token, no 1 QPS limit and no Ads account to keep alive.

For a small site the free stack still wins on value: Search Console for what you already get, Keyword Planner for demand, suggest for phrasing, Trends and its trending feed for timing. The paid tools earn their money on two jobs specifically, competitor gap analysis and scheduled collection, and both of those are worth paying for only once you are acting on the output weekly.

What breaks at scale

Dedupe is harder than lowercasing. Google groups near-variants in Planner and does not group them in suggest, so your CSV will hold "web scraping tool" and "web scraping tools" as two rows carrying one volume between them. Decide early whether your unit is the string or the concept, and write it down, because half a keyword set built each way is worse than either.

Every set is a snapshot. Suggestions churn continuously. A phrase list without a collection date is a claim you cannot check three months later, so store query, surface, country, language and collected_at as the key from the first row onward. Retrofitting that column is a data-migration afternoon nobody enjoys.

The fan-out multiplies faster than the value. Seeds times 37 times markets, as computed above. Cut markets before you cut seeds.

Rate limits bind in different places on different paths. On the suggest endpoint the constraint is the address, and proxies solve it. On the Ads API the constraint is the customer ID at 1 QPS with 15,000 operations a day on Basic access, and proxies solve nothing at all, because the limit is attached to the account. Teams routinely discover this after building a proxy pool they did not need.

Volume for a large list costs real money or real time. A million keywords through DataForSEO's standard queue is $60, billed in tasks of up to a thousand keywords each. The same million through your own Ads account carries no charge and runs into two ceilings instead: 1 QPS per customer ID, and 15,000 operations a day on Basic access. How many keywords ride along inside one request therefore decides the whole schedule, and that is the one number the quota page does not give you. Pick based on which resource you actually have, and measure the batch size yourself before you promise anyone a date.

Questions Google's own data cannot answer

Zero-volume is not zero demand. Trends only reports popular terms, and everything below its threshold arrives as a flat zero. A term that three hundred people search per month across a country will read as nothing in Trends and as a range in Planner, and it can still be the best-converting page you publish this year.

New terms have no history. Anything named in the last few weeks has no trailing average to average, and both Planner and vendor databases will be quiet about it. The trending RSS feed and plain suggest are the only two surfaces here that see a phrase the week it appears.

Search Console cannot describe a query you have never appeared for. It is the highest-quality data in the stack and the narrowest. Used alone it will keep pointing you deeper into what you already rank for.

The tail Google counts is the tail Google will not name. The is_anonymized_query flag means the volume you most want to discover is precisely the volume your own analytics is required to withhold.

Not all demand goes through a search box. Marketplace search, YouTube, app stores and in-app discovery generate intent that never appears in any Google surface, and the share of it grew every year that assistants got better at answering without a link. A keyword set built purely from Google is a map of one channel, drawn accurately.

Is it legal to scrape Google keywords?

The data is public and the access is contractual, which is why this sits in grey rather than in black or white. Google's Terms of Service discourage automated access, so collection can breach the terms even where nothing about the data itself is protected.

One correction is overdue, and this article's earlier version carried the error too. hiQ Labs v. LinkedIn gets cited everywhere as the case that made scraping public data lawful. It is not that case. Both Ninth Circuit rulings were preliminary-injunction decisions about likelihood of success, and the 2019 opinion was vacated by the Supreme Court and remanded. On the merits hiQ lost: LinkedIn won summary judgment on breach of contract in November 2022, and in December 2022 hiQ accepted a $500,000 judgment and a permanent injunction barring it from scraping LinkedIn at all. The durable lesson is narrower than the popular one. The Computer Fraud and Abuse Act is a weak instrument against logged-out collection, so platforms sue on contract instead, and on contract they win.

For keyword work the practical position is comfortable and the reasoning matters more than the comfort. Suggestions and volumes carry no personal data, the surfaces are public and unauthenticated, and nothing in the process requires defeating a login. Stay on that side of the line: no personal data, no gated content, honest rate limits, and your own legal advice when the money is real. This is not it.

What to build first

Order matters more than tooling, and the order most teams use is backwards. Start with Search Console, because it is free, authenticated, honest about its own gaps, and it tells you which of your existing pages are one position away from mattering. Add Keyword Planner next for demand on terms you do not own yet, accepting the billing form as the price of entry. Then run suggest for phrasing, in the two or three markets you actually sell in. Only after those three produce something you are acting on weekly does a paid keyword database start paying for itself, and by then you will know exactly which of its columns you need.

Where the whole pipeline should be someone else's problem, it can be: continuous Google SERP scraping is a deliverable rather than a project, a managed extraction service makes sense when the sources multiply faster than the engineering time, and data as a service is the right shape when what you want is a scheduled file rather than a system to maintain. The keyword set then feeds the work it was collected for: site structure and page mapping, title and description generation, and rank tracking once the pages are live.

Whatever you build, record where each number came from and on what date. Four surfaces, four units, one column. That is the whole failure mode, and it is the cheapest one to avoid.