Scraping Google Autocomplete: Collecting Search Suggestions for SEO

Search suggestions — the query options Google shows right in the search box as you type — are one of the most valuable keyword sources in SEO. They are generated from real people's queries, refresh almost daily, and reflect the "live" phrasing of almost any topic. Building a Google autocomplete scraper is a standard step in assembling a complete keyword set.

This article is part of our series on web scraping for SEO. Here we will cover how suggestion collection works, which settings matter, the actual endpoints you can call, and the tools that do it for you.

Why suggestions are a separate source, not a clone of Keyword Planner

Many people assume that Google Keyword Planner covers all of demand. It does not. Keyword Planner shows search-volume statistics, but the suggestion database exists separately from the keyword-volume database and often contains phrasings that are not in Keyword Planner or are hard to "pull out." Suggestions are especially good at catching long-tail queries, conversational phrasing, and fresh trends. So in a full keyword research workflow, autocomplete and Keyword Planner complement each other rather than replace one another.

One important caveat: among the suggestions you will find auto-generated phrases that almost nobody actually searches. So after collection, always run the suggestions through a volume check — otherwise your keyword set fills up with dead weight.

How suggestion collection works

The logic is the same across all scrapers. You load a list of seed phrases, and the scraper appends characters to them and collects the drop-down options. The key parameters:

Character iteration. You add a space to the seed phrase and then, one by one, letters and digits: [a-z], [0-9]. Each character opens a new set of suggestions. The wider the iteration, the more complete the result — but also the more noise.

Crawl depth. At depth 1, you collect suggestions only for the original phrases. At depth 2, the phrases found in the first pass are fed back in and used to collect suggestions again. Depth 3 gives maximum coverage — high-, mid-, and a huge number of low-volume queries. For most tasks, depth 1–2 is enough; you use depth 3 when you need exhaustive coverage of a niche.

Region and language. Suggestions are localized, so set the collection region and language per project. With the raw endpoints this is the gl (country, e.g. us, gb, au, ca) and hl (interface language, e.g. en) parameters.

Stop words. A list of words that halt collection. If the scraper encounters a suggestion containing a stop word, it does not go "deeper" down that branch. This saves budget and cuts irrelevant phrases up front.

The suggestion endpoints (Google, Bing, YouTube)

You do not have to render the search page to get suggestions. Each engine exposes a lightweight autocomplete endpoint that returns a compact JSON array. These are the same endpoints the search box calls as you type.

Google Suggest returns a JSON array of [query, [suggestions...]]:

https://suggestqueries.google.com/complete/search?client=firefox&hl=en&gl=us&q=web+scraping

YouTube uses the same host with ds=yt:

https://suggestqueries.google.com/complete/search?client=firefox&ds=yt&hl=en&q=web+scraping

Bing exposes an OpenSearch JSON endpoint:

https://www.bing.com/osjson.aspx?query=web+scraping

All three return the same shape — the seed query followed by a list of completions — which makes them easy to unify in a single scraper.

A minimal Google autocomplete scraper in Python

The snippet below implements the two core ideas from above: character iteration (appending a–z and 0–9 to each seed) and depth (feeding discovered phrases back in). It hits the raw endpoint, so it needs no API key.

# pip install requests
import time
import string
import requests

ENDPOINT = "https://suggestqueries.google.com/complete/search"

def fetch_suggestions(query: str, gl: str = "us", hl: str = "en") -> list[str]:
    params = {"client": "firefox", "q": query, "gl": gl, "hl": hl}
    headers = {"User-Agent": "Mozilla/5.0"}
    resp = requests.get(ENDPOINT, params=params, headers=headers, timeout=10)
    resp.raise_for_status()
    # Response shape: ["query", ["suggestion 1", "suggestion 2", ...]]
    return resp.json()[1]

def expand(seed: str, alphabet=string.ascii_lowercase + string.digits) -> set[str]:
    found = set(fetch_suggestions(seed))
    for ch in alphabet:
        found.update(fetch_suggestions(f"{seed} {ch}"))
        time.sleep(0.3)  # be polite — random, throttled requests avoid blocks
    return found

def collect(seeds: list[str], depth: int = 2, stop_words: set[str] | None = None) -> set[str]:
    stop_words = stop_words or set()
    results: set[str] = set()
    frontier = list(seeds)
    for _ in range(depth):
        next_frontier: list[str] = []
        for phrase in frontier:
            for s in expand(phrase):
                if any(w in s for w in stop_words):
                    continue
                if s not in results:
                    results.add(s)
                    next_frontier.append(s)
        frontier = next_frontier
    return results

if __name__ == "__main__":
    keywords = collect(["web scraping"], depth=1, stop_words={"free download"})
    for kw in sorted(keywords):
        print(kw)

Swap the ENDPOINT and response index to point at Bing or YouTube and you have a multi-engine collector. At scale, add rotating proxies and randomized delays — Google will start returning empty results or challenges if you hammer the endpoint from one IP.

Tools for scraping suggestions: an overview

If you would rather not maintain a script, plenty of tools collect keyword suggestions for you:

Keyword Tool (keywordtool.io) — one of the most convenient online services for autocomplete. It pulls suggestions from Google, Bing, YouTube, Amazon, and more in one project, supports character iteration, and exports large, clean lists. A good fit when you need volume without wrangling proxies.

AnswerThePublic — visualizes autocomplete around questions and prepositions (how, what, vs, for, near), which is great for content and long-tail ideation.

Ahrefs and Semrush — full SEO suites whose keyword tools ingest autocomplete data and pair it with volume, difficulty, and SERP metrics. Convenient if you already work in these platforms.

Serpstat and Ubersuggest — mid-range all-in-one tools that also collect suggestions alongside volume and competition data.

Soovle — a free, lightweight tool that shows suggestions from several engines (Google, Bing, YouTube, Amazon, Wikipedia) side by side.

Scrapebox / A-Parser — desktop/bulk scrapers for very large jobs and non-standard automation, collecting suggestions across several engines at once. Powerful, but they need technical setup, your own proxies, and CAPTCHA handling.

Where the suggestions go next

Collected suggestions are raw material. Next you need to clean them, check volume through Keyword Planner, and merge them with your other sources as part of a full keyword research pass. Some of the phrases from suggestions will become the basis for new sections of the site — which means they influence your site structure and the title tags and headings you write on your pages.

And if you need to analyze not just the suggestions but the results ranking for those queries — who is in the top and how the snippets are built — that is the job of search results scraping, which we offer as a separate service. If you would rather receive a clean, deduplicated keyword set and SERP data on a schedule than build and babysit the scraper yourself, scraping.pro can run it as a done-for-you data extraction service.