Keyword Scraper: Build and Cluster a Semantic Core

A semantic core — the keyword map behind a site — is a structured, cleaned, and grouped set of search queries tied to specific pages. It isn't just a list of keywords; it's the foundation of an entire SEO strategy. You build your site structure from it, write your content against it, set your meta tags, and plan your paid campaigns. For a large project you can't assemble a quality core by hand — the volumes are too big and it's too easy to miss important phrases. That's why a keyword scraper does the heavy lifting, pulling queries from several sources at once and merging them.

This article is part of our series on scraping for SEO and ties together the individual pieces on Google Keyword Planner, autocomplete suggestions, and competitor research.

Sources for keyword scraping

A complete core isn't scraped from one place but from several complementary sources:

• Google Keyword Planner — baseline demand data: search volume, competition, and forecasts. It's free with a Google Ads account and is the standard starting point in the US, UK, AU, and CA markets.
• Google autocomplete (Suggest) — the "live" phrasings straight from the search box, including long and conversational queries. This is where you scrape SEO keyword suggestions real people actually type.
• People Also Ask & related searches — question-shaped queries and adjacent topics Google surfaces on the results page. Great for FAQ sections and long-tail coverage.
• Competitor keywords — the queries your niche leaders rank for. Pulled from tools like Ahrefs, Semrush, or Moz, they surface phrases you'd otherwise miss.
• Google Search Console & GA4 — the queries already bringing you visitors. The cheapest wins hide here: pages ranking on page 2 that a little work can push to page 1.

The more sources you combine, the more complete the core. But raw collection is only half the job.

The keyword workflow, stage by stage

Collection. Starting from a handful of seed terms ("markers"), the scraper pulls phrases from every source. Depth, iteration, and stop-words matter at this step — get them wrong and you end up with either junk or gaps.

Cleaning. You strip out irrelevant, junk, and "empty" phrases. Autocomplete pulls in particular need a volume check, because search engines auto-generate suggestions that nobody actually searches for.

Volume enrichment. Every phrase gets a search volume attached (including exact-match figures) so you understand real demand, not vanity counts.

Clustering. Queries are grouped by meaning and by SERP overlap — phrases that can rank on a single page get merged into one cluster. Those clusters become your site structure: one cluster, one landing page.

Scrape SEO keyword suggestions yourself

You don't need a paid tool to harvest autocomplete. Google exposes a lightweight suggestion endpoint that returns JSON, and a short script can expand one seed into hundreds of long-tail phrases by appending each letter of the alphabet:

import requests
import string
import time

def scrape_suggestions(seed: str, lang="en", country="us") -> set[str]:
    endpoint = "https://suggestqueries.google.com/complete/search"
    found = set()

    # seed, then "seed a", "seed b", ... to fan out the long tail
    queries = [seed] + [f"{seed} {c}" for c in string.ascii_lowercase]
    for q in queries:
        params = {"client": "firefox", "q": q, "hl": lang, "gl": country}
        resp = requests.get(endpoint, params=params, timeout=10)
        if resp.ok:
            _, suggestions = resp.json()
            found.update(suggestions)
        time.sleep(0.5)  # be polite; avoid hammering the endpoint

    return found

for phrase in sorted(scrape_suggestions("running shoes")):
    print(phrase)

Run that across a list of seeds and you have raw material for thousands of queries in minutes. At scale, though, Google rate-limits and blocks repeat callers, so a production keyword research scraper needs rotating proxies and backoff — the same infrastructure any serious scraper relies on.

Tools for collecting and processing keywords

If you'd rather not build the pipeline yourself, mature platforms cover the whole workflow:

For clustering specifically, dedicated tools (or the built-in clustering in Serpstat, SE Ranking, and Keyword Insights) group queries by how much their top-10 SERPs overlap — the most reliable signal that two phrases belong on the same page.

Why SERP-overlap clustering works

Two keywords belong together when Google ranks the same pages for both. If "buy running shoes" and "running shoes online store" share, say, five or more of the same top-10 URLs, one page can realistically rank for both — so they go in one cluster. If their SERPs barely overlap, they need separate pages, no matter how similar the words look. This is why keyword clustering leans on scraped SERP data rather than on word similarity alone.

Collecting seeds is cheap; scraping accurate, up-to-date SERPs at volume is the hard part — that's where a dedicated Google SERP scraping service earns its keep, feeding clustering with fresh, geo-targeted results.

How the core connects to the rest of SEO

Keyword scraping leans on SERP analysis — who ranks at the top and how clusters group together. Once the core is built, it flows into rank tracking so you can monitor how each cluster's landing page performs over time. And the raw material — clean, deduplicated queries — is only as good as the data normalization behind it: consistent casing, deduped near-duplicates, and stripped tracking parameters.

A sensible starting point is the two cheapest, highest-signal sources: Google Keyword Planner volumes and autocomplete suggestions.

FAQ

Is scraping keyword suggestions legal? Autocomplete and SERP data are publicly served, and scraping public data is generally defensible in the US (see hiQ v. LinkedIn and the narrowing of the CFAA). Stay within reasonable request rates, respect each service's terms, and avoid collecting personal data, which brings GDPR/CCPA into play.

How many keywords should a semantic core have? It depends on the site. A small local business might need a few hundred well-clustered queries; a large e-commerce catalog can run into tens of thousands. Completeness matters less than clean clustering — every cluster should map to a page you can actually build.

Scraper or paid tool — which should I use? For a one-off small project, a script plus Keyword Planner is plenty. For ongoing, large-scale research across competitors and SERPs, a paid platform (or a managed scraping service) saves weeks and dodges the proxy/CAPTCHA arms race.

Building and maintaining a keyword scraper — proxies, autocomplete endpoints, SERP collection, clustering — is a project in itself. If you'd rather receive clean, clustered keyword and SERP data on a schedule, scraping.pro offers SERP scraping and custom data as a service feeds, so your team can focus on strategy instead of infrastructure.