Rank tracking means capturing the positions your site holds in search results for the queries that matter. Without regular monitoring you can't tell whether your SEO is actually working: rankings are the headline metric that shows whether a site is climbing or slipping, which queries are paying off, and where competitors are overtaking you. Checking positions by hand across a large keyword set is impossible, so people use a rank tracker — software that automatically pulls the results from search servers in specified locations.
This article continues our web scraping for SEO series. We'll cover how rank tracking works, what to watch for, whether to build or buy, and how to stand up your own tracker that scrapes Google positions at scale.
Why you scrape rankings instead of checking by hand
If you check positions manually in your browser, the result is skewed by personalization: the search engine mixes in your past queries, history and location. A rank tracker pulls positions from "clean" servers in a specific region, with no personalization, so the data is objective. Add automation on top: the tool checks thousands of keywords on a schedule, charts the trend over time, and keeps the history.
Manual checks in incognito mode are only justified for 5–10 queries. Anything larger is a job for a scraper.
What matters when you check google rankings
Location and device. Results differ between cities and between desktop and mobile. So you track positions per target location, separately for desktop and mobile. For a local business, a ranking in "London" and "Manchester" can be worlds apart; for a national brand you might sample several metros and average them.
Search depth. Top 100 is usually enough, but large projects go deeper. Note that engines increasingly use continuous scrolling instead of fixed 10-result pages, so "page 2" is a softer concept than it used to be — track by absolute position, not page number.
Frequency. Results shift daily, especially during algorithm updates. For active projects, positions are pulled daily or several times a week.
Tying to landing pages. It's useful to track not just the position but which page ranks for a query — that surfaces "cannibalization," where two of your pages compete for the same keyword.
The engine mix. Google dominates most Western markets, so a rank tracker is Google-first. But if you serve regions where a second engine has real share — Yandex in parts of Eastern Europe and Central Asia, Baidu in China, Naver in South Korea — track those too, since their results and ranking factors differ from Google's.
Build vs buy: ready-made rank trackers
If you just need dashboards and don't want to maintain scraping infrastructure, an off-the-shelf tracker is the fast path.
- AccuRanker and Nightwatch — dedicated rank trackers built for speed and large keyword sets, with daily updates, granular geo/device targeting, and API access.
- SE Ranking and Serpstat — mid-market all-in-one SEO suites with solid position tracking, competitor views, and white-label client reports.
- Ahrefs and Semrush — powerful, pricier platforms where rank tracking sits alongside backlink and keyword research data; a common choice for agencies and international SEO.
- Google Search Console — free, and the only source of your true average position and impressions as Google itself measured them. But it's your data only (not competitors'), it's averaged and delayed, and it caps query detail. Most teams pair GSC with a scraping-based tracker for a full picture.
The trade-off: SaaS trackers bill per keyword or per check, so at very large volumes — or when you need custom SERP features, unusual locations, or data piped straight into your own warehouse — the numbers push you toward building your own.
Building your own rank tracker
A DIY rank tracker has three moving parts: get the SERP, find your position in it, and store/alert on the result.
Getting the SERP: API vs direct scraping
Google has no official API that returns organic positions (Search Console gives your own averaged data, not a live SERP). So you have two options:
- A SERP API — third-party services (SerpApi, DataForSEO, Bright Data SERP API, Zenserp and others) that handle proxies, CAPTCHAs and parsing, and return the results as clean JSON. Easiest to start with; you pay per query.
- Direct scraping — you fetch the results pages yourself. Cheaper at volume, but you own the hard parts: rotating proxies with geo-targeting, realistic headers, CAPTCHA handling, and a parser that survives Google's frequent markup changes.
The example below uses a SERP-API pattern because it isolates the ranking logic from the block-avoidance plumbing. Swapping in your own SERP scraping layer later means replacing one function.
A minimal tracker in Python
import requests
API_ENDPOINT = "https://serpapi.example/search" # your SERP API of choice
API_KEY = "YOUR_KEY"
def fetch_serp(query, location="United States", device="desktop", depth=100):
"""Return an ordered list of result domains for a query."""
resp = requests.get(API_ENDPOINT, params={
"q": query,
"location": location,
"device": device,
"num": depth,
"engine": "google",
"api_key": API_KEY,
}, timeout=30)
resp.raise_for_status()
data = resp.json()
return [r["link"] for r in data.get("organic_results", [])]
def find_position(target_domain, results):
"""1-based rank of the first URL on target_domain, or None if not found."""
for i, url in enumerate(results, start=1):
if target_domain in url:
return i, url
return None, None
def track(target_domain, keywords, location="United States", device="desktop"):
report = []
for kw in keywords:
results = fetch_serp(kw, location=location, device=device)
rank, url = find_position(target_domain, results)
report.append({"keyword": kw, "rank": rank, "url": url})
print(f"{kw:<35} {rank if rank else '>100':>5} {url or ''}")
return report
if __name__ == "__main__":
track(
target_domain="example.com",
keywords=["running shoes", "trail running shoes", "best marathon shoes"],
location="London, United Kingdom",
device="mobile",
)
This returns the position of your domain for each keyword, in a given location and on a given device. To track competitors, run find_position for their domains against the same results — one SERP fetch covers everyone ranking on it.
Handling blocks in direct scraping
If you scrape Google directly instead of through an API, expect rate limiting and CAPTCHAs the moment you exceed a trickle of requests. The essentials:
- Geo-targeted rotating proxies. Residential or mobile IPs in the target country, rotated per request, are what make location-accurate results possible in the first place.
- The
gl,hlanduuleparameters. Google's country (gl), interface language (hl) and location (uule) query parameters let you pin the locale precisely. - Realistic pacing and headers. Randomized delays, proper
User-Agentand header ordering. Hammering from one IP is the fastest way to get blocked. - A CAPTCHA-solving fallback for when a challenge appears anyway.
This is exactly the plumbing a SERP API hides from you — and the reason many teams outsource it.
Storing and alerting
A rank number is only useful in context. Persist each run (date, keyword, location, device, rank, ranking URL) to a database or even a spreadsheet, then compute the day-over-day delta and alert on meaningful moves:
def diff_and_alert(today, yesterday, drop_threshold=5):
"""Flag keywords that fell by more than drop_threshold positions."""
prev = {r["keyword"]: r["rank"] for r in yesterday}
for r in today:
old, new = prev.get(r["keyword"]), r["rank"]
if old and new and (new - old) >= drop_threshold:
notify(f"'{r['keyword']}' dropped {old} -> {new}") # Slack / email webhook
Wire track() to a daily cron job or scheduler, push the deltas to Slack or email, and you have an automated rank-tracking pipeline that flags problems before they show up in traffic. Feeding the same history into a dashboard alongside Google Analytics 4 data lets you connect ranking moves to actual sessions and conversions.
How rankings connect to the rest of the work
Rank tracking is how you measure results — but the results themselves are built at other stages. Positions rise when the keyword research is done right, and the site structure and title/heading tags are dialed in. So it makes sense to launch rank monitoring right after technical fixes, to measure their effect.
The positions themselves are read out of the search results — and collecting those results in detail (analyzing snippets, SERP features, and the competitors in the top) is a job in its own right: SERP scraping at scale.
Let us run it for you
Standing up geo-accurate rank tracking across thousands of keywords — with proxies, CAPTCHA handling and a parser that keeps up with Google's changes — is real infrastructure to build and maintain. If you'd rather have the rank data than the plumbing, scraping.pro offers SERP scraping and ongoing rank monitoring as a done-for-you service, delivered as clean data or a dashboard on whatever schedule you need.