Google Scraper Tips: How to Scrape Google Safely

Late one night in the second week of September 2025, Google stopped honouring a single URL parameter, and the cost of tracking a keyword to position 100 went up tenfold. Until then, &num=100 returned a hundred results in one request. After it, ten. Every rank tracker, every SEO platform and every hand-rolled Google scraper on the planet had to start making ten requests where one had been doing the job.

Asked about it by Search Engine Land, a Google spokesperson gave one sentence: "The use of this URL parameter is not something that we formally support." That is the entire official record. Accounts disagree on the exact night, placing the change somewhere between 11 and 14 September 2025, because nobody was told in advance.

The damage turned up in other people's dashboards. Tyler Gargula of LOCOMOTIVE Agency looked at 319 properties and published the numbers on 18 September 2025: 87.7% of them saw Search Console impressions fall, and 77.6% lost unique ranking terms. Nothing about those sites had changed. What changed is that the scrapers inflating their impression counts could no longer see page four.

The legal caveat deserves more precision than it usually gets. Google's Terms of Service, in the version effective 30 July 2026, prohibit "using automated means to access content ... in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)". Open google.com/robots.txt and you will find Disallow: /search, with narrow exceptions for /search/about and /search/howsearchworks. The machine-readable instruction exists, it says no, and the terms point straight at it. That is a tighter argument than the vague "it violates the ToS" line most guides open with, and it is worth knowing which one you are actually up against.

While updating this article we pointed a robots-respecting fetcher at a live results page. It refused before sending the request. Everything below assumes you have made that judgement yourself, for your jurisdiction and your use case, and that you are not collecting personal data without a lawful basis.

What broke since the last guide you read about this

Three things changed inside eighteen months, and together they invalidate most of what is written on this subject.

JavaScript became mandatory on 15 January 2025. Google's statement at the time said the requirement let it "better protect" against "bots and evolving forms of abuse and spam" while delivering "the most relevant and up-to-date information", and put the affected share of its users at under 0.1%. For scrapers the affected share was 100%. A plain HTTP client that fetches /search and parses the returned HTML no longer lands on results.

This article's own previous version printed a requests plus BeautifulSoup function and said it worked until it did not. It did not work at all. That code had been dead for the better part of a year before that version was published, and any tutorial still showing you requests.get("https://www.google.com/search", ...) with a rotated User-Agent is describing 2024.

num=100 stopped working in September 2025. Values above 10 are ignored and you get ten organic results per request no matter what you ask for. This is the parameter most older guides open with, and its death changes the shape of every budget downstream.

Google's own search API is closing. The Custom Search JSON API page now carries a flat notice: "The Custom Search JSON API is closed to new customers." Existing customers "have until January 1, 2027 to transition to an alternative solution." The replacement Google names is Vertex AI Search, described on the same page as "a favorable alternative for searching up to 50 domains", which is site search rather than web search. If you are starting a project today, the official door is already shut.

Why scrape Google at all

The results page is where demand is visible before it reaches anyone's analytics. Rank tracking, competitor monitoring, harvesting "People also ask" questions, local pack and shopping data, and measuring the shape of the SERP itself all start there.

That last one is why people who never cared about SEO tooling now want SERP data. Similarweb put AI Overviews on 43% of Google searches as of July 2026, against 15% a year earlier; Semrush Sensor, watching a different sample by direct observation rather than panel modelling, reads closer to 48%. SparkToro's June 2026 study found 68% of US Google searches ending without a click, with 276 clicks per 1,000 searches reaching the open web. None of that is measurable from your own logs. An impression that never becomes a visit leaves no trace on your server, so the only way to see the surface your customers are looking at is to go and look at it.

If ranking data specifically is what you are after, SEO scraping and rank tracker scraping go further into what to collect and how often.

The URL is still the interface

Every Google query is a URL with parameters, which is the only reason automating it is possible at all. A search for white doves is:

https://www.google.com/search?q=white+doves

None of the parameters below is documented by Google for this purpose. They are observed behaviour, they change without notice, and one of them changed in the middle of this article's life.

• q — the query, URL-encoded. Search operators work here.
• hl — interface language, ISO 639-1 (hl=en). Also nudges how the query is interpreted.
• gl — country of the search, ISO 3166-1 alpha-2 (gl=us). Changes which country's results you get, and matters more than most people expect.
• start — pagination offset. start=10 is page two, start=20 page three. Ten results per page, no exceptions.
• num — dead since September 2025. Pass num=100 and you get ten results and no error.
• udm — surface selector, and the most useful recent addition for anyone parsing organic results. udm=14 returns plain web results with no AI Overview and none of the other SERP furniture. The third-party udm14.com exists only to prepend it, and describes it as "a quick way to get an AI-free search without any extra work". udm=50 sends you into AI Mode instead. If you are tracking ten blue links, udm=14 removes a whole class of parsing problems at zero cost, and almost no scraping guide mentions it.
• uule — a base64 canonical location string, city-level rather than gl's country level.
• tbm and tbs — vertical (isch, nws, vid, shop) and time or sort filters (qdr:d for the last day, sbd:1 for sort by date).
• lr and cr — restrict by the language a page is written in and by the country it is hosted in. Both are different from gl, and mixing them up produces data that looks fine and is wrong.
• safe, filter, pws — SafeSearch, duplicate clustering (filter=0 shows the clustered duplicates), and personalisation (pws=0 turns it off).

Because these are plain URLs, you can generate a million of them from a template and a keyword file. Favour a scraper that accepts an external URL feed over one that wants queries written into the job definition, so that re-running a whole market is a file swap. Bright Data keeps a maintained list of these parameters if you want a second opinion on any of them.

The arithmetic nobody redid after September 2025

Take a modest brief: 5,000 keywords, tracked daily, to depth 30. That is three requests per keyword per day and 450,000 requests a month. Track the same 5,000 to depth 100 and it is ten requests per keyword and 1.5 million a month. Before September 2025 that second brief cost 150,000 requests.

One parameter carried a factor of ten on the largest line in the budget. Run your own version of that sum before you decide anything else, because it determines which of the paths below you can afford, and run it against current vendor prices rather than the ones you remember.

What Google actually sells you

Three doors, and none of them is the one most articles point at.

Custom Search JSON API. 100 queries a day free, then "$5 per 1000 queries, up to 10k queries per day". Closed to new customers, with existing users out by 1 January 2027. Two caps catch people who get in under the wire: the cse.list reference puts num at "integers between 1 and 10, inclusive", and states that "The JSON API will never return more than 100 results, even if more than 100 documents match the query, so setting the sum of start + num to a number greater than 100 will produce an error". Google's own API never sold you page eleven either. It also searches a Programmable Search Engine that you configure, not google.com, so the ordering it returns is not the ordering a user sees.

Grounding with Google Search, in the Gemini API. Priced at 5,000 free search requests a month shared across the Gemini 3.x models and $14 per 1,000 after that, with older models such as 2.5 Pro at $35 per 1,000 grounded prompts. What comes back is model prose with citations. There are no positions, no ranked list, and no way to ask what sits at number seven.

Search Console API. Free, accurate, and restricted to properties you own and have verified. The documented quotas are 1,200 queries per minute per site and 40,000 per minute per project. It tells you what Google showed for your own pages and nothing whatsoever about anyone else's.

There is no supported way to ask Google for a ranked list of results for an arbitrary query. That absence is the entire reason the next section exists.

SERP APIs, with prices read on 13 August 2026

Every figure below came off the vendor's own pricing page or documentation on 13 August 2026. Promotions were running on two of them, and prices in this market move.

The spread between the top and bottom of that table is roughly eighty to one, and it is not a quality gradient. Read what each vendor counts before comparing anything. A "result" at Oxylabs is a page. A credit at ScraperAPI is a twenty-fifth of a Google request. SerpApi's price includes the U.S. Legal Shield indemnity it advertises on every plan, which is a different product from a proxy pool with a JSON parser bolted on, and you should decide on purpose whether you are buying it.

Against all of that, the do-it-yourself floor is proxy bandwidth. Bright Data's residential proxy pricing listed $8 per GB pay-as-you-go on 13 August 2026, with committed plans from $499 a month for 141 GB and a 50%-off promotion live on the page. A JavaScript-rendered results page is not a 30 KB HTML document, and you pay for every byte, twice on a retry. We did not measure that page weight for this update, for the robots.txt reason above. Measure it on your own traffic before assuming the build is cheaper than the buy.

Proxies, pacing, and which parts of the folklore survive

Residential or mobile addresses, not datacenter ranges. This is the one piece of old advice that has aged well and got more expensive. Datacenter address space is identifiable by ASN and treated accordingly. Residential exits share space with real subscribers, which makes them costly to ban without collateral damage, and that is exactly what you are paying per gigabyte for. The older recommendation of generic "high-anonymous proxies", and of rotating proxies chosen on price alone, was written for a web that no longer exists. If you cannot sustain your target rate on clean addresses, lower the rate rather than pushing harder through the challenges. Pushing burns addresses and poisons the data you do collect, because a session that limps past a challenge often returns a degraded results page rather than a normal one.

The numeric rules of thumb are folklore. Figures like "50 to 150 proxies" and "under 500 requests per IP per day" circulate endlessly and have no published source behind them. Google documents no rate limit for /search, for the obvious reason that it does not want you there. Every specific number in circulation traces back to a vendor blog or a forum post. Tune empirically, back off hard on the first challenge, and treat any article quoting a precise safe rate as guessing in public.

User-Agent rotation buys much less than it used to. With JavaScript mandatory, your client is a real browser engine or it is nothing, and a browser announces hundreds of things about itself before your parser runs. Coherence beats disguise: a residential exit in Warsaw driving a browser that reports en-US, an America/Chicago timezone and 64 CPU cores is not a disguise, it is a signature. Set the browser context's locale and timezone to match the exit you are using.

Cookies. Rotate or clear them together with the address, so one identity does not accumulate thousands of queries. The blanket "always disable cookies" rule is crude. The goal is that no single session looks like a machine working through a keyword list.

Concurrency. Modest and steady beats bursty. Parallel floods are the fastest route to a challenge, and the throughput you gain is smaller than the throughput you lose while blocked.

Geography, end to end. Set gl and hl, add uule if you need a city, and route through an exit in that country. A US address asking for gl=uk returns something, and that something is not what a British user sees.

A challenge is information, not an obstacle. A CAPTCHA or an unusual-traffic interstitial means you have already been classified. The correct first response is to stop, rotate, and slow down. If you must continue through them, a CAPTCHA solving service will take the challenge off your hands, but a steady rate of challenges is a diagnosis of your pacing and your address pool rather than a line item to budget for. Rotate residential proxies on evidence, not on a timer, and retire any address the moment it starts drawing challenges.

Parsing what comes back

Google's markup is opaque by construction: short generated class names that carry no meaning, changed on Google's release schedule and announced to nobody. The claim that those names are randomised per request gets repeated constantly, and we could find no demonstration of it anywhere. The weaker true version is enough to plan around. The names are meaningless, and they will change while you are asleep.

We could not inspect live SERP markup for this update, and there is no point pretending otherwise. Disallow: /search means a compliant fetcher declines the URL. So what follows is structural advice rather than a list of current class names, which is the advice that ages well anyway.

Anchor on the things Google cannot change without breaking its own page: an h3 inside a result link, the heading-then-link-then-snippet relationship, ARIA roles, and anything expressed as structured data. Add udm=14 so the page you are parsing is the simple one. Then treat the parser as a monitored system rather than as a function.

The failure that costs money is the silent one. A parser whose selectors have drifted does not raise an exception. It writes an empty array, the job exits 0, and the dashboard shows a flat line that reads like a ranking collapse rather than a broken selector. Alert on the shape of the output instead: results per page below a floor, an empty snippet field on more than a small percentage of rows, position one unchanged across every keyword in the set. Catching that in an hour rather than a fortnight is the difference between one bad day and a month of history you have to throw away.

Here is the shape of a current approach. Everything below was written against Playwright for Python 1.62, the release listed at the top of the release notes on 13 August 2026.

from urllib.parse import quote_plus
from playwright.sync_api import sync_playwright

def serp(query, page=0, proxy=None, locale="en-US", tz="America/Chicago"):
    url = ("https://www.google.com/search"
           f"?q={quote_plus(query)}&hl=en&gl=us&udm=14&pws=0&start={page * 10}")
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True, proxy=proxy)
        ctx = browser.new_context(locale=locale, timezone_id=tz)
        pg = ctx.new_page()
        pg.goto(url, wait_until="domcontentloaded", timeout=20_000)
        # Any interstitial takes you off /search. Treat that as a stop signal,
        # not as something to retry in a loop.
        if "/search" not in pg.url:
            browser.close()
            raise RuntimeError(f"redirected to {pg.url}: rotate and slow down")
        rows = [{"title": a.locator("h3").first.inner_text(),
                 "url": a.get_attribute("href")}
                for a in pg.locator("a:has(h3)").all()]
        browser.close()
    if not rows:
        raise RuntimeError("zero results parsed: selector drift or a soft block")
    return rows

The last three lines matter more than the rest. An empty parse is an error condition, and code that returns [] quietly is code that will lie to you for a month. For selector technique in general, see CSS selectors for web scraping.

What breaks at scale

A hundred queries from a laptop is a script. Ten thousand a night is a system, and four things go wrong on the way from one to the other.

Retries multiply the bill without improving coverage. A blocked request retried three times costs four requests and returns one row, or none. At 25 credits a call, a 20% challenge rate does not cost you 20%. Budget on delivered rows and instrument the ratio of requests made to rows written, because that ratio is the one that quietly triples.

Wasted time compounds. One extra second per request across the 450,000-request month above is 125 hours of wall clock. That is the gap between a nightly job that finishes before the business wakes up and one still running at lunchtime, usually decided by a sleep() someone tuned once and never revisited.

Results are not reproducible, and that is not a bug you can fix. Two runs of the same query minutes apart, from different exits, will disagree. Turn off personalisation with pws=0, pin uule, gl and hl, run at the same hour, and you will still see drift. Store the full parameter set and a timestamp with every row, or your own history will not be comparable with itself, and you will spend a quarter arguing about whether a ranking moved.

Keep the raw response. Storage is the cheapest thing in this stack, and a parser fix six months from now is worth nothing if the pages are gone. Keep the HTML, or a compressed sample of it, so a repaired selector can be replayed over the period it was broken.

Where this stops working

AI Mode is a conversation, not a page. Under udm=50 there is no stable list of ten positions to record, and a follow-up question produces a different answer from the same starting query. If your metric is a rank, that surface does not have one.

Result counts are estimates and always were. The "About 12,300,000 results" line has never been a count you can reconcile, and keyword-difficulty maths built on it produces numbers with the texture of data and none of the properties.

Search Console impressions are not SERP positions. September 2025 proved that from the wrong direction. Impressions fell across most properties because scrapers stopped looking, and not because users did. Any before-and-after comparison spanning that fortnight is measuring the measuring instrument.

Below city level, uule is guesswork. It encodes a canonical location from Google's own geotargeting vocabulary, and that vocabulary may not go as fine as your local pack study needs.

Depth is thinner than it looks. Pagination past roughly start=90 mostly returns nothing worth having, which mirrors the hundred-result ceiling Google enforces explicitly in its own JSON API. The num=100 era let people pretend otherwise by fetching that ceiling in one call. The ceiling did not move; only the number of requests needed to reach it did.

The legal layer, and one myth worth retiring

hiQ did not win. The claim that hiQ Labs v. LinkedIn established a right to scrape public data turns up in nearly every article on this subject, and it is wrong in the part that matters. Both Ninth Circuit rulings were preliminary-injunction decisions about likelihood of success, and the 2019 opinion was vacated by the Supreme Court and remanded after Van Buren. On the merits hiQ lost: the district court granted LinkedIn summary judgment on breach of contract in November 2022, and in December 2022 hiQ accepted a $500,000 judgment and a permanent injunction barring it from scraping LinkedIn at all. The lesson is narrower than the slogan: the Computer Fraud and Abuse Act is weak against logged-out scraping, so platforms sue over contract instead, and they win.

GDPR does not care that the SERP is public. Names, profile URLs and anything that identifies a person carry obligations regardless of where you found them. Sixteen data protection authorities restated exactly that jointly in October 2024. A rank-tracking dataset is usually fine. A dataset built from People Also Ask results about named individuals is a different question with a different answer.

The market may open from a direction nobody was watching. On 2 September 2025, Judge Mehta's remedies decision in the US search antitrust case ordered Google to make certain search index and user-interaction data available to Qualified Competitors, and to offer search and search text ads syndication on terms consistent with its current commercial practices, under a technical committee and for six years. It is not a licence for you, and the class of qualifying firms is narrow. It does mean the number of parties who can legally resell Google-derived results is set to grow, which is a reason to keep the buy option under review rather than deciding once.

None of this is legal advice, and all of it moves.

Build it or buy it

The honest split has narrowed since this article was first written. Before 2025, a request loop and a bag of cheap addresses got a small team a long way, and maintenance was a selector or two a quarter. The floor now is a headless browser plus residential bandwidth, which is most of what a SERP API charges for, minus their parser maintenance and their volume discount on addresses. Build because you need something no vendor returns, not because the per-request price looks large next to a proxy invoice.

There is a third position between those columns, for the case where the data matters and the plumbing does not. A managed extraction service or a scheduled data-as-a-service feed hands over the rows and keeps this particular arms race off your roadmap entirely.

A dependable Google scraper is about a fifth parsing and four fifths not being noticed, and that ratio got worse in 2025. Generate your URLs in bulk, page with start, add udm=14 when you want ten blue links and nothing else, keep geography consistent from parameter to exit address, anchor the parser on structure, and alert on empty output rather than on exceptions. Then check the numbers again in six months. The ones here were read on 13 August 2026, and last time Google gave nobody any notice at all.