Web Scraping for SEO Audits & Competitive Research

A URL sits in your sitemap. Your crawler fetches it and gets a clean 200. Search Console files it under Crawled - currently not indexed. Your access log says Googlebot last requested it in March. One page, four systems, four stories, and only one of the four is describing what a search engine actually did.

Pointing a scraper at your own site is not about collecting facts you could have looked up. It is about finding the places where things you shipped contradict each other, and then the places where your picture of the site contradicts the engine's. That is a different job from scraping somebody else's site, it runs on different data, and almost none of it needs a proxy.

This article stays on the site you own. The SERP side, the competitor side and the tool comparisons live in their own places and are linked where they belong: SEO scraping across all six directions for the overview, an SEO scraper for titles, metas and headings for the build, site crawler tools for the buy. If the mechanics of extraction are new to you, start with what is web scraping. Everything below was rechecked against vendor and Google documentation on 13 August 2026.

The site you own is a different data problem

Scraping a competitor gets you the intersection of what they publish and what a search result will show you. Scraping yourself gets you a superset, because three data sources exist for your property and for nobody else's.

Your server logs. Every request Googlebot made, with timestamp, status, byte count and user agent. No sampling, no lag, no aggregation. This is the only record of what the crawler did rather than what you hoped it would do.

Your Search Console and Bing Webmaster properties. Queries, impressions, positions, index state per URL, crawl statistics. Free, first-party, and unavailable for any domain you cannot verify.

Your CMS and database. The canonical answer to which pages should exist, which is the one thing a crawl can never tell you. A crawl finds what is reachable. It cannot find what should have been reachable and is not.

The access economics are equally lopsided. Google's Search Console API allows 1,200 queries per minute per site and per user, free. The Search Analytics method accepts a rowLimit between 1 and 25,000 with a default of 1,000, paged through startRow, which is twenty-five times what the interface table will show you. The URL Inspection API gives 2,000 queries per day and 600 per minute per property. The CrUX API is "limited to 150 queries per minute per Google Cloud project, which is offered without charge." No proxy pool, no CAPTCHA budget, no per-request fee.

Against that, a baseline for what normal looks like. The HTTP Archive's 2025 Web Almanac, whose SEO chapter published in January 2026, found title elements on 98.62% of desktop pages, meta descriptions on 67.7%, a non-empty H1 on 66%, and a canonical link on 68%. On 84.9% of sites robots.txt returned a 200. Those numbers are worth carrying into your first crawl, because they set expectations: missing titles are rare and mean something is broken, while a missing meta description is one page in three across the whole web and means almost nothing.

What a self-crawl is actually for

The field list every crawler produces is the same list, and reproducing it here would waste your time. Status, title, description, headings, canonical, robots directives, hreflang, structured data, internal links, word count. Fine. The list is not the point; the contradictions between fields are.

A URL in your sitemap that carries noindex. You told the engine this page matters enough to enumerate, then told it not to index the page. One of those statements is wrong and you shipped both.

A URL in your sitemap whose canonical points somewhere else. Same contradiction, quieter. The sitemap says "index this"; the canonical says "index that instead."

A noindex on a page disallowed in robots.txt. This one is worse than useless, and Google states the mechanism plainly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file." If the crawler cannot fetch the page, "the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." The combination that feels twice as safe is the combination that fails.

A 200 that is really a redirect. Your crawler asked for /product/42 and, because it follows redirects by default, recorded a 200 for a different URL. Record the URL you landed on separately from the URL you requested, or your inventory quietly merges pages.

A heading outline that skips a level. More on this below, because the popular version of the rule is wrong.

Everything else a crawl produces is inventory, and inventory is worth having. Contradictions are worth acting on.

The audit script, and what it fails on

Below is a self-crawl small enough to read in one sitting and honest about its exit code. It was written against Python 3.11, httpx 0.28.1 and selectolax 0.4.11 (the selectolax release of 15 July 2026), and run against a local fixture site before publication.

import sys
import xml.etree.ElementTree as ET
import httpx
from selectolax.parser import HTMLParser

SITEMAP = sys.argv[1]
NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def sitemap_urls(client, url):
    root = ET.fromstring(client.get(url).text)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall(".//s:sitemap/s:loc", NS):
            yield from sitemap_urls(client, loc.text)
    else:
        for loc in root.findall(".//s:url/s:loc", NS):
            yield loc.text

def audit(client, url):
    r = client.get(url)
    tree = HTMLParser(r.text)
    title = tree.css_first("title")
    levels = [int(n.tag[1]) for n in tree.css("h1,h2,h3,h4,h5,h6")]
    canonical = tree.css_first('link[rel="canonical"]')
    robots = tree.css_first('meta[name="robots"]')
    return {
        "url": url,
        "status": r.status_code,
        "landed_on": str(r.url),
        "title": title.text().strip() if title else "",
        "h1_count": len(tree.css("h1")),
        "level_jumps": [(a, b) for a, b in zip(levels, levels[1:]) if b > a + 1],
        "canonical": canonical.attributes.get("href") if canonical else "",
        "robots": (robots.attributes.get("content") or "") if robots else "",
    }

def contradictions(page):
    if page["status"] != 200:
        yield f"status {page['status']}"
    if page["landed_on"] != page["url"]:
        yield f"redirects to {page['landed_on']}"
    if not page["title"]:
        yield "no title element"
    if "noindex" in page["robots"]:
        yield "noindex on a URL you put in the sitemap"
    if page["canonical"] and page["canonical"] != page["url"]:
        yield f"canonical points at {page['canonical']}"
    if page["h1_count"] == 0:
        yield "no h1"
    for a, b in page["level_jumps"]:
        yield f"heading level jumps h{a} to h{b}"

failed = 0
with httpx.Client(follow_redirects=True, timeout=20,
                  headers={"user-agent": "own-site-audit/1.0"}) as client:
    for url in sitemap_urls(client, SITEMAP):
        page = audit(client, url)
        problems = list(contradictions(page))
        if problems:
            failed += 1
            print(f"{url}\n  " + "\n  ".join(problems))
print(f"\n{failed} URL(s) with contradictions")
sys.exit(1 if failed else 0)

Against a three-page fixture containing one clean page, one noindexed page with an H1 followed by an H3, and one page with no title and a foreign canonical, it prints:

http://127.0.0.1:8731/b.html
  noindex on a URL you put in the sitemap
  heading level jumps h1 to h3
http://127.0.0.1:8731/c.html
  no title element
  canonical points at http://127.0.0.1:8731/a.html

2 URL(s) with contradictions

Note what it does not flag. The third fixture page has two H1 elements and passes, which is deliberate and explained two sections down. Note also the sys.exit(1). A script that prints findings is an audit you will run twice and forget; a script that fails a build is a control.

Titles: the rule everyone repeats measures the wrong thing

"Keep titles under 60 characters" is the most repeated instruction in on-page SEO, and it is a proxy for a variable Google does not use. From Google's own title link documentation: "While there's no limit on how long a <title> element can be, the title link is truncated in Google Search results as needed, typically to fit the device width." Truncation is a width problem. Characters are not a unit of width.

We measured this on the 205 article titles in this site's own corpus, rendering each with Pillow 12.2.0 in Liberation Sans at 20 px. Liberation Sans is metric-compatible with Arial, so the widths are close to what a desktop result actually occupies, though the browser stack is not identical and the absolute pixel values should be read as a scale, not a specification.

The median title is 51 characters and 490 px. The range runs 33 to 76 characters and 318 to 701 px, and the two orderings disagree. Two titles of exactly 48 characters measure 424 px and 504 px, a 19% spread from character-identical strings. Better still, a pair that inverts the rule outright:

The 60-character title is 36 px wider than the 61-character one. Sort your inventory by character count and you will fix the wrong title first. Across the 205, picking any plausible width cut-off puts eight titles on the wrong side of a 60-character rule: six that are over 60 characters and comfortably narrow, two that are under and wide. That is 4% of a small corpus, and the proportion climbs with capitals, ampersands and words like "WordPress" that cost far more per character than "elite" does.

The character rule survives in a different form, and this is where it earns its keep. Zyppy's title-rewrite study by Cyrus Shepard, first published in 2022 and last updated in May 2026, examined 80,959 URLs across 2,370 sites and found Google rewrote 61.6% of titles. Broken out by length, titles of 51 to 60 characters were rewritten least often, 39% to 42% of the time. Titles over 60 characters were rewritten more than 76% of the time, and titles over 70 characters 99.9% of the time. Titles of 20 characters or fewer were also rewritten more than half the time, which is the half of the rule nobody repeats.

So the 60-character guideline is a decent predictor of rewriting and a poor predictor of truncation, and most articles present it as the second thing. Measure width when you care how the result looks. Count characters when you care whether Google will keep your wording at all. Two questions, two units.

Three more rules that do not survive their own documentation

Multiple H1 elements are not a defect. The WHATWG HTML standard permits several, and the conformance requirement is about sequence, not count: "Each heading following another heading lead in the outline must have a heading level that is less than, equal to, or 1 greater than lead's heading level." An H1 followed by an H3 is the violation. Two H1s are not. Earlier versions of this article listed "multiple H1s" among the things a crawl should flag, along with roughly every other audit checklist on the web, and the HTML sections spec does not support it. That is why the script above counts H1s only to catch zero, and flags jumps.

Your meta description is an input, not an output. Google's snippet documentation says snippets "are primarily created from the page content itself," and that the meta description gets used when "it might give users a more accurate description of the page than content taken directly from the page." The controls that do bind are nosnippet, data-nosnippet on a page fragment, and max-snippet:[number]. A missing description is not a bug on two thirds of the web; a description that contradicts the page is worse than none.

Your canonical is a preference. Google's duplicate URL consolidation guide is blunt about the strength of the signal: "While we encourage you to use these methods, none of them are required; your site will likely do just fine without specifying a canonical preference." Google identifies "which version of the URL is objectively the best version to show to users in Search" whatever you declare. The audit value is not in checking that a canonical exists. It is in checking that yours agrees with the one Search Console reports as the Google-selected canonical, and investigating every URL where those two differ.

FAQ and HowTo markup no longer earn what the checklists promise. This is a straight correction to the earlier version of this article, which told you to study competitors' FAQ and HowTo schema for rich results. Google removed HowTo rich results on 14 September 2023, noting "this rich result is no longer shown in search results, on both desktop and mobile devices." FAQ went the same way and the notice is dated: "As of May 7, 2026, FAQ rich results are no longer appearing in Google Search." The search appearance, the rich result report and Rich Results Test support were dropped in June 2026, with Search Console API support ending in August 2026. Keep the markup if it serves other consumers. Stop counting it as a click-through tactic, and stop scoring competitors on it.

Your crawler is not Googlebot

A self-crawl tells you what your server sent to a Python client. It does not tell you what a search engine saw, and four differences account for most of the gap.

Rendering happens later, in a separate pass. Google's JavaScript SEO guide describes crawl, render and index as distinct phases, with a queue between them: "The page may stay on this queue for a few seconds, but it can take longer than that." Rendering runs on an evergreen version of Chromium. Your httpx client renders nothing, so anything injected client-side is invisible to it, and anything that depends on a second render pass may be invisible to the engine for longer than you assume.

Googlebot is stateless. From Google's JavaScript troubleshooting page: "Local Storage and Session Storage data are cleared across page loads. HTTP Cookies are cleared across page loads." Google also says to "expect Googlebot to decline user permission requests," and that it "does not support other types of connections, such as WebSockets or WebRTC connections." Content behind a consent banner that stores acceptance in localStorage, or a listing that loads over a socket, is content the engine will not accumulate the way a user does.

Crawl budget is a real constraint on a small number of sites. Google's crawl budget guide scopes it to "Large sites (1 million+ unique pages) with content that changes moderately often (once a week)" and "Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)." Below that, the honest answer is that your crawl budget is not your problem. What is your problem is the feedback loop: "If the site slows down (latency increases or response times become longer), or responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429), the limit goes down and Google crawls less." A slow site gets crawled less, which delays the discovery of the fix.

Search Console's crawl view is partial by its own admission. The Crawl stats report covers 90 days, broken down by response, file type, purpose (discovery against refresh) and Googlebot type, with a caveat printed on the page: "The Crawl Stats report currently reports most crawl requests, but some requests might not be counted for various reasons." It reports requested URLs rather than canonical ones. For anything precise, you need the log.

Read your own logs

Log analysis is the part of technical SEO that nobody sells you a scraper for, because there is nothing to scrape. The file is already yours, and it answers questions no crawl can.

Start by throwing away the requests that are lying. User-agent: Googlebot is a string anyone can send, and a meaningful share of your "Googlebot" traffic is not Google. Google's verification procedure is a reverse lookup on the requesting address, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the original address:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For bulk work, skip the DNS round trip and match against the CIDR ranges Google publishes as JSON at developers.google.com/static/crawling/ipranges/common-crawlers.json, with special-crawlers.json, user-triggered-fetchers.json and user-triggered-agents.json alongside it. Google states that "the common crawlers generally crawl from the IP ranges published in the common-crawlers.json object" and that they "always obey robots.txt rules when crawling automatically."

Once the traffic is verified, five questions are worth the parse:

• Which URLs does Googlebot request that your crawl never reaches? Orphans, old parameter combinations, URLs still linked from somewhere you forgot. These consume crawling and return nothing.
• What is the status mix over time? A rising 5xx or 429 share is the input to the crawl-rate feedback loop above, visible in your log days before it shows anywhere else.
• How much crawling lands on non-canonical variants? Sorted, filtered and paginated URLs are where crawl goes to die on large catalogues.
• What is the discovery-to-refresh ratio? Heavy refresh crawling of unchanged pages usually means your lastmod values are noise.
• How much of the bot traffic is neither Google nor Bing? AI crawlers now make up a large and growing share, and this is the number that tells you whether the robots.txt decisions below are theoretical or expensive.

For tooling, GoAccess is free, open source and current: release 1.11 landed on 19 July 2026, and it parses Apache, Nginx, S3, ELB, CloudFront and Caddy formats into a terminal or HTML dashboard. Screaming Frog's Log File Analyser is £99 per year with a free tier capped at 1,000 log events and one project, and it verifies search engine and AI bots automatically, showing you which requests were spoofing. Both are a fraction of the price of the platforms that will infer the same thing from a sample.

The arithmetic that justifies the effort is dull and reliable. One wasted Googlebot request per URL across a 100,000-page catalogue is 100,000 requests spent on nothing, and on a site large enough for crawl budget to bind, that is the difference between new pages being found this week and next month.

What breaks at ten thousand pages

A self-crawl of 200 pages is a script. At scale, four things change.

You can take down your own site. Nothing is throttling you, which is exactly the danger. An unbounded async crawler against your own origin is a load test you did not schedule, and the first symptom is 5xx responses that also depress your crawl rate. Cap concurrency, run against staging when the point is markup rather than production behaviour, and crawl production slowly enough that it looks like traffic.

Sitemaps hit hard limits. Google's sitemap documentation is explicit: "All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs." Past that you need index files. Two more lines from the same page save arguments: Google "ignores <priority> and <changefreq> values," and it uses <lastmod> only "if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." A build pipeline that stamps today's date on every URL has disabled its own signal.

The Search Console API needs paging and patience. At 25,000 rows per request you page with startRow, and a page-by-query pull for a large site is thousands of requests against a 1,200 QPM ceiling. Beyond that, the bulk data export to BigQuery is the intended route, and it carries its own warnings: the first export arrives "up to 48 hours after your successful configuration," schema changes break it, and "data will be accumulated forever for your project, unless you set an expiration time for your data." Set partition expiry, which Google puts at a 14-day minimum, or you are paying to store 2027's copy of 2026.

Your totals will never reconcile, and the reasons are documented. Search Console omits "some queries that are searched a very small number of times" for privacy, so rows do not sum to totals. The interface table caps at 1,000 rows. Data is labelled in Pacific Time and "usually available in 2-3 days." Analytics tools "track traffic only from users who have enabled JavaScript." Anyone who reports a single blended number across these sources without saying which is which is reporting an artefact.

robots.txt has a size limit you can hit. Google parses 500 kibibytes and states that "content which is after the maximum file size is ignored." Generated disallow lists on large catalogues reach that. Worth knowing too: 4xx responses other than 429 are treated as though no robots.txt existed, so a broken deploy that 404s the file opens everything, while a 5xx stops crawling for the first 12 hours and then falls back to a cached copy for up to 30 days.

Make it a deploy check, not an audit

An audit is a thing you do quarterly and then argue about. The higher-value move is to run the same checks on every deploy and let the build fail, which is available to you for your own site and to nobody for theirs.

Three pieces make this practical:

• The script above, pointed at a staging sitemap, exiting non-zero on contradictions. Start with three assertions you will never argue with: no noindex on a sitemap URL, no missing title, no 5xx.
• Lighthouse CI for the performance and accessibility half, which exists to prevent "regressions in accessibility, SEO, offline support, and performance best practices" and supports budgets and assertions per pull request. The current release is 0.15.1 from 26 June 2025. For whole-site coverage rather than a sampled page, Unlighthouse runs Lighthouse across every page in parallel, discovering URLs via robots.txt, sitemap and internal links, MIT-licensed and free, on Node 22 or newer.
• Google's own robots.txt parser, Apache-licensed and described in its README as "slightly modified production code used by Googlebot," so you can assert that a path is allowed using the same matcher the crawler uses instead of your reading of the file. Pair it with validator.schema.org for markup that has no Google rich result attached to it any more.

Publishing is the other half of the loop, and one habit here is out of date. Google's ping endpoint for sitemaps is gone, announced in a June 2023 Search Central post titled "Sitemaps ping endpoint is going away"; accurate lastmod plus a submitted sitemap is what remains. For the other engines, IndexNow takes "up to 10,000 URLs per post" against a key file of 8 to 128 hexadecimal characters, and submissions propagate to Bing, Yandex, Naver, Seznam.cz, Yep and Amazon. Google does not participate. The documentation asks for at least five minutes between updates to the same URL and treats a 429 as your signal to stop, since per-engine thresholds are not published.

Four checks, one build step, and the class of defect that used to survive for a quarter now survives for a commit.

What moved under you since 2024

Three changes since this article was first written alter what your own numbers mean, and none of them were caused by anything you did.

The &num=100 parameter went away in September 2025. Rank trackers had used it to read a hundred results in one request, and its removal made scraped rankings roughly ten times more expensive to collect. The side effect landed in your Search Console: desktop impressions fell sharply for many sites, because a share of those impressions had been generated by tools loading result pages a hundred at a time. Average position often improved at the same moment. If you have a year-over-year chart crossing September 2025, that discontinuity is not your site. The economics of collecting rankings after the change are worked through in rank tracker scraping.

AI answers changed the click, and the measurement lagged. The Pew Research Center's July 2025 analysis followed 900 US adults across 68,879 Google searches in March 2025, of which 12,593 produced an AI summary. Users who saw an AI summary clicked a traditional result on 8% of visits, against 15% for visits without one. Search Console did not report generative-AI visibility separately at all until June 2026, when, as PPC Land reported on 3 June, Google added a dedicated generative-AI view covering AI Overviews, AI Mode and Discover. It reports impressions, pages, countries, devices and dates, the data also sits inside your overall totals, and clicks are not broken out. You can see whether you appear. You cannot yet compute a click-through rate for it.

Your site now has a knob for AI crawlers, and defaults have moved. Cloudflare announced on 1 July 2025 that it would block AI crawlers by default unless they pay, arguing from the collapse in the traffic-for-content trade: by its figures, getting a visitor from OpenAI is "750 times more difficult" than from the Google of old, and from Anthropic "30,000 times more difficult." Whatever you decide, decide it explicitly, and know what the tokens do. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It is a training and grounding opt-out, not a crawler and not a Search control. The full set of controls, and their limits, is in how to protect a website from scraping.

Competitors and SERPs: the same crawl, minus everything that matters

Point the crawler at a competitor and the technique is identical while the yield is thinner. You get their public HTML: heading structure, topic coverage, internal linking, schema, publishing cadence from their sitemap. You do not get their logs, their impressions, their index state, or any of the three data sources above. Treat competitor crawling as reading their intentions, not their results.

The sitemap trick still earns its place: fetch sitemap.xml, take the full URL inventory, crawl it, and you have their content footprint in an afternoon. Cross it against the queries where they outrank you and you have a work list. The pattern generalises to any competitor signal you want on a schedule, which is the subject of competitor monitoring.

Search results are where this stops being cheap. SERP data means geo and device control, rotating residential proxies, CAPTCHA handling, and selectors that break on Google's schedule rather than yours, all for data with no first-party equivalent. Keyword demand has the same shape, and the cheap end of it, autocomplete and related questions, is covered in keyword research scraping. Targets that actively defend themselves are where a managed extraction service earns its keep, and where a SERP API is usually the rational purchase rather than an admission of defeat. If what you need is rankings, competitor inventories and backlink snapshots arriving on a schedule rather than a scraper to maintain, that is data as a service and a different line in the budget.

What the first-party layer costs, read on 13 August 2026

The row that will change first is PageSpeed Insights. Google's own note on the PSI API reads: "We plan to discontinue including real-world data from the Chrome User Experience Report in this API," pointing at the CrUX API and CrUX History API instead. No date is published. If your dashboard pulls field data from PSI, that dependency has an expiry attached, and moving it is an afternoon.

While you are there, the Core Web Vitals thresholds at the 75th percentile are LCP 2.5 seconds, INP 200 milliseconds and CLS 0.1. INP became the stable metric in 2024, replacing First Input Delay, which means any monitoring built before that is watching a metric that no longer exists.

Where auditing your own site stops working

A self-crawl measures what you shipped. Four questions it cannot answer:

Whether the page should exist. The crawl sees a page with a title, an H1 and 900 words and reports it clean. Whether anyone searches for it, and whether you have four other pages competing for the same query, is a question for query data and human judgement.

Why rankings moved. Nothing in an on-page audit establishes causality. Fixing 40 duplicate titles and watching traffic rise the same week proves nothing, especially in a period when the click-through baseline itself is shifting.

What a JavaScript application actually renders. If your content assembles client-side, a plain HTTP crawler is auditing a shell. You need a headless browser in the crawl, which multiplies the cost per URL by an order of magnitude and makes the concurrency question above urgent.

What a personalised or A/B-tested page shows. Your crawler sees one variant, from one address, with no cookie. So does Googlebot, which is a useful symmetry, and neither of you sees what a logged-in customer sees.

There is also the case where crawling your own site is the wrong instrument entirely. If the CMS database is the source of truth, query it. Reconstructing your own product catalogue by parsing HTML you generated from that catalogue an hour earlier is an expensive way to test your templates, and a direct database export answers the same question in seconds. Crawl to see what the site emits. Query to see what it holds.

Staying inside the lines

Your own site is yours, and the only self-imposed limit that matters is load. Everything else has edges.

• Competitor public pages. Scraping publicly available data is broadly permissible in the United States and comparable jurisdictions, with robots.txt, terms and server load still mattering. Identify your crawler, cache what you have, and do not log in. The nuance, including how badly the popular reading of hiQ v. LinkedIn misstates the outcome, is in is web scraping legal.
• Search engines. Automated querying conflicts with their terms. A SERP API does not make the underlying activity approved by the engine; it moves the operational and compliance burden to a vendor built for it.
• Personal data. If what you collect includes personal data, GDPR and CCPA obligations attach regardless of how public the source was. Public is not the same as free to process.
• robots.txt on your own domain. Test it with Google's parser rather than by reading it, because a rule that blocks your own audit crawler is usually a rule that blocks something else too.

Start with the disagreements

The pattern that pays is not a bigger crawl. It is putting two views of the same URL next to each other and looking at the gap: your declared canonical against the Google-selected one, your sitemap against your log, your crawl against Search Console's index state, your title against the one in the result.

Every one of those pairs is free to assemble for a site you own, and none of them is available for a site you do not. That asymmetry is the whole argument for doing the boring first-party work before buying anything: the data with the highest signal per pound is data nobody is selling, because it is already sitting in your access log and your own HTML.

Three assertions in a build step. One log parse a month. One measurement of the thing you are actually optimising rather than the proxy everyone repeats.