Title tags, meta descriptions, and the H1–H6 headings are how a search engine understands what a page is about — and how a user decides whether to click your snippet. On a site with hundreds or thousands of pages, checking them by hand is impossible: somewhere the title is empty, somewhere it's duplicated across dozens of pages, somewhere the H1 is missing or the heading hierarchy is broken. An SEO scraper pulls every page's metadata into one table, and the problems jump out immediately.
This guide is part of our series on scraping for SEO. Below we cover what to collect, why it matters, and which tools to use.
What a metadata scraper collects
Crawling the site URL by URL, the scraper extracts for each page:
- Title — the page title shown in the browser tab and the search snippet.
- Meta description — the snippet text.
- H1 — the page's main heading.
- H2–H6 — subheadings and their hierarchy.
- Tag length, so you can find values that are too short or get truncated in the SERP.
- Canonical, robots meta, and Open Graph tags — increasingly worth grabbing in the same pass.
This is essentially the same crawl as scraping a site's structure: one pass of the crawler gives you both the tree of sections and the metadata table. That's why titles, tags, and headings are usually collected by the same tools that map structure.
Which problems you're hunting for
Duplicate titles and descriptions. Identical meta tags on different pages are a common reason pages compete with each other and rank worse. A scraper surfaces every repeat instantly.
Empty and missing tags. Pages with no title or H1 lose relevance. In the table, they show up as empty cells.
Over-optimization and length. Titles that are too long get truncated in the SERP; headings stuffed with keywords can trip filters. An export with lengths helps you clean things up.
Broken heading hierarchy. Multiple H1s on a page, skipped levels, an H2 before the H1 — structural markup errors that hurt both crawlers and accessibility.
The output of this audit is a spec for rewriting meta tags: which pages to fix, where to remove duplicates, and where to fill in what's missing.
Tools to scrape tags and headings
Screaming Frog SEO Spider and Sitebulb are the go-to desktop crawlers. On a normal crawl they already collect titles, descriptions, and headings, and flag duplicate, empty, and over-length values. Through custom extraction (by XPath, CSS selector, or regular expressions) you can also pull non-standard elements — for example, every <strong> tag or the contents of a specific block. Netpeak Spider covers the same ground and is a lighter, cheaper option for smaller sites.
For competitor research, the interesting move is collecting tags and headings not from your own site but from the SERP or from rivals' pages. Many competitors optimize their meta tags for search, so their titles, descriptions, and H1–H6 are a ready source of keyword phrases and content-plan ideas. This overlaps with keyword research: competitor headings are a good source of missed queries. SERP-based title scrapers and a spreadsheet add-in like SEO Tools for Excel make this quick.
Roll your own with a few lines of Python
You don't always need a full crawler. For a quick bulk check across a known list of URLs, a small script is enough — this is the core of any title tag scraper or meta description scraper:
import csv
import requests
from bs4 import BeautifulSoup
def scrape_meta(url):
r = requests.get(url, timeout=10, headers={"User-Agent": "seo-audit/1.0"})
soup = BeautifulSoup(r.text, "lxml")
title = soup.title.get_text(strip=True) if soup.title else ""
desc_tag = soup.find("meta", attrs={"name": "description"})
description = desc_tag["content"].strip() if desc_tag and desc_tag.get("content") else ""
h1s = [h.get_text(strip=True) for h in soup.find_all("h1")]
return {
"url": url,
"title": title,
"title_len": len(title),
"description": description,
"desc_len": len(description),
"h1_count": len(h1s),
"h1": h1s[0] if h1s else "",
}
urls = ["https://example.com/", "https://example.com/pricing"]
with open("meta_audit.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=[
"url", "title", "title_len", "description", "desc_len", "h1_count", "h1"
])
writer.writeheader()
for u in urls:
writer.writerow(scrape_meta(u))
Load meta_audit.csv into a spreadsheet, sort by title, and duplicates line up next to each other; filter h1_count > 1 or title_len > 60 to catch structural and length issues. For a full-site crawl rather than a fixed URL list, feed the script a sitemap or add a link-following queue — the pattern is the same as any website scraper.
How this fits the rest of the work
Scraping tags is content-side diagnostics, paired with the technical diagnostics you get from crawling site structure. To rewrite meta tags correctly you need keywords — and those come from keyword research, search autocomplete suggestions, and volume data from Google Keyword Planner. After the edits, you track the effect through rank checking.
If the goal is to analyze snippets and titles straight from the top of the results rather than specific sites' tags, that's SERP scraping — pulling titles, descriptions, and rankings from Google (and Bing) at scale.
FAQ
What is an SEO scraper? A tool that crawls a website and extracts on-page SEO elements — title tags, meta descriptions, H1–H6 headings, canonicals, and more — into a structured table so you can audit them in bulk instead of page by page.
How do I scrape meta tags from an entire site?
Point a crawler like Screaming Frog or Sitebulb at your domain, or run a script that follows your sitemap and extracts <title>, <meta name="description">, and heading tags for every URL, then export to CSV.
Can I scrape competitors' title tags and descriptions? Yes — from their pages or from the SERP. It's a standard way to find keyword phrases and content ideas. Only collect publicly visible metadata and stay within each site's terms and applicable law.
Auditing one site is a script; monitoring thousands of pages (or competitors) on a schedule is a pipeline. scraping.pro runs SERP scraping and custom data extraction as a managed service, delivering the metadata and ranking feeds ready for your dashboards.