- Home
- Methodology
How we measure
Every number on this site comes from our own crawler. Here is what it does and where its limits are.
Samples
| Market | Rule | Sites | Crawled | Homepage parsed |
|---|---|---|---|---|
| Global | Top 25,000 of the Tranco list | 100,000 | 100,000 | 68,803 |
| Netherlands | Top .nl domains | 17,493 | 17,493 | 8,312 |
| Japan | Top .jp domains | 16,836 | 16,836 | 12,231 |
| Italy | Top .it domains | 15,301 | 15,301 | 7,715 |
| United Kingdom | Top .uk domains | 30,000 | 30,000 | 11,907 |
| Spain | .es domains | 5,560 | 5,560 | 4,445 |
| Latin America | .mx .ar .cl .pe .uy .com.co … and top .br | 26,777 | 26,777 | 19,912 |
What we check on each site
- DNS: A/AAAA, NS, MX, TXT (SPF and SaaS verification), DMARC, CAA, DNSSEC, BIMI, MTA-STS.
- TLS: certificate authority, validity, protocol version — with a direct handshake.
- Homepage: technologies (500 signatures), SEO tags, Schema.org, page weight, third parties.
- Response headers: security headers, compression, HTTP version, cookies.
- robots.txt (incl. AI crawlers), sitemap.xml, llms.txt, security.txt, ads.txt.
- One visit per site and file; robots.txt is respected; bot challenges are recorded, not bypassed.
Sources
TrancoK9YYW
DateNov 5, 2026
IANA1,436 zones
Lists: Tranco (research ranking of popular domains), IANA root zone, Public Suffix List, iptoasn.com.