Skip to content
GET /methodology 200 OK
<scraping.pro/>
  1. Home
  2. Methodology

How we measure

Every number on this site comes from our own crawler. Here is what it does and where its limits are.

Samples

MarketRuleSitesCrawledHomepage parsed
GlobalTop 25,000 of the Tranco list100,000100,00068,803
NetherlandsTop .nl domains17,49317,4938,312
JapanTop .jp domains16,83616,83612,231
ItalyTop .it domains15,30115,3017,715
United KingdomTop .uk domains30,00030,00011,907
Spain.es domains5,5605,5604,445
Latin America.mx .ar .cl .pe .uy .com.co … and top .br26,77726,77719,912

What we check on each site

  • DNS: A/AAAA, NS, MX, TXT (SPF and SaaS verification), DMARC, CAA, DNSSEC, BIMI, MTA-STS.
  • TLS: certificate authority, validity, protocol version — with a direct handshake.
  • Homepage: technologies (500 signatures), SEO tags, Schema.org, page weight, third parties.
  • Response headers: security headers, compression, HTTP version, cookies.
  • robots.txt (incl. AI crawlers), sitemap.xml, llms.txt, security.txt, ads.txt.
  • One visit per site and file; robots.txt is respected; bot challenges are recorded, not bypassed.
Sources
TrancoK9YYW
DateNov 5, 2026
IANA1,436 zones

Lists: Tranco (research ranking of popular domains), IANA root zone, Public Suffix List, iptoasn.com.