Skip to content
GET /insights/robots-txt200 OK
<scraping.pro/>
  1. Home
  2. Data
  3. robots.txt
68,167 sitesGlobal

robots.txt statistics worldwide

How websites write robots.txt: sizes, sitemaps, blocked bots.

63%
robots.txt
53,980 / 86,147
66%
Sitemap in robots.txt
35,857 / 53,980
3.6%
robots.txt blocks everything
1,948 / 53,980
67%
sitemap.xml
45,554 / 68,167

By market

MetricGlobalUnited KingdomNetherlandsItalySpainJapanLatin America
robots.txt63%78%78%63%59%43%58%
Sitemap in robots.txt66%88%87%68%70%64%71%
robots.txt blocks everything3.6%61%53%3.8%5.1%3%2.7%
sitemap.xml67%76%76%69%67%58%65%
Blocks an AI crawler12%6.5%6.7%11%11%4.5%8.9%

Medians and spread

MetricP10P25MedianP75P90Sites
robots.txt size63 B132 B465 B1.8 KB3.5 KB53,980
User-agent groups11143153,980

SEO bots blocked

  1. 1MJ12bot8.7%4,699
  2. 2AhrefsBot8.4%4,508
  3. 3SemrushBot8%4,322
  4. 4DotBot7.9%4,248
  5. 5BLEXBot7%3,776
  6. 6DataForSeoBot6.5%3,521
  7. 7Barkrowler5.1%2,771
  8. 8rogerbot4.6%2,501
  9. 9serpstatbot4.3%2,348
  10. 10Screaming Frog SEO Spider4%2,150

Share of 53,980 sites in the sample.

Search bots blocked

  1. 1PetalBot15%8,115
  2. 2Baiduspider13%7,160
  3. 3YandexBot5.9%3,200
  4. 4Applebot4.6%2,502
  5. 5SeznamBot4.5%2,410
  6. 6Sogou4.1%2,212
  7. 7Naverbot4.1%2,190
  8. 8Yeti4.1%2,187
  9. 9DuckDuckBot3.3%1,758
  10. 10Bingbot3.1%1,671

Share of 53,980 sites in the sample.

Our crawler: 68,167 sites · Global · updated Nov 13, 2026 · Methodology

Need data from a website?

Send the URL and fields — we reply with a free sample and a quote.