63%
robots.txt
53,980 / 86,147
66%
Sitemap in robots.txt
35,857 / 53,980
3.6%
robots.txt blocks everything
1,948 / 53,980
67%
sitemap.xml
45,554 / 68,167
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| robots.txt | 63% | 78% | 78% | 63% | 59% | 43% | 58% |
| Sitemap in robots.txt | 66% | 88% | 87% | 68% | 70% | 64% | 71% |
| robots.txt blocks everything | 3.6% | 61% | 53% | 3.8% | 5.1% | 3% | 2.7% |
| sitemap.xml | 67% | 76% | 76% | 69% | 67% | 58% | 65% |
| Blocks an AI crawler | 12% | 6.5% | 6.7% | 11% | 11% | 4.5% | 8.9% |
Medians and spread
| Metric | P10 | P25 | Median | P75 | P90 | Sites |
|---|---|---|---|---|---|---|
| robots.txt size | 63 B | 132 B | 465 B | 1.8 KB | 3.5 KB | 53,980 |
| User-agent groups | 1 | 1 | 1 | 4 | 31 | 53,980 |
SEO bots blocked
- 1MJ12bot8.7%4,699
- 2AhrefsBot8.4%4,508
- 3SemrushBot8%4,322
- 4DotBot7.9%4,248
- 5BLEXBot7%3,776
- 6DataForSeoBot6.5%3,521
- 7Barkrowler5.1%2,771
- 8rogerbot4.6%2,501
- 9serpstatbot4.3%2,348
- 10Screaming Frog SEO Spider4%2,150
Share of 53,980 sites in the sample.
Search bots blocked
- 1PetalBot15%8,115
- 2Baiduspider13%7,160
- 3YandexBot5.9%3,200
- 4Applebot4.6%2,502
- 5SeznamBot4.5%2,410
- 6Sogou4.1%2,212
- 7Naverbot4.1%2,190
- 8Yeti4.1%2,187
- 9DuckDuckBot3.3%1,758
- 10Bingbot3.1%1,671
Share of 53,980 sites in the sample.
Our crawler: 68,167 sites · Global · updated Nov 13, 2026 · Methodology
Need data from a website?
Send the URL and fields — we reply with a free sample and a quote.