65%
robots.txt
56,191 / 85,917
68%
Sitemap in robots.txt
38,099 / 56,191
3.7%
robots.txt blocks everything
2,077 / 56,191
68%
sitemap.xml
46,535 / 68,769
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| robots.txt | 65% | 80% | 84% | 70% | 72% | 43% | 61% |
| Sitemap in robots.txt | 68% | 86% | 87% | 69% | 70% | 64% | 73% |
| robots.txt blocks everything | 3.7% | 53% | 49% | 1.9% | 2.3% | 3.2% | 2.7% |
| sitemap.xml | 68% | 77% | 77% | 72% | 72% | 58% | 67% |
| Blocks an AI crawler | 10% | 6% | 6.4% | 11% | 11% | 3.9% | 7% |
Medians and spread
| Metric | P10 | P25 | Median | P75 | P90 | Sites |
|---|---|---|---|---|---|---|
| robots.txt size | 61 B | 125 B | 420 B | 1.5 KB | 3.5 KB | 56,191 |
| User-agent groups | 1 | 1 | 1 | 3 | 15 | 56,191 |
SEO bots blocked
- 1 MJ12bot 9% 5,075
- 2 AhrefsBot 8.6% 4,808
- 3 SemrushBot 8.2% 4,611
- 4 DotBot 8.1% 4,528
- 5 BLEXBot 7.2% 4,039
- 6 DataForSeoBot 6.7% 3,752
- 7 Barkrowler 5.3% 2,953
- 8 rogerbot 4.7% 2,664
- 9 serpstatbot 4.5% 2,531
- 10 Screaming Frog SEO Spider 4.1% 2,303
Share of 56,191 sites in the sample.
Search bots blocked
- 1 PetalBot 11% 6,271
- 2 Baiduspider 9.2% 5,194
- 3 YandexBot 6.1% 3,450
- 4 Applebot 4.7% 2,655
- 5 SeznamBot 4.6% 2,589
- 6 Sogou 4.2% 2,359
- 7 Naverbot 4.2% 2,353
- 8 Yeti 4.2% 2,334
- 9 DuckDuckBot 3.3% 1,877
- 10 Bingbot 3.2% 1,797
Share of 56,191 sites in the sample.
Our crawler: 68,769 sites · Global · updated Oct 23, 2026 · Methodology
Need data from a website?
Send the URL and fields — we reply with a free sample and a quote.