10%
Blocks an AI crawler
9,012 / 85,917
9.6%
Blocks GPTBot
8,222 / 85,917
9.4%
Blocks CCBot
8,036 / 85,917
7.7%
Blocks Google-Extended
6,589 / 85,917
AI crawlers blocked in robots.txt
| Crawler | Operator | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|---|
| GPTBot | OpenAI (training) | 9.6% | 47% | 45% | 7.4% | 8.2% | 3.7% | 5.5% |
| ChatGPT-User | OpenAI (user fetch) | 5.2% | 44% | 43% | 3.3% | 4.6% | 2.3% | 2.3% |
| OAI-SearchBot | OpenAI (search) | 4% | 44% | 42% | 2.6% | 3.6% | 2.2% | 1.8% |
| ClaudeBot | Anthropic | 8.6% | 46% | 45% | 6.9% | 7.6% | 3.4% | 5.2% |
| anthropic-ai | Anthropic (legacy) | 7.4% | 46% | 44% | 5.6% | 5.6% | 2.4% | 4.5% |
| Google-Extended | Google (Gemini) | 7.7% | 45% | 44% | 5.9% | 5% | 2.7% | 4.4% |
| CCBot | Common Crawl | 9.4% | 4.9% | 4.7% | 8.6% | 8% | 3.1% | 5.4% |
| PerplexityBot | Perplexity | 5.1% | 44% | 43% | 3.4% | 4.2% | 2.1% | 2.3% |
| Bytespider | ByteDance | 9.1% | 4.6% | 4.4% | 7.1% | 8% | 3.4% | 6.7% |
| Applebot-Extended | Apple | 7.5% | 3.9% | 3.2% | 5.8% | 5.6% | 2.7% | 4.3% |
| Amazonbot | Amazon | 7.9% | 46% | 45% | 6.5% | 7% | 3.1% | 5.1% |
| meta-externalagent | Meta | 7.7% | 46% | 44% | 6.3% | 6.8% | 3.3% | 4.6% |
| cohere-ai | Cohere | 7.1% | 45% | 44% | 5.4% | 4.9% | 2.4% | 5.3% |
| Diffbot | Diffbot | 7% | 45% | 43% | 5.3% | 5% | 2.2% | 5.3% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| Blocks an AI crawler | 10% | 6% | 6.4% | 11% | 11% | 3.9% | 7% |
| Blocks GPTBot | 9.6% | 47% | 45% | 7.4% | 8.2% | 3.7% | 5.5% |
| Blocks ClaudeBot | 8.6% | 46% | 45% | 6.9% | 7.6% | 3.4% | 5.2% |
| Blocks CCBot | 9.4% | 4.9% | 4.7% | 8.6% | 8% | 3.1% | 5.4% |
| Blocks Google-Extended | 7.7% | 45% | 44% | 5.9% | 5% | 2.7% | 4.4% |
| Blocks PerplexityBot | 5.1% | 44% | 43% | 3.4% | 4.2% | 2.1% | 2.3% |
| Content-Signal in robots.txt | 4.1% | 2.2% | 2% | 3.5% | 2.8% | 0.5% | 3.5% |
| llms.txt | 15% | 17% | 14% | 14% | 14% | 7.4% | 18% |
Search bots blocked
- 1 PetalBot 11% 6,271
- 2 Baiduspider 9.2% 5,194
- 3 YandexBot 6.1% 3,450
- 4 Applebot 4.7% 2,655
- 5 SeznamBot 4.6% 2,589
- 6 Sogou 4.2% 2,359
- 7 Naverbot 4.2% 2,353
- 8 Yeti 4.2% 2,334
- 9 DuckDuckBot 3.3% 1,877
- 10 Bingbot 3.2% 1,797
Share of 56,191 sites in the sample.
SEO bots blocked
- 1 MJ12bot 9% 5,075
- 2 AhrefsBot 8.6% 4,808
- 3 SemrushBot 8.2% 4,611
- 4 DotBot 8.1% 4,528
- 5 BLEXBot 7.2% 4,039
- 6 DataForSeoBot 6.7% 3,752
- 7 Barkrowler 5.3% 2,953
- 8 rogerbot 4.7% 2,664
- 9 serpstatbot 4.5% 2,531
- 10 Screaming Frog SEO Spider 4.1% 2,303
Share of 56,191 sites in the sample.
Our crawler: 68,769 sites · Global · updated Oct 23, 2026 · Methodology
Need data from a website?
Send the URL and fields — we reply with a free sample and a quote.