- Home
- Data
68,769 sitesGlobal8 topics
How websites worldwide are built and secured
How websites worldwide handle HTTPS, email security, robots.txt, AI crawlers, SEO and more in 2026 — measured on 68,769 sites.
Bot protection
WAFs, bot managers and challenge pages on popular websites.
18%Bot protection / WAF8.6%CAPTCHA
AI crawlers
Which AI bots websites block in robots.txt — GPTBot, ClaudeBot, CCBot, Google-Extended and others.
10%Blocks an AI crawler9.6%Blocks GPTBot
robots.txt
How websites write robots.txt: sizes, sitemaps, blocked bots.
65%robots.txt68%Sitemap in robots.txt
Sitemaps & feeds
sitemap.xml, feeds and other machine-readable entry points.
68%sitemap.xml68%Sitemap in robots.txt
llms.txt
Adoption of llms.txt next to robots.txt and AI-crawler rules.
15%llms.txt10%Blocks an AI crawler
Frameworks & libraries
JavaScript frameworks, CSS frameworks and libraries.
24%JS framework37%CSS framework
E-commerce
Shop platforms, payments and reviews on popular websites.
5.5%E-commerce platform2%Payment widget
Hosting & cloud
Where popular websites are hosted, by network (AS) and platform.
36%IPv6 (AAAA record)58%CDN
Our crawler: 68,769 sites · Global · updated Oct 23, 2026 · Methodology
Markets
Markets compared
The same checks on popular sites in seven markets.
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| Bot protection / WAF | 18% | 19% | 12% | 13% | 13% | 4.9% | 15% |
| CAPTCHA | 8.6% | 20% | 16% | 18% | 19% | 7.7% | 17% |
| Served a bot challenge | 13% | 10% | 5.1% | 12% | 11% | 9.3% | 13% |
| CDN | 58% | 64% | 47% | 50% | 49% | 47% | 59% |
| robots.txt blocks everything | 3.7% | 53% | 49% | 1.9% | 2.3% | 3.2% | 2.7% |
| Blocks an AI crawler | 10% | 6% | 6.4% | 11% | 11% | 3.9% | 7% |
| sitemap.xml | 68% | 77% | 77% | 72% | 72% | 58% | 67% |
| llms.txt | 15% | 17% | 14% | 14% | 14% | 7.4% | 18% |