- Home
- Data
68,167 sitesGlobal8 topics
How websites worldwide are built and secured
How websites worldwide handle HTTPS, email security, robots.txt, AI crawlers, SEO and more in 2026 — measured on 68,167 sites.
Bot protectionWAFs, bot managers and challenge pages on popular websites.18%Bot protection / WAF8%CAPTCHAAI crawlersWhich AI bots websites block in robots.txt — GPTBot, ClaudeBot, CCBot, Google-Extended and others.12%Blocks an AI crawler11%Blocks GPTBotrobots.txtHow websites write robots.txt: sizes, sitemaps, blocked bots.63%robots.txt66%Sitemap in robots.txtSitemaps & feedssitemap.xml, feeds and other machine-readable entry points.67%sitemap.xml66%Sitemap in robots.txtllms.txtAdoption of llms.txt next to robots.txt and AI-crawler rules.15%llms.txt12%Blocks an AI crawlerFrameworks & librariesJavaScript frameworks, CSS frameworks and libraries.25%JS framework37%CSS frameworkE-commerceShop platforms, payments and reviews on popular websites.4.8%E-commerce platform1.6%Payment widgetHosting & cloudWhere popular websites are hosted, by network (AS) and platform.36%IPv6 (AAAA record)59%CDN
Our crawler: 68,167 sites · Global · updated Nov 13, 2026 · Methodology
Markets
Markets compared
The same checks on popular sites in seven markets.
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| Bot protection / WAF | 18% | 20% | 12% | 13% | 13% | 5.2% | 16% |
| CAPTCHA | 8% | 16% | 13% | 16% | 18% | 6.6% | 15% |
| Served a bot challenge | 14% | 12% | 5.9% | 14% | 12% | 11% | 15% |
| CDN | 59% | 66% | 51% | 49% | 50% | 47% | 59% |
| robots.txt blocks everything | 3.6% | 61% | 53% | 3.8% | 5.1% | 3% | 2.7% |
| Blocks an AI crawler | 12% | 6.5% | 6.7% | 11% | 11% | 4.5% | 8.9% |
| sitemap.xml | 67% | 76% | 76% | 69% | 67% | 58% | 65% |
| llms.txt | 15% | 13% | 11% | 12% | 13% | 5.9% | 18% |