Skip to content
GET /insights200 OK
<scraping.pro/>
  1. Home
  2. Data
68,167 sitesGlobal8 topics

How websites worldwide are built and secured

How websites worldwide handle HTTPS, email security, robots.txt, AI crawlers, SEO and more in 2026 — measured on 68,167 sites.

Markets

Markets compared

The same checks on popular sites in seven markets.

MetricGlobalUnited KingdomNetherlandsItalySpainJapanLatin America
Bot protection / WAF18%20%12%13%13%5.2%16%
CAPTCHA8%16%13%16%18%6.6%15%
Served a bot challenge14%12%5.9%14%12%11%15%
CDN59%66%51%49%50%47%59%
robots.txt blocks everything3.6%61%53%3.8%5.1%3%2.7%
Blocks an AI crawler12%6.5%6.7%11%11%4.5%8.9%
sitemap.xml67%76%76%69%67%58%65%
llms.txt15%13%11%12%13%5.9%18%