EZY Research · Study 4 · July 27, 2026
Seven in ten business websites have never told AI crawlers anything at all
69%
No AI instructions
robots.txt silent on AI crawlers
52%
No llms.txt
Among never-optimized sites
12.7%
Block GPTBot
Often via plugin templates
10.3%
No XML sitemap
Mean robots score: 48/100
We audited 387 business websites before any optimization touched them. The question was simple: has this site made any decision - allow, block, anything - about the AI crawlers that now read the web?
69% have made no decision at all. Their robots.txt never mentions GPTBot, ClaudeBot, PerplexityBot or Google-Extended - and for 8.5% of sites that's because there is no robots.txt to mention them in. These sites are crawlable by default, invisible by design to no one, and governed by nothing.
The rest split two ways. 12.7% explicitly block GPTBot - some deliberately, many via a security plugin or a copied template the owner has never read, which means a meaningful slice of businesses are opted out of ChatGPT without knowing it. And roughly one in five have written some AI directive, the early adopters.
EZY Research · 387 sites, audited before any optimization

Most business websites have made no AI decision
Share of never-optimized sites failing each check
EZY Research · pre-intervention robots.txt / llms.txt / sitemap audits · Jul 2026
Share of never-optimized sites failing each check
| Check | Share failing |
|---|---|
| Give AI crawlers no instructions at all | 69% |
| Have no llms.txt | 52% |
| Explicitly block GPTBot | 12.7% |
| Have no XML sitemap | 10.3% |
The supporting files tell the same story. 52% of audited sites have no llms.txt (and our panel skews toward owners already curious about AI visibility - the open web almost certainly sits higher). One in ten has no XML sitemap. The average robots.txt scores 48 out of 100 against AI-readiness checks.
The fix list is short and unglamorous: a robots.txt that names the AI crawlers and states your choice, an XML sitemap submitted where the AI engines actually look (Bing feeds several of them), an llms.txt as forward cover, and schema so machines read your pages the way you meant them. EZY's free tier generates and uploads all four on WordPress and Cloudflare in one click.
Methodology
Data: EZY.ai's 556-domain research panel - 125 connected business websites (WordPress and Cloudflare) with server-side logs, and 387 never-connected audited sites, 27 Apr-19 Jul 2026, UTC. Bots identified by user agent with signature verification where available. Panel skews toward small-business sites whose owners sought an AI-visibility tool. Aggregates only. Full tables available on request.
Aggregate CSVs for every table are available on request. The fixed-cohort check (42 domains present all quarter) is applied to all growth claims; raw-panel growth is reported only alongside it. Dataset license: CC BY 4.0.