EZY Research · Study 4 · July 27, 2026

Seven in ten business websites have never told AI crawlers anything at all

69%

No AI instructions

robots.txt silent on AI crawlers

52%

No llms.txt

Among never-optimized sites

12.7%

Block GPTBot

Often via plugin templates

10.3%

No XML sitemap

Mean robots score: 48/100

We audited 387 business websites before any optimization touched them. The question was simple: has this site made any decision - allow, block, anything - about the AI crawlers that now read the web?

69% have made no decision at all. Their robots.txt never mentions GPTBot, ClaudeBot, PerplexityBot or Google-Extended - and for 8.5% of sites that's because there is no robots.txt to mention them in. These sites are crawlable by default, invisible by design to no one, and governed by nothing.

The rest split two ways. 12.7% explicitly block GPTBot - some deliberately, many via a security plugin or a copied template the owner has never read, which means a meaningful slice of businesses are opted out of ChatGPT without knowing it. And roughly one in five have written some AI directive, the early adopters.

EZY Research · 387 sites, audited before any optimization

EZY.AI

Most business websites have made no AI decision

Share of never-optimized sites failing each check

Give AI crawlers no instructions at all
69%
Have no llms.txt
52%
Explicitly block GPTBot
12.7%
Have no XML sitemap
10.3%

EZY Research · pre-intervention robots.txt / llms.txt / sitemap audits · Jul 2026

Most business websites have made no AI decision: 69% give AI crawlers no instructions, 52% have no llms.txt, 12.7% explicitly block GPTBot, 10.3% have no XML sitemap.

Share of never-optimized sites failing each check

CheckShare failing
Give AI crawlers no instructions at all69%
Have no llms.txt52%
Explicitly block GPTBot12.7%
Have no XML sitemap10.3%

The supporting files tell the same story. 52% of audited sites have no llms.txt (and our panel skews toward owners already curious about AI visibility - the open web almost certainly sits higher). One in ten has no XML sitemap. The average robots.txt scores 48 out of 100 against AI-readiness checks.

The fix list is short and unglamorous: a robots.txt that names the AI crawlers and states your choice, an XML sitemap submitted where the AI engines actually look (Bing feeds several of them), an llms.txt as forward cover, and schema so machines read your pages the way you meant them. EZY's free tier generates and uploads all four on WordPress and Cloudflare in one click.

Methodology

Data: EZY.ai's 556-domain research panel - 125 connected business websites (WordPress and Cloudflare) with server-side logs, and 387 never-connected audited sites, 27 Apr-19 Jul 2026, UTC. Bots identified by user agent with signature verification where available. Panel skews toward small-business sites whose owners sought an AI-visibility tool. Aggregates only. Full tables available on request.

Aggregate CSVs for every table are available on request. The fixed-cohort check (42 domains present all quarter) is applied to all growth claims; raw-panel growth is reported only alongside it. Dataset license: CC BY 4.0.

More from EZY Research