How to Set Up robots.txt for ChatGPT, Claude, Perplexity and Gemini Crawlers
You set up robots.txt for AI crawlers by publishing a plain-text file at your domain root that names each crawler's user-agent and states which paths it may fetch. EZY.ai generates and publishes that file on the free Starter plan, covering ChatGPT, Claude, Perplexity and Gemini, through WordPress, Cloudflare or a script tag.
What robots.txt does for AI crawlers
A robots.txt file is the first thing a well-behaved crawler requests from your site. It lists user-agents, the names crawlers use to identify themselves, and gives each one Allow or Disallow rules for URL paths. AI assistants such as ChatGPT, Claude, Perplexity and Gemini each send their own crawlers, and each reads the file on its own terms.
The file controls access. Whether an assistant mentions your business also depends on content quality, structured data and entity clarity. But a crawler that can't fetch your pages has nothing to read, so robots.txt is the first item in an AI visibility audit. Our AI Visibility Audit and Score checks it alongside presence, citations, entity coverage and schema coverage.
How do you write robots.txt rules for AI crawlers?
Write one User-agent block per crawler, then list Allow and Disallow lines beneath it. Use the exact user-agent name each AI vendor publishes, keep rules at the path level, and add a Sitemap line pointing to your XML sitemap. Test the file by fetching it in a browser at yourdomain.com/robots.txt.
Here is a minimal pattern that welcomes AI search crawlers and keeps a private area closed:
User-agent: GPTBot
Allow: /
Disallow: /account/
User-agent: ClaudeBot
Allow: /
Disallow: /account/
User-agent: PerplexityBot
Allow: /
Disallow: /account/
User-agent: Google-Extended
Allow: /
Disallow: /account/
Sitemap: https://yourdomain.com/sitemap.xml
Replace /account/ with whatever paths on your site should stay private, such as checkout, login or staging folders. Each vendor documents its own user-agents and sometimes runs several, with separate ones for training, search indexing and live user requests. The table below shows the main names. Confirm them against each vendor's published documentation before you publish, because vendors add and rename crawlers.
| AI engine | Vendor | Example user-agent |
|---|---|---|
| ChatGPT | OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Claude | Anthropic | ClaudeBot |
| Perplexity | Perplexity | PerplexityBot |
| Gemini | Google-Extended |
Decide what each crawler should see
Start from your goal. A business that wants to be named when buyers ask ChatGPT for a recommendation needs its service pages, pricing, location pages and FAQs open to crawlers. Admin areas, cart pages and duplicate parameter URLs can stay closed.
Some publishers allow search-oriented crawlers and restrict training crawlers. That is a legitimate choice, and robots.txt lets you express it at the user-agent level. Decide per crawler rather than applying one rule to all of them, and write the decision down so the next person who edits the file understands it.
Publish the file with EZY.ai Starter
The EZY.ai Starter plan is free. It includes generated and live robots.txt, llms.txt and XML sitemap files, plus WordPress, Cloudflare and script-tag integrations. You choose the connection that matches your site:
- WordPress: the integration publishes the generated file to your site.
- Cloudflare: the integration serves the file at the edge.
- Script tag: a single tag connects sites that run on other platforms.
We also audit the robots.txt you already have. The audit reads the live file, flags rules that shut out ChatGPT, Claude, Perplexity or Gemini crawlers, and gives you a corrected version to publish. The Growth plan at $49 per month and the Scale plan at $99 per month add the full AEO file set on top of Starter: schema, meta titles and descriptions, facts files and llms-full.txt.
Your sitemap and your llms.txt file work alongside robots.txt. The robots.txt file opens the door, the sitemap lists the pages, and llms.txt gives AI systems a readable summary of what matters on your site.
How do you check that AI crawlers are reading your robots.txt?
Check your server traffic for AI crawler visits after you publish. EZY.ai AI Crawler Analytics tracks visits from ChatGPT, Claude, Perplexity and Gemini crawlers, separates bot traffic from human traffic, and reports at page level every day. It is included on the free Starter plan.
Page-level data shows which URLs the crawlers actually fetch. If a service page you opened in robots.txt never appears in the crawler report, something else is in the way: a firewall rule, a Cloudflare bot setting, a login wall or a noindex tag. A rule in robots.txt only works when the server also lets the request through.
Reading the crawl data also tells you which pages the assistants favor, and that connects to how AI systems choose sources to cite. Once crawl access is confirmed, Prompt and Citation Tracking shows whether the access translates into mentions. Growth tracks 25 prompts and 15 competitors. Scale tracks 50 prompts and 25 competitors.
Mistakes that block AI crawlers by accident
Most robots.txt problems come from a few patterns:
- A blanket Disallow. A line reading
Disallow: /underUser-agent: *closes the whole site to every crawler that has no block of its own. - Staging rules left in production. A file copied from a development site often carries the blanket Disallow along with it.
- Edge-level bot blocking. A security setting at the CDN or firewall can refuse AI crawlers before they ever read robots.txt.
- A missing sitemap line. Without the Sitemap directive, crawlers have to discover pages by following links.
- Typos in user-agent names. A misspelled name matches no crawler, so the rule does nothing.
A typical scenario: a plumbing company on WordPress installed a security plugin that added a blanket Disallow during a redesign. The site looked fine to human visitors for months. A robots.txt audit would have caught it on the first scan, and the fix is a one-line change.
Frequently asked questions
Does robots.txt guarantee ChatGPT will mention my business?
No file guarantees a mention. Robots.txt makes your pages fetchable, which is a precondition. Whether ChatGPT names your business depends on whether your content answers buyer questions clearly, whether your structured data describes the business accurately, and whether other sources corroborate your claims. Tracking prompts over time shows whether the access is turning into citations.
Can I allow AI search crawlers but block training crawlers?
Yes. Robots.txt rules apply per user-agent, so you can open a search-oriented crawler and close a training crawler from the same vendor, where the vendor publishes separate names. Check each vendor's current documentation for the exact user-agents, then write one block per crawler with the access you want.
How long does it take for a robots.txt change to take effect?
Crawlers fetch robots.txt on their own schedules, so timing varies by vendor. Once your edit is live at the domain root, you can confirm the new file loads in a browser straight away. AI Crawler Analytics then shows, daily and page by page, when each crawler starts visiting the pages you opened.
If you want a second pair of eyes on your current file, or you're deciding between Starter, Growth and Scale, send us your domain and your question through the form below, and we'll tell you what your robots.txt is allowing and what it's closing off for ChatGPT, Claude, Perplexity and Gemini.
Get your AI Visibility Score and see what AI crawlers can read
We reply fast.
What would you like help with?
How soon?
Cancel anytime.