OpenAI (ChatGPT/GPT) Crawlers and robots.txt

OpenAI's AI tools use standard web crawlers that honor robots.txt. By contrast, leaving no rule (or explicitly using Allow) permits those crawlers. In short, these AI tools follow normal robots.txt rules: if you want them to access your content, ensure your robots.txt does not disallow their user-agent; if you want to block them, list them with Disallow directives.

ChatGPT-User (ChatGPT browsing)

Used by ChatGPT (Plus/Enterprise) to fetch content on demand. It respects robots.txt; by default it can crawl any page not disallowed. To allow access to all content, you could add User-agent: ChatGPT-User Disallow: (no path). To block it, use Disallow: /.

GPTBot (OpenAI's web crawler)

Used to gather training data. OpenAI states GPTBot obeys robots.txt and can be managed with User-agent: GPTBot rules. If you allow GPTBot (by omitting a block), OpenAI says this may improve AI models. To opt out of training data collection, add User-agent: GPTBot Disallow: /.

OAI-SearchBot (ChatGPT search crawler)

Used to index sites for ChatGPT's SearchGPT feature. OpenAI explicitly recommends allowing this bot. Their guidance is: "ChatGPT uses a web crawler called OAI-SearchBot… For your site to be discoverable in ChatGPT, make sure you aren't blocking OAI-SearchBot". In practice, if you have a blanket Disallow: / for User-agent: *, you must add an exception for OAI-SearchBot. Otherwise, any content not blocked for Googlebot is generally visible.

These OpenAI crawlers identify themselves with clear user-agent strings (e.g. ChatGPT-User/1.0), so you can target them in robots.txt. Official guidance (OpenAI documentation) emphasizes using robots.txt to manage access: for example, to grant full access, their example robots.txt shows "User-agent: ChatGPT-User Disallow:" with no path. If you see AI traffic from these agents, you can always verify by IP (OpenAI publishes their ranges) or robots.txt logs.

Blocking OpenAI Crawlers

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Understanding how these crawlers work and properly configuring your robots.txt is crucial for controlling how OpenAI's AI tools interact with your website content.

Google AI Overviews and robots.txt

Google's new AI Overviews (Search Generative Experience) do not use a special bot – they rely on Google's existing crawler. In other words, Googlebot (the same crawler used for standard Search) fetches pages for AI Overviews. Google's documentation explicitly says there are no new crawling requirements for AI Overviews: if Googlebot can crawl and index a page, it can be used in an AI Overview.

Googlebot (AI Overviews)

No separate user-agent token is used. To allow your content in AI Overviews, simply allow Googlebot as usual in robots.txt. To prevent your site (or certain pages) from appearing in Overviews, you would have to block or noindex those pages for Googlebot. However, Google warns that blocking Googlebot entirely (e.g. User-agent: Googlebot Disallow: /) will remove your site from search results as well. In practice, Google's official guidance is to use standard SEO controls: ensure pages are indexed and accessible if you want them in AI features, or use noindex/data-nosnippet/noarchive to prevent content from being used in answers.

Google-Extended (Gemini/Bard training)

This is a different token used by Google to opt sites out of AI training, but it is not used by AI Overviews. Blocking Google-Extended in robots.txt only affects Google's model training, not the Search Overviews. (Google and SEOs note that disallowing Google-Extended will not stop AI Overviews.)

Key Takeaways & Best Practices

Default Access

Both OpenAI and Google crawlers default to allowing access unless specifically disallowed in robots.txt. There is no "opt-in" file needed; the absence of a disallow means the bot can crawl.

OpenAI Bots

To make your site available to ChatGPT browsing or SearchGPT, ensure you do not disallow ChatGPT-User or OAI-SearchBot in robots.txt. If you want to block ChatGPT, explicitly add User-agent: ChatGPT-User Disallow: /. For GPTBot (training), block with User-agent: GPTBot Disallow: / if desired. OpenAI's documentation and experts note that these bots will obey such directives.

Google AI Overviews

No special configuration is needed beyond standard SEO. If Googlebot can crawl and index your pages, they may be used in AI Overviews. To exclude content, the only reliable method is to prevent Googlebot from crawling or indexing it (e.g. noindex or robots.txt for Googlebot). Google's developer docs confirm that robots.txt for Googlebot is the control for AI features.

Sources: OpenAI's documentation and public statements (via OpenAI's site and press coverage) detail its crawling bots and recommend using robots.txt to manage them. Google's own developer documentation confirms that AI Overviews use Googlebot and require no special crawler permissions. All cited sources above reflect the current (2025) behavior of these systems.