AI crawlers — the bots AI companies use to read the web. They are not all doing the same job, and a robots.txt written in a hurry usually blocks the wrong ones.
Three jobs, not one
- Training. GPTBot (OpenAI) and ClaudeBot (Anthropic) collect content that may be used to train future models. Blocking these is a legitimate business decision with no effect on whether you appear in answers.
- Search indexing. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes those products search when a user asks a question. Perplexity states plainly that PerplexityBot “is not used to crawl content for AI foundation models.” Block these and you remove yourself from the answers.
- Live fetching. ChatGPT-User, Claude-User and Perplexity-User visit a page at the moment a user’s question requires it.
OpenAI also runs OAI-AdsBot, which checks the safety of pages submitted as ads.
The Google-Extended trap
This is the one that costs people visibility. Blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google’s own wording: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
Google-Extended governs whether your content trains future Gemini models and grounds answers in the Gemini app and Vertex AI. AI Overviews and AI Mode are features of Google Search, and Search crawling is governed by Googlebot. The only ways to limit what Search shows from your pages are the ordinary snippet controls — nosnippet, data-nosnippet, max-snippet, noindex — and all of them cost you ordinary search too.
Bing has no separate AI user agent. Control there runs through NOCACHE and NOARCHIVE meta tags on bingbot.
Sources: Google common crawlers · Google, AI features and your website · OpenAI bots · Anthropic crawler documentation · Perplexity crawlers.
Related: llms.txt, AI Overviews, retrieval-augmented generation.