4 min read
There is a file at the root of your website that most business owners have never opened, and a handful of lines in it decide whether ChatGPT, Claude and Perplexity are allowed to see you at all. Most of the ones we read are wrong — not maliciously, just written by someone who treated “AI bots” as one thing.
They are not one thing. They do three different jobs, and the difference between them is the difference between protecting your content and deleting yourself from the answers.
Three jobs, three sets of bots
Every major AI company runs more than one crawler, and they are deliberately separated so that site owners can make different decisions about each.
- Training.
GPTBotfrom OpenAI,ClaudeBotfrom Anthropic. These collect content that may be used to train future models. Blocking them has no effect on whether you show up in an answer today. - Search indexing.
OAI-SearchBot,Claude-SearchBot,PerplexityBot. These build the indexes those products actually search when a person asks a question. Perplexity states outright that PerplexityBot “is not used to crawl content for AI foundation models” — its only job is surfacing sites in results. - Live fetching.
ChatGPT-User,Claude-User,Perplexity-User. These visit a page at the moment a specific user’s question requires it.
Blocking the training crawlers is a defensible business decision, and plenty of publishers have made it. Blocking the indexing crawlers is not a decision most businesses would make on purpose — it is the machine-readable equivalent of asking to be left out of the recommendation.
The Google-Extended mistake
This is the one that costs the most visibility, because the intuition is exactly backwards.
People block Google-Extended believing it keeps them out of AI Overviews. It does not. Google’s own documentation says so in a single sentence: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
What Google-Extended actually governs is whether your content trains future Gemini models and grounds answers in the Gemini app and Vertex AI. AI Overviews and AI Mode are features of Google Search, and Search crawling is governed by Googlebot. There is no separate opt-out for AI Overviews. The only levers are the ordinary snippet controls — nosnippet, data-nosnippet, max-snippet, noindex — and every one of them costs you ordinary search results too.
So the honest position is uncomfortable and worth stating plainly: if you want to be in Google Search, you are in AI Overviews. There is currently no way to take one without the other.
Bing works differently again. There is no separate Bing AI user agent — control runs through NOCACHE and NOARCHIVE meta tags read by bingbot.
What blocking actually costs
A business that blocks the indexing crawlers does not get a warning. Nothing breaks. Traffic does not fall off a cliff, because the traffic was never coming from there in the first place. What happens is that a customer asks an assistant for the best provider in your category and city, and your name is simply not among the candidates it can consider.
That failure is invisible from inside your analytics, which is precisely why it persists. You cannot see a recommendation you were never in.
A robots.txt that says what most businesses actually mean
If your position is “do not train on my content, but do recommend me” — which is where most local service businesses land once the distinction is explained — that looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /If you have no objection to training either, the simplest correct answer is to have no AI-specific rules at all. An empty robots.txt is not a mistake. A confidently wrong one is.
What to check this week
- Open
yourdomain.com/robots.txtand read it. Most people have never seen theirs. - Look for any
Disallow: /under an agent whose name containsSearchBot,PerplexityBot, orGooglebot. Those are the ones that cost you answers. - Check whether a security plugin, CDN or host-level bot filter is blocking AI crawlers before they ever reach robots.txt. This is common, and robots.txt will look innocent while the block happens upstream.
- If you blocked
Google-Extendedhoping to stay out of AI Overviews, decide again now that you know what it does.
Sources: Google common crawlers · Google, AI features and your website · OpenAI bots · Anthropic crawler documentation · Perplexity crawlers. All checked 29 August 2026.
Definitions: AI crawlers, llms.txt, AI Overviews. Or have us check your setup as part of a free AI Visibility Snapshot.