My robots file has one rule beyond “allow everything,” and it disallows GPTBot. I mentioned in the AI Overviews post that this was a decision I made by accident. This is the follow-up, because I went and read what each of Google's crawler tokens actually controls, and the answer is cleaner than I expected.
The tokens, from Google's document
Googlebot affects Google Search, including Discover and all Google Search features, plus Images, Video, and News. AI Overviews and AI Mode are Search features. They are fed by Googlebot. There is no separate crawler for them.
Googlebot-Image affects Images, Discover, Video, and any Search feature that shows an image, logo, or favicon.
GoogleOther affects no specific product. It is used for research and development crawls.
Google-Extended is the one people misread. Google's description is that publishers can use it to manage whether crawled content “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and on Vertex AI. Then the sentence that settles it:
Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
What that means in practice
There are three separate things you might want to control, and they map to three separate switches.
| You want to | Use |
|---|---|
| Stay out of Gemini training and grounding, keep Search | Disallow for Google-Extended |
| Stay out of AI Overviews and AI Mode | nosnippet, max-snippet, or noindex on the page. There is no crawler for this. |
| Stay out of OpenAI's training | Disallow for GPTBot |
The middle row is the one that trips people. Blocking Google-Extended does nothing to AI Overviews. The only lever for AI Overviews is the snippet controls, and those also shrink or remove your regular snippet. Google has tied the two together on purpose.
My file, read against that
Right now this site says: Googlebot, everything. Google-Extended, not mentioned, so allowed. GPTBot, blocked.
Which means my actual policy is: Google may index me, summarize me in AI Overviews, and train Gemini on me. OpenAI may do none of those. That is an inconsistent position and I did not choose it. I chose “block GPTBot” in a commit that was mostly about adding an analytics tag.
The consistent versions are:
- Open. Remove the GPTBot rule. Everyone may crawl and train. I get whatever visibility comes from being in every model's index.
- Search yes, training no. Keep GPTBot blocked, add a
Disallowfor Google-Extended. Search and AI Overviews unaffected, per Google's own sentence. No model trains on the posts. - Search only, no AI summaries. Option 2 plus
max-snippetlimits. This costs regular snippets too.
The second is what I thought the first rule meant when I wrote it, and it is the one I am leaning toward. The change would be one more block in the robots file, and it would not touch a single thing about how the site ranks. Google said so in writing.
What I would not do
Block Googlebot from anything on this site to keep it out of AI features. The document makes clear that is not a lever that exists. Blocking Googlebot is leaving Search.
And I would not trust a tool that reports “AI crawler blocked” as a single checkbox. There are at least three distinct things behind that checkbox, and they do not do the same job.
Source: Google Search Central, “Google's common crawlers,” and “AI features and your website.”