Google-Extended
Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.
At a glance
- Operator
- Category
-
Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
- User-Agent token
-
None — this is a robots.txt control token, not a crawler.
Google fetches with its own crawler and reads this token only to decide what it may then do with the result. No request ever carries the name
Google-Extended, so it will never appear in your access logs no matter how often your content is used. - robots.txt token
Google-Extended- Respects robots.txt
-
Documented as compliant
Google documents that Google-Extended has no separate HTTP user-agent string of its own — the crawling is done by Googlebot. You will therefore never see it as a distinct hit in your access logs, no matter how many Gemini answers your content ends up in.
- Operator docs
- developers.google.com
What Google-Extended does
Should you block Google-Extended?
Our take
This is the low-risk opt-out. Because Google-Extended governs what may be done with content Google has already fetched — rather than whether it may fetch at all — disallowing it keeps you in search while keeping you out of model training. That's the trade the token exists to let you make.
How to block Google-Extended
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Google-Extended token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Google-Extended
User-agent: Google-Extended Disallow: /
Allow Google-Extended explicitly
User-agent: Google-Extended Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Google-Extended
MetaPilot detects Google-Extended in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Google-Extended between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Google-CloudVertexBot
The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.
Gemini-Deep-Research
The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
Google-InspectionTool
The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test.
Googlebot
Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.
FAQ
What is Google-Extended?
Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.
Google-Extended is a robots.txt control token rather than a crawler in its own right. Googlebot does the fetching; this token tells Google what it may then do with the result — specifically, whether the content may train future Gemini models or be used for grounding in Gemini apps. It exists so publishers can separate "you may index me" from "you may train on me".
Does Google-Extended respect robots.txt?
Yes — Google documents that Google-Extended honours robots.txt. A Disallow for the Google-Extended token is respected.
Google documents that Google-Extended has no separate HTTP user-agent string of its own — the crawling is done by Googlebot. You will therefore never see it as a distinct hit in your access logs, no matter how many Gemini answers your content ends up in.
How do I block Google-Extended?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: Google-ExtendedDisallow: /
Use the Google-Extended token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Google-Extended hurt my traffic?
Disallowing Google-Extended withdraws your content from Gemini training and grounding. Google states this does not affect your inclusion in Google Search and is not a ranking signal, which makes it the lowest-risk training opt-out on this list. The flip side is that grounding is how Gemini quotes live sources, so you are giving up some AI visibility along with the training.
Ask an AI whether to block Google-Extended