# Google-Extended

**Category:** Training · **Operator:** Google

> Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.

---

## At a Glance

| Property | Value |
|---|---|
| Operator | Google |
| Category | Training |
| User-Agent token | None — this is a robots.txt control token, not a crawler |
| robots.txt token | `Google-Extended` |
| Respects robots.txt | ✅ Documented as compliant |

**Note on User-Agent:** Google fetches with its own crawler (Googlebot) and reads the `Google-Extended` token only to decide what it may then do with the result. No request ever carries the name `Google-Extended`, so it will never appear in your access logs no matter how often your content is used.

---

## What Google-Extended Does

Google-Extended is a robots.txt control token rather than a crawler in its own right. Googlebot does the fetching; this token tells Google what it may then do with the result — specifically, whether the content may train future Gemini models or be used for grounding in Gemini apps. It exists so publishers can separate "you may index me" from "you may train on me".

---

## Should You Block Google-Extended?

Disallowing Google-Extended withdraws your content from Gemini training and grounding. Google states this does not affect your inclusion in Google Search and is not a ranking signal, which makes it the lowest-risk training opt-out available. The flip side is that grounding is how Gemini quotes live sources, so you are giving up some AI visibility along with the training.

**Our take:** This is the low-risk opt-out. Because Google-Extended governs what may be done with content Google has already fetched — rather than whether it may fetch at all — disallowing it keeps you in search while keeping you out of model training. That's the trade the token exists to let you make.

---

## How to Block Google-Extended

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Google-Extended` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block Google-Extended:**
```
User-agent: Google-Extended
Disallow: /
```

**Allow Google-Extended explicitly:**
```
User-agent: Google-Extended
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot Handles Google-Extended

MetaPilot detects Google-Extended in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Google-Extended between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related Crawlers

- **Google-CloudVertexBot** (Training) — The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.
- **Gemini-Deep-Research** (Answer engine) — The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
- **Google-InspectionTool** (Search engine) — The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test.
- **Googlebot** (Search engine) — Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.

---

## FAQ

**What is Google-Extended?**
Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini. Googlebot does the fetching; this token tells Google what it may then do with the result — specifically, whether the content may train future Gemini models or be used for grounding in Gemini apps.

**Does Google-Extended respect robots.txt?**
Yes — Google documents that Google-Extended honours `robots.txt`. A `Disallow` for the `Google-Extended` token is respected. It has no separate HTTP user-agent string; crawling is done by Googlebot, so it never appears as a distinct hit in access logs.

**How do I block Google-Extended?**
Add `User-agent: Google-Extended` and `Disallow: /` to your `robots.txt` file. Repeat the rule on every subdomain you want covered.

**Will blocking Google-Extended hurt my traffic?**
Google states that disallowing Google-Extended does not affect your inclusion in Google Search and is not a ranking signal. You will give up some AI visibility (Gemini grounding) but retain your Search presence.