# Google-CloudVertexBot **Category:** Training | **Operated by:** Google The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data. --- ## At a glance | Property | Value | |---|---| | Operator | Google | | Category | Training | | User-Agent token | `Google-CloudVertexBot` | | robots.txt token | `Google-CloudVertexBot` | | Respects robots.txt | Documented as compliant | | Operator docs | developers.google.com | > **Training category:** Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live. --- ## What Google-CloudVertexBot does Google-CloudVertexBot fetches pages on behalf of a Google Cloud customer who is building a Vertex AI Agent and has asked for that site to be crawled. Google describes it as crawling sites at the site owner's request. What it collects feeds that one customer's agent — it does not go into Google Search or general Gemini training. --- ## Should you block Google-CloudVertexBot? Blocking it stops your site being pulled into third-party Vertex AI agents. Google states it has no effect on Google Search or other Google products, so the cost is limited. Do check that nobody at your own company is the Cloud customer in question first, because blocking this token will break their build. --- ## How to block Google-CloudVertexBot Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Google-CloudVertexBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block Google-CloudVertexBot:** ``` User-agent: Google-CloudVertexBot Disallow: / ``` **Allow Google-CloudVertexBot explicitly:** ``` User-agent: Google-CloudVertexBot Allow: / ``` > **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles Google-CloudVertexBot MetaPilot detects Google-CloudVertexBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Google-CloudVertexBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## FAQ ### What is Google-CloudVertexBot? Google-CloudVertexBot is the crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data. It fetches pages on behalf of a Google Cloud customer building a Vertex AI Agent. What it collects feeds that one customer's agent — it does not go into Google Search or general Gemini training. ### Does Google-CloudVertexBot respect robots.txt? Yes — Google documents that Google-CloudVertexBot honours `robots.txt`. A `Disallow` for the `Google-CloudVertexBot` token is respected. ### How do I block Google-CloudVertexBot? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: Google-CloudVertexBot Disallow: / ``` Use the `Google-CloudVertexBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking Google-CloudVertexBot hurt my traffic? Blocking it stops your site being pulled into third-party Vertex AI agents. Google states it has no effect on Google Search or other Google products, so the cost is limited. Check that nobody at your own company is the Cloud customer in question first, because blocking this token will break their build. --- ## Related crawlers - **Google-Extended** (Training) — Google's AI opt-out token: controls whether content Google already crawls may be used to train and ground Gemini. - **Gemini-Deep-Research** (Answer engine) — The agent behind Gemini's Deep Research mode. - **Google-InspectionTool** (Search engine) — The fetcher behind Search Console testing tools. - **Googlebot** (Search engine) — Google's main search crawler.