Google-CloudVertexBot
The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.
At a glance
- Operator
- Category
-
Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
- User-Agent token
-
Google-CloudVertexBot - robots.txt token
Google-CloudVertexBot- Respects robots.txt
- Documented as compliant
- Operator docs
- developers.google.com
What Google-CloudVertexBot does
Should you block Google-CloudVertexBot?
How to block Google-CloudVertexBot
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Google-CloudVertexBot token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Google-CloudVertexBot
User-agent: Google-CloudVertexBot Disallow: /
Allow Google-CloudVertexBot explicitly
User-agent: Google-CloudVertexBot Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Google-CloudVertexBot
MetaPilot detects Google-CloudVertexBot in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Google-CloudVertexBot between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Google-Extended
Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.
Gemini-Deep-Research
The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
Google-InspectionTool
The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test.
Googlebot
Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.
FAQ
What is Google-CloudVertexBot?
The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.
Google-CloudVertexBot fetches pages on behalf of a Google Cloud customer who is building a Vertex AI Agent and has asked for that site to be crawled. Google describes it as crawling sites at the site owner's request. What it collects feeds that one customer's agent — it does not go into Google Search or general Gemini training.
Does Google-CloudVertexBot respect robots.txt?
Yes — Google documents that Google-CloudVertexBot honours robots.txt. A Disallow for the Google-CloudVertexBot token is respected.
How do I block Google-CloudVertexBot?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: Google-CloudVertexBotDisallow: /
Use the Google-CloudVertexBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Google-CloudVertexBot hurt my traffic?
Blocking it stops your site being pulled into third-party Vertex AI agents. Google states it has no effect on Google Search or other Google products, so the cost is limited. Do check that nobody at your own company is the Cloud customer in question first, because blocking this token will break their build.
Ask an AI whether to block Google-CloudVertexBot