Gemini-Deep-Research
The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
At a glance
- Operator
- Category
-
Answer engine
Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open.
- User-Agent token
-
Gemini-Deep-Research - robots.txt token
Gemini-Deep-Research- Respects robots.txt
-
Not reliably
Google has not documented this agent: it appears on neither Google's crawler list nor its list of user-triggered fetchers, and is known from server logs and third-party bot directories. Google's stated rule for fetches a user requested is that they generally ignore robots.txt rules. Some directories say this agent respects robots.txt; none cite Google.
What Gemini-Deep-Research does
This page is about the crawler. For the product it works for, read What is Gemini Deep Research?
Should you block Gemini-Deep-Research?
Our take
Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention.
How to block Gemini-Deep-Research
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Gemini-Deep-Research token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Gemini-Deep-Research
User-agent: Gemini-Deep-Research Disallow: /
Allow Gemini-Deep-Research explicitly
User-agent: Gemini-Deep-Research Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Gemini-Deep-Research
MetaPilot detects Gemini-Deep-Research in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Gemini-Deep-Research between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Google-CloudVertexBot
The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.
Google-Extended
Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.
Google-InspectionTool
The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test.
Googlebot
Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.
FAQ
What is Gemini-Deep-Research?
The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
Deep Research runs a long chain of searches and page reads to assemble a written report for a single user. Where a normal crawl is broad and steady, this arrives as a burst of targeted fetches whenever someone researches a topic your pages are relevant to. Sources it reads are typically listed and linked in the finished report.
Does Gemini-Deep-Research respect robots.txt?
Not reliably. A Disallow for Gemini-Deep-Research is worth adding, but it should not be treated as a guarantee.
Google has not documented this agent: it appears on neither Google's crawler list nor its list of user-triggered fetchers, and is known from server logs and third-party bot directories. Google's stated rule for fetches a user requested is that they generally ignore robots.txt rules. Some directories say this agent respects robots.txt; none cite Google.
How do I block Gemini-Deep-Research?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: Gemini-Deep-ResearchDisallow: /
Use the Gemini-Deep-Research token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Gemini-Deep-Research hurt my traffic?
A block aims at the long, heavily cited reports where being a named source is worth the most, and it may not keep you out of them. Deep Research also draws on the Google Search index, so a page can reach a report without this agent visiting; the documented control over that route is the Google-Extended token. Neither has any bearing on Google Search rankings.
Ask an AI whether to block Gemini-Deep-Research