# Gemini-Deep-Research **Category:** Answer engine | **Operated by:** Google The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user. --- ## At a glance | Property | Value | |---|---| | Operator | Google | | Category | Answer engine | | User-Agent token | `Gemini-Deep-Research` | | robots.txt token | `Gemini-Deep-Research` | | Respects robots.txt | Not reliably | **Note on robots.txt compliance:** Google has not documented this agent — it appears on neither Google's crawler list nor its list of user-triggered fetchers, and is known from server logs and third-party bot directories. Google's stated rule for user-triggered fetches is that they generally ignore robots.txt rules. Some directories say this agent respects robots.txt; none cite Google. **Category note:** Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open. --- ## What Gemini-Deep-Research does Deep Research runs a long chain of searches and page reads to assemble a written report for a single user. Where a normal crawl is broad and steady, this arrives as a burst of targeted fetches whenever someone researches a topic your pages are relevant to. Sources it reads are typically listed and linked in the finished report. --- ## Should you block Gemini-Deep-Research? A block aims at the long, heavily cited reports where being a named source is worth the most, and it may not keep you out of them. Deep Research also draws on the Google Search index, so a page can reach a report without this agent visiting; the documented control over that route is the `Google-Extended` token. Neither has any bearing on Google Search rankings. **Our take:** Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention. --- ## How to block Gemini-Deep-Research Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Gemini-Deep-Research` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block Gemini-Deep-Research:** ``` User-agent: Gemini-Deep-Research Disallow: / ``` **Allow Gemini-Deep-Research explicitly:** ``` User-agent: Gemini-Deep-Research Allow: / ``` > **A note on what this does and doesn't do.** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles Gemini-Deep-Research MetaPilot detects Gemini-Deep-Research in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Gemini-Deep-Research between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## FAQ ### What is Gemini-Deep-Research? The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user. Deep Research runs a long chain of searches and page reads to assemble a written report for a single user. Where a normal crawl is broad and steady, this arrives as a burst of targeted fetches whenever someone researches a topic your pages are relevant to. Sources it reads are typically listed and linked in the finished report. ### Does Gemini-Deep-Research respect robots.txt? Not reliably. A `Disallow` for `Gemini-Deep-Research` is worth adding, but it should not be treated as a guarantee. Google has not documented this agent and Google's stated rule for user-triggered fetches is that they generally ignore robots.txt rules. ### How do I block Gemini-Deep-Research? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: Gemini-Deep-Research Disallow: / ``` Use the `Gemini-Deep-Research` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking Gemini-Deep-Research hurt my traffic? A block aims at the long, heavily cited reports where being a named source is worth the most, and it may not keep you out of them. Deep Research also draws on the Google Search index, so a page can reach a report without this agent visiting. Neither blocking this agent nor `Google-Extended` has any bearing on Google Search rankings. --- ## Related crawlers - **Google-CloudVertexBot** (Google) — Training: The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data. - **Google-Extended** (Google) — Training: Google's AI opt-out token; controls whether content Google already crawls may be used to train and ground Gemini. - **Google-InspectionTool** (Google) — Search engine: The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test. - **Googlebot** (Google) — Search engine: Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.