# Googlebot **Category:** Search engine · **Operated by:** Google Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds. --- ## At a glance | Property | Value | |---|---| | Operator | Google | | Category | Search engine | | User-Agent token | `Googlebot` | | robots.txt token | `Googlebot` | | Respects robots.txt | Documented as compliant | | Operator docs | developers.google.com | **Category note:** A classic search crawler whose index also feeds that company's AI answers. Blocking it removes you from both the blue links and the AI summaries above them. --- ## What Googlebot does Googlebot fetches pages to build the Google Search index, crawling as both a smartphone and a desktop client from the same token. It is the single most consequential visit your site receives. The same index is what AI Overviews and Gemini grounding draw on, which is why an AI-visibility question so often turns out to be a plain crawlability question underneath. --- ## Should you block Googlebot? Blocking Googlebot removes you from Google Search. Not "hurts your ranking" — removes. It also removes you from AI Overviews, since those are assembled from the index Googlebot fills. There is essentially no situation where a site wants this, which is why MetaPilot will not write that rule for you. > **Our take:** Don't. Blocking Googlebot removes you from Google altogether — the ordinary results and the AI answers built on top of them. If what you actually want is to stay in search while staying out of model training, that is a separate token, and it's the one to reach for. MetaPilot shows Googlebot in your crawler analytics but won't write a block for it from the dashboard. --- ## The robots.txt rules for Googlebot Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Googlebot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. ⚠️ **The allow rule is the one you want.** The disallow variant is documented below for completeness, but it removes your site from Google entirely. Don't paste it unless that is genuinely what you intend. ### Allow Googlebot explicitly ``` User-agent: Googlebot Allow: / ``` ### Block Googlebot — removes you from Google ``` User-agent: Googlebot Disallow: / ``` **A note on what this does and doesn't do.** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles Googlebot MetaPilot detects Googlebot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Googlebot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related crawlers - **Google-InspectionTool** (Google) — The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test. - **Googlebot-Image** (Google) — Googlebot's image-fetching counterpart, which populates Google Images. - **Googlebot-News** (Google) — A robots.txt token for Google News, not a separate crawler — the fetching is done by Googlebot. - **Googlebot-Video** (Google) — Googlebot's video-fetching counterpart, which populates Google's video results. --- ## FAQ **What is Googlebot?** Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds. Googlebot fetches pages to build the Google Search index, crawling as both a smartphone and a desktop client from the same token. It is the single most consequential visit your site receives. The same index is what AI Overviews and Gemini grounding draw on. **Does Googlebot respect robots.txt?** Yes — Google documents that Googlebot honours `robots.txt`. A `Disallow` for the `Googlebot` token is respected. Whether Google may use what it fetched for generative AI is a separate decision, controlled by the Google-Extended token. **Should I block Googlebot?** Almost certainly not. Disallowing `Googlebot` removes you from Google's search results, and because those same results feed Google's AI answers, you lose both at once. If you want to stay in search but out of generative AI, that is a different token. MetaPilot will not write a block for Googlebot from the dashboard for this reason. **Will blocking Googlebot hurt my traffic?** Blocking Googlebot removes you from Google Search. Not "hurts your ranking" — removes. It also removes you from AI Overviews, since those are assembled from the index Googlebot fills. There is essentially no situation where a site wants this.