# Googlebot-News **Category:** Search engine · **Operated by:** Google A robots.txt token for Google News, not a separate crawler — the fetching is done by Googlebot. --- ## At a glance | Property | Value | |---|---| | Operator | Google | | Category | Search engine | | User-Agent token | None — this is a robots.txt control token, not a crawler. | | robots.txt token | `Googlebot-News` | | Respects robots.txt | Documented as compliant | | Operator docs | developers.google.com | > Google fetches with its own crawler and reads this token only to decide what it may then do with the result. No request ever carries the name `Googlebot-News`, so it will never appear in your access logs no matter how often your content is used. --- ## What Googlebot-News does Google-News lets a publisher control inclusion in Google News independently of ordinary search. Google documents that it has no user agent of its own: crawling happens under the Googlebot user agent, and this token only governs what Google News may do with the result. --- ## Should you block Googlebot-News? Disallowing it removes you from Google News while leaving ordinary Search inclusion intact. Only relevant if you publish news. **Our take:** Don't. Blocking Googlebot-News removes you from Google altogether — the ordinary results and the AI answers built on top of them. If what you actually want is to stay in search while staying out of model training, that is a separate token, and it's the one to reach for. MetaPilot shows Googlebot-News in your crawler analytics but won't write a block for it from the dashboard. --- ## The robots.txt rules for Googlebot-News Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Googlebot-News` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. > **The allow rule is the one you want.** The disallow variant is documented below for completeness, but it removes your site from Google entirely. Don't paste it unless that is genuinely what you intend. **Allow Googlebot-News explicitly:** ``` User-agent: Googlebot-News Allow: / ``` **Block Googlebot-News — removes you from Google:** ``` User-agent: Googlebot-News Disallow: / ``` **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles Googlebot-News MetaPilot detects Googlebot-News in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Googlebot-News between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related crawlers - **Google-InspectionTool** (Google) — The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test. - **Googlebot** (Google) — Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds. - **Googlebot-Image** (Google) — Googlebot's image-fetching counterpart, which populates Google Images. - **Googlebot-Video** (Google) — Googlebot's video-fetching counterpart, which populates Google's video results. --- ## FAQ **What is Googlebot-News?** A robots.txt token for Google News, not a separate crawler — the fetching is done by Googlebot. Google-News lets a publisher control inclusion in Google News independently of ordinary search. Google documents that it has no user agent of its own: crawling happens under the Googlebot user agent, and this token only governs what Google News may do with the result. **Does Googlebot-News respect robots.txt?** Yes — Google documents that Googlebot-News honours `robots.txt`. A `Disallow` for the `Googlebot-News` token is respected. Because this is a control token rather than a crawler, it will never appear as a hit in your logs. MetaPilot shows it as "policy only" for exactly that reason. **Should I block Googlebot-News?** Almost certainly not. Disallowing `Googlebot-News` removes you from Google's search results, and because those same results feed Google's AI answers, you lose both at once. If you want to stay in search but out of generative AI, that is a different token. MetaPilot will not write a block for Googlebot-News from the dashboard for this reason. **Will blocking Googlebot-News hurt my traffic?** Disallowing it removes you from Google News while leaving ordinary Search inclusion intact. Only relevant if you publish news.