What is Claude-SearchBot? (Anthropic crawler)

At a glance

Operator: Anthropic
Category: Search index
Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.
User-Agent token: Claude-SearchBot
robots.txt token: Claude-SearchBot
Respects robots.txt: Documented as compliant
Operator docs: support.claude.com

What Claude-SearchBot does

Claude-SearchBot crawls content to improve the quality and relevance of Claude's search results. It sits between the other two Anthropic bots: not training, not a live one-off fetch, but the index Claude consults to decide which pages are worth looking at in the first place.

Should you block Claude-SearchBot?

Blocking Claude-SearchBot prevents Anthropic from indexing your content for search. Claude can then only reach you if a user hands it your URL directly. If your goal was to stay out of training, ClaudeBot is the token that does that — this one just makes you harder to find.

How to block Claude-SearchBot

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the Claude-SearchBot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block Claude-SearchBot

robots.txt

User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot explicitly

robots.txt

User-agent: Claude-SearchBot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles Claude-SearchBot

MetaPilot detects Claude-SearchBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Claude-SearchBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

Claude-User

Anthropic

Answer engine

The fetcher Claude uses to read a specific page while answering a user who asked about it.

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Amazonbot

Amazon

Search index

Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.

Meta-WebIndexer

FAQ

What is Claude-SearchBot?

Anthropic's indexing crawler, which builds the index Claude's web search draws on.

Claude-SearchBot crawls content to improve the quality and relevance of Claude's search results. It sits between the other two Anthropic bots: not training, not a live one-off fetch, but the index Claude consults to decide which pages are worth looking at in the first place.

Does Claude-SearchBot respect robots.txt?

Yes — Anthropic documents that Claude-SearchBot honours robots.txt. A Disallow for the Claude-SearchBot token is respected.

How do I block Claude-SearchBot?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: Claude-SearchBot
Disallow: /

Use the Claude-SearchBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking Claude-SearchBot hurt my traffic?

Blocking Claude-SearchBot prevents Anthropic from indexing your content for search. Claude can then only reach you if a user hands it your URL directly. If your goal was to stay out of training, ClaudeBot is the token that does that — this one just makes you harder to find.