# Claude-SearchBot

**Category:** Search index
**Operator:** Anthropic

Anthropic's indexing crawler, which builds the index Claude's web search draws on.

---

## At a glance

| Property | Value |
|---|---|
| Operator | Anthropic |
| Category | Search index |
| User-Agent token | `Claude-SearchBot` |
| robots.txt token | `Claude-SearchBot` |
| Respects robots.txt | Documented as compliant |
| Operator docs | support.claude.com |

**About this category:** Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.

---

## What Claude-SearchBot does

Claude-SearchBot crawls content to improve the quality and relevance of Claude's search results. It sits between the other two Anthropic bots: not training, not a live one-off fetch, but the index Claude consults to decide which pages are worth looking at in the first place.

---

## Should you block Claude-SearchBot?

Blocking Claude-SearchBot prevents Anthropic from indexing your content for search. Claude can then only reach you if a user hands it your URL directly. If your goal was to stay out of training, ClaudeBot is the token that does that — this one just makes you harder to find.

---

## How to block Claude-SearchBot

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Claude-SearchBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block Claude-SearchBot:**
```
User-agent: Claude-SearchBot
Disallow: /
```

**Allow Claude-SearchBot explicitly:**
```
User-agent: Claude-SearchBot
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles Claude-SearchBot

MetaPilot detects Claude-SearchBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Claude-SearchBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related crawlers

- **Claude-User** (Anthropic) — Answer engine. The fetcher Claude uses to read a specific page while answering a user who asked about it.
- **ClaudeBot** (Anthropic) — Training. Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.
- **Amazonbot** (Amazon) — Search index. Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
- **Meta-WebIndexer** (Meta) — Search index. The crawler that builds the search index Meta AI consults when it answers a question.

---

## FAQ

**What is Claude-SearchBot?**
Anthropic's indexing crawler, which builds the index Claude's web search draws on. Claude-SearchBot crawls content to improve the quality and relevance of Claude's search results. It sits between the other two Anthropic bots: not training, not a live one-off fetch, but the index Claude consults to decide which pages are worth looking at in the first place.

**Does Claude-SearchBot respect robots.txt?**
Yes — Anthropic documents that Claude-SearchBot honours `robots.txt`. A `Disallow` for the `Claude-SearchBot` token is respected.

**How do I block Claude-SearchBot?**
Add these two lines to the `robots.txt` file at the root of your domain:
`User-agent: Claude-SearchBot`
`Disallow: /`
Use the `Claude-SearchBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Will blocking Claude-SearchBot hurt my traffic?**
Blocking Claude-SearchBot prevents Anthropic from indexing your content for search. Claude can then only reach you if a user hands it your URL directly. If your goal was to stay out of training, ClaudeBot is the token that does that — this one just makes you harder to find.