# ClaudeBot

**Category:** Training | **Operated by:** Anthropic

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

---

## At a glance

| Property | Value |
|---|---|
| Operator | Anthropic |
| Category | Training |
| User-Agent token | `ClaudeBot` (also counted: `Claude-Web`, `anthropic-ai`) |
| robots.txt token | `ClaudeBot` |
| Respects robots.txt | Documented as compliant |
| Operator docs | support.claude.com |

---

## What ClaudeBot does

ClaudeBot crawls publicly available pages and adds them to the pool of content Anthropic may use to train its models. Anthropic runs it as a distinct token from the two Claude crawlers that serve live traffic, so you can decline training without declining citation. The older `Claude-Web` and `anthropic-ai` user agents belong to the same operator and MetaPilot counts them under this entry.

---

## Should you block ClaudeBot?

In Anthropic's own wording, blocking ClaudeBot means a site's future materials should be excluded from its AI model training datasets. It has no effect on whether Claude can find you (Claude-SearchBot) or read you mid-conversation (Claude-User) — those are separate tokens you would have to disallow separately.

---

## How to block ClaudeBot

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `ClaudeBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block ClaudeBot:**

```
User-agent: ClaudeBot
Disallow: /
```

**Allow ClaudeBot explicitly:**

```
User-agent: ClaudeBot
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles ClaudeBot

MetaPilot detects ClaudeBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip ClaudeBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related crawlers

- **Claude-SearchBot** (Anthropic) — Search index: Anthropic's indexing crawler, which builds the index Claude's web search draws on.
- **Claude-User** (Anthropic) — Answer engine: The fetcher Claude uses to read a specific page while answering a user who asked about it.
- **Applebot-Extended** (Apple) — Training: Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.
- **Bytespider** (ByteDance) — Training: ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

---

## FAQ

### What is ClaudeBot?
Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. ClaudeBot crawls publicly available pages and adds them to the pool of content Anthropic may use to train its models. Anthropic runs it as a distinct token from the two Claude crawlers that serve live traffic, so you can decline training without declining citation. The older `Claude-Web` and `anthropic-ai` user agents belong to the same operator and MetaPilot counts them under this entry.

### Does ClaudeBot respect robots.txt?
Yes — Anthropic documents that ClaudeBot honours `robots.txt`. A `Disallow` for the `ClaudeBot` token is respected.

### How do I block ClaudeBot?
Add these two lines to the `robots.txt` file at the root of your domain:

```
User-agent: ClaudeBot
Disallow: /
```

Use the `ClaudeBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

### Will blocking ClaudeBot hurt my traffic?
In Anthropic's own wording, blocking ClaudeBot means a site's future materials should be excluded from its AI model training datasets. It has no effect on whether Claude can find you (Claude-SearchBot) or read you mid-conversation (Claude-User) — those are separate tokens you would have to disallow separately.