# Claude-User

**Category:** Answer engine
**Operated by:** Anthropic

> The fetcher Claude uses to read a specific page while answering a user who asked about it.

---

## At a glance

| Property | Value |
|---|---|
| Operator | Anthropic |
| Category | Answer engine |
| User-Agent token | `Claude-User` |
| robots.txt token | `Claude-User` |
| Respects robots.txt | Documented as compliant |
| Operator docs | support.claude.com |

**Answer engine:** Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open.

---

## What Claude-User does

When someone asks Claude a question that needs the contents of a web page, Claude-User retrieves that page at that moment. It is not a bulk crawl and Anthropic does not use it to build a training set. What it reads can be summarised and cited back to the user with a link.

---

## Should you block Claude-User?

Anthropic's own description is blunt: disallowing Claude-User prevents its system from retrieving your content in response to a user query. That is a person actively asking, and you are opting out of being the answer. Leave this one open unless you have a specific reason not to.

**Our take:** Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention.

---

## How to block Claude-User

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Claude-User` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block Claude-User:**
```
User-agent: Claude-User
Disallow: /
```

**Allow Claude-User explicitly:**
```
User-agent: Claude-User
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It is also not retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles Claude-User

MetaPilot detects Claude-User in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Claude-User between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## FAQ

**What is Claude-User?**
The fetcher Claude uses to read a specific page while answering a user who asked about it. When someone asks Claude a question that needs the contents of a web page, Claude-User retrieves that page at that moment. It is not a bulk crawl and Anthropic does not use it to build a training set. What it reads can be summarised and cited back to the user with a link.

**Does Claude-User respect robots.txt?**
Yes — Anthropic documents that Claude-User honours `robots.txt`. A `Disallow` for the `Claude-User` token is respected.

**How do I block Claude-User?**
Add these two lines to the `robots.txt` file at the root of your domain:
`User-agent: Claude-User`
`Disallow: /`
Use the `Claude-User` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Will blocking Claude-User hurt my traffic?**
Anthropic's own description is blunt: disallowing Claude-User prevents its system from retrieving your content in response to a user query. That is a person actively asking, and you are opting out of being the answer. Leave this one open unless you have a specific reason not to.

---

## Related crawlers

- **Claude-SearchBot** (Anthropic) — Search index: Anthropic's indexing crawler, which builds the index Claude's web search draws on.
- **ClaudeBot** (Anthropic) — Training: Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.
- **DuckAssistBot** (DuckDuckGo) — Answer engine: DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results.
- **Gemini-Deep-Research** (Google) — Answer engine: The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.