Claude-User
The fetcher Claude uses to read a specific page while answering a user who asked about it.
At a glance
- Operator
- Anthropic
- Category
-
Answer engine
Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open.
- User-Agent token
-
Claude-User - robots.txt token
Claude-User- Respects robots.txt
- Documented as compliant
- Operator docs
- support.claude.com
What Claude-User does
Should you block Claude-User?
Our take
Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention.
How to block Claude-User
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Claude-User token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Claude-User
User-agent: Claude-User Disallow: /
Allow Claude-User explicitly
User-agent: Claude-User Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Claude-User
MetaPilot detects Claude-User in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Claude-User between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Claude-SearchBot
Anthropic
Anthropic's indexing crawler, which builds the index Claude's web search draws on.
ClaudeBot
Anthropic
Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.
DuckAssistBot
DuckDuckGo
DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results.
Gemini-Deep-Research
The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.
FAQ
What is Claude-User?
The fetcher Claude uses to read a specific page while answering a user who asked about it.
When someone asks Claude a question that needs the contents of a web page, Claude-User retrieves that page at that moment. It is not a bulk crawl and Anthropic does not use it to build a training set. What it reads can be summarised and cited back to the user with a link.
Does Claude-User respect robots.txt?
Yes — Anthropic documents that Claude-User honours robots.txt. A Disallow for the Claude-User token is respected.
How do I block Claude-User?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: Claude-UserDisallow: /
Use the Claude-User token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Claude-User hurt my traffic?
Anthropic's own description is blunt: disallowing Claude-User prevents its system from retrieving your content in response to a user query. That is a person actively asking, and you are opting out of being the answer. Leave this one open unless you have a specific reason not to.
Ask an AI whether to block Claude-User