What is ChatGPT-User? (OpenAI's AI crawler)

At a glance

Operator: OpenAI
Category: Answer engine
Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open.
User-Agent token: ChatGPT-User
robots.txt token: ChatGPT-User
Respects robots.txt: Not reliably
OpenAI's own documentation says that because these fetches are initiated by a user rather than by an automatic crawl, "robots.txt rules may not apply" to ChatGPT-User. A Disallow for this token is best read as a request OpenAI may honour, not a hard block.
Operator docs: developers.openai.com

What ChatGPT-User does

When a ChatGPT user — or a custom GPT, or an app calling OpenAI's browsing tools — needs the contents of a particular URL, ChatGPT-User fetches that one page on the spot. It is not a bulk crawler and it is not what decides whether you appear in ChatGPT search. The page it retrieves can end up quoted and linked in the answer the person sees.

Should you block ChatGPT-User?

Blocking ChatGPT-User is the expensive one. It removes your page from an answer being written for a real person who is asking about your subject right now, and it takes the citation link with it. Very few sites come out ahead on that trade.

Our take

Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention.

How to block ChatGPT-User

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the ChatGPT-User token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block ChatGPT-User

robots.txt

User-agent: ChatGPT-User
Disallow: /

Allow ChatGPT-User explicitly

robots.txt

User-agent: ChatGPT-User
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles ChatGPT-User

MetaPilot detects ChatGPT-User in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip ChatGPT-User between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

GPTBot

OpenAI

Training

OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models.

OAI-SearchBot

OpenAI

Search index

The crawler that builds the search index behind ChatGPT's search results.

Claude-User

Anthropic

Answer engine

The fetcher Claude uses to read a specific page while answering a user who asked about it.

DuckAssistBot

DuckDuckGo

Answer engine

DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results.

Browse every AI crawler

FAQ

What is ChatGPT-User?

The fetcher ChatGPT uses to open a specific page in real time while answering someone.

When a ChatGPT user — or a custom GPT, or an app calling OpenAI's browsing tools — needs the contents of a particular URL, ChatGPT-User fetches that one page on the spot. It is not a bulk crawler and it is not what decides whether you appear in ChatGPT search. The page it retrieves can end up quoted and linked in the answer the person sees.

Does ChatGPT-User respect robots.txt?

Not reliably. A Disallow for ChatGPT-User is worth adding, but it should not be treated as a guarantee.

OpenAI's own documentation says that because these fetches are initiated by a user rather than by an automatic crawl, "robots.txt rules may not apply" to ChatGPT-User. A Disallow for this token is best read as a request OpenAI may honour, not a hard block.

How do I block ChatGPT-User?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: ChatGPT-User
Disallow: /

Use the ChatGPT-User token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking ChatGPT-User hurt my traffic?

Blocking ChatGPT-User is the expensive one. It removes your page from an answer being written for a real person who is asking about your subject right now, and it takes the citation link with it. Very few sites come out ahead on that trade.