What is GPTBot? (OpenAI's AI crawler)

At a glance

Operator: OpenAI
Category: Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
User-Agent token: GPTBot
robots.txt token: GPTBot
Respects robots.txt: Documented as compliant
Operator docs: developers.openai.com

What GPTBot does

GPTBot crawls publicly reachable pages across the open web and keeps them as candidate training data for OpenAI's foundation models. OpenAI states it filters out paywalled sources, content that violates its policies, and sources that gather personally identifiable information. It is a separate token from the two OpenAI crawlers that serve live traffic, so the training decision can be made on its own.

Should you block GPTBot?

Blocking GPTBot keeps your future pages out of OpenAI training runs. It does not remove you from ChatGPT: search results still come from OAI-SearchBot and live page reads still come from ChatGPT-User, each of which reads its own robots.txt token. This is a rights decision, not a visibility one.

How to block GPTBot

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the GPTBot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block GPTBot

robots.txt

User-agent: GPTBot
Disallow: /

Allow GPTBot explicitly

robots.txt

User-agent: GPTBot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles GPTBot

MetaPilot detects GPTBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip GPTBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ChatGPT-User

OpenAI

Answer engine

The fetcher ChatGPT uses to open a specific page in real time while answering someone.

OAI-SearchBot

OpenAI

Search index

The crawler that builds the search index behind ChatGPT's search results.

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Applebot-Extended

Apple

Training

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

Browse every AI crawler

FAQ

What is GPTBot?

OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models.

GPTBot crawls publicly reachable pages across the open web and keeps them as candidate training data for OpenAI's foundation models. OpenAI states it filters out paywalled sources, content that violates its policies, and sources that gather personally identifiable information. It is a separate token from the two OpenAI crawlers that serve live traffic, so the training decision can be made on its own.

Does GPTBot respect robots.txt?

Yes — OpenAI documents that GPTBot honours robots.txt. A Disallow for the GPTBot token is respected.

How do I block GPTBot?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: GPTBot
Disallow: /

Use the GPTBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking GPTBot hurt my traffic?

Blocking GPTBot keeps your future pages out of OpenAI training runs. It does not remove you from ChatGPT: search results still come from OAI-SearchBot and live page reads still come from ChatGPT-User, each of which reads its own robots.txt token. This is a rights decision, not a visibility one.