What is OAI-SearchBot? (OpenAI's AI crawler)

At a glance

Operator: OpenAI
Category: Search index
Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.
User-Agent token: OAI-SearchBot
robots.txt token: OAI-SearchBot
Respects robots.txt: Documented as compliant
Operator docs: developers.openai.com

What OAI-SearchBot does

OAI-SearchBot crawls pages to build and refresh the index ChatGPT consults before it answers. OpenAI states it is not used for model training. Being in that index is what makes your site eligible to be surfaced and linked when someone searches inside ChatGPT.

Should you block OAI-SearchBot?

Blocking OAI-SearchBot takes you out of ChatGPT search entirely — you cannot be linked from a result you are not in. It also does nothing about training data, which is GPTBot's remit, so blocking it as a privacy measure gets you the cost without the benefit. OpenAI notes robots.txt changes take roughly a day to be picked up.

How to block OAI-SearchBot

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the OAI-SearchBot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block OAI-SearchBot

robots.txt

User-agent: OAI-SearchBot
Disallow: /

Allow OAI-SearchBot explicitly

robots.txt

User-agent: OAI-SearchBot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles OAI-SearchBot

MetaPilot detects OAI-SearchBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip OAI-SearchBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ChatGPT-User

OpenAI

Answer engine

The fetcher ChatGPT uses to open a specific page in real time while answering someone.

GPTBot

OpenAI

Training

OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models.

Amazonbot

Amazon

Search index

Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.

Claude-SearchBot

Anthropic

Search index

Anthropic's indexing crawler, which builds the index Claude's web search draws on.

Browse every AI crawler

FAQ

What is OAI-SearchBot?

The crawler that builds the search index behind ChatGPT's search results.

OAI-SearchBot crawls pages to build and refresh the index ChatGPT consults before it answers. OpenAI states it is not used for model training. Being in that index is what makes your site eligible to be surfaced and linked when someone searches inside ChatGPT.

Does OAI-SearchBot respect robots.txt?

Yes — OpenAI documents that OAI-SearchBot honours robots.txt. A Disallow for the OAI-SearchBot token is respected.

How do I block OAI-SearchBot?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: OAI-SearchBot
Disallow: /

Use the OAI-SearchBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking OAI-SearchBot hurt my traffic?

Blocking OAI-SearchBot takes you out of ChatGPT search entirely — you cannot be linked from a result you are not in. It also does nothing about training data, which is GPTBot's remit, so blocking it as a privacy measure gets you the cost without the benefit. OpenAI notes robots.txt changes take roughly a day to be picked up.