# OAI-SearchBot

**Category:** Search index | **Operated by:** OpenAI

The crawler that builds the search index behind ChatGPT's search results.

---

## At a glance

| Property | Value |
|---|---|
| Operator | OpenAI |
| Category | Search index |
| User-Agent token | `OAI-SearchBot` |
| robots.txt token | `OAI-SearchBot` |
| Respects robots.txt | Documented as compliant |
| Operator docs | developers.openai.com |

**Search index** — Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.

---

## What OAI-SearchBot does

OAI-SearchBot crawls pages to build and refresh the index ChatGPT consults before it answers. OpenAI states it is not used for model training. Being in that index is what makes your site eligible to be surfaced and linked when someone searches inside ChatGPT.

---

## Should you block OAI-SearchBot?

Blocking OAI-SearchBot takes you out of ChatGPT search entirely — you cannot be linked from a result you are not in. It also does nothing about training data, which is GPTBot's remit, so blocking it as a privacy measure gets you the cost without the benefit. OpenAI notes robots.txt changes take roughly a day to be picked up.

---

## How to block OAI-SearchBot

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `OAI-SearchBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block OAI-SearchBot:**

```
User-agent: OAI-SearchBot
Disallow: /
```

**Allow OAI-SearchBot explicitly:**

```
User-agent: OAI-SearchBot
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles OAI-SearchBot

MetaPilot detects OAI-SearchBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip OAI-SearchBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related crawlers

- **ChatGPT-User** (OpenAI) — Answer engine. The fetcher ChatGPT uses to open a specific page in real time while answering someone.
- **GPTBot** (OpenAI) — Training. OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models.
- **Amazonbot** (Amazon) — Search index. Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
- **Claude-SearchBot** (Anthropic) — Search index. Anthropic's indexing crawler, which builds the index Claude's web search draws on.

---

## FAQ

**What is OAI-SearchBot?**
The crawler that builds the search index behind ChatGPT's search results. OAI-SearchBot crawls pages to build and refresh the index ChatGPT consults before it answers. OpenAI states it is not used for model training. Being in that index is what makes your site eligible to be surfaced and linked when someone searches inside ChatGPT.

**Does OAI-SearchBot respect robots.txt?**
Yes — OpenAI documents that OAI-SearchBot honours `robots.txt`. A `Disallow` for the `OAI-SearchBot` token is respected.

**How do I block OAI-SearchBot?**
Add these two lines to the `robots.txt` file at the root of your domain:
`User-agent: OAI-SearchBot`
`Disallow: /`
Use the `OAI-SearchBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Will blocking OAI-SearchBot hurt my traffic?**
Blocking OAI-SearchBot takes you out of ChatGPT search entirely — you cannot be linked from a result you are not in. It also does nothing about training data, which is GPTBot's remit, so blocking it as a privacy measure gets you the cost without the benefit. OpenAI notes robots.txt changes take roughly a day to be picked up.