# GPTBot **Operated by:** OpenAI **Category:** Training OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models. --- ## At a glance | Property | Value | |---|---| | Operator | OpenAI | | Category | Training | | User-Agent token | `GPTBot` | | robots.txt token | `GPTBot` | | Respects robots.txt | Documented as compliant | | Operator docs | developers.openai.com | > **Training category:** Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live. --- ## What GPTBot does GPTBot crawls publicly reachable pages across the open web and keeps them as candidate training data for OpenAI's foundation models. OpenAI states it filters out paywalled sources, content that violates its policies, and sources that gather personally identifiable information. It is a separate token from the two OpenAI crawlers that serve live traffic, so the training decision can be made on its own. --- ## Should you block GPTBot? Blocking GPTBot keeps your future pages out of OpenAI training runs. It does not remove you from ChatGPT: search results still come from OAI-SearchBot and live page reads still come from ChatGPT-User, each of which reads its own robots.txt token. This is a rights decision, not a visibility one. --- ## How to block GPTBot Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `GPTBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block GPTBot:** ``` User-agent: GPTBot Disallow: / ``` **Allow GPTBot explicitly:** ``` User-agent: GPTBot Allow: / ``` > **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles GPTBot MetaPilot detects GPTBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip GPTBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related crawlers - **ChatGPT-User** (OpenAI) — Answer engine: The fetcher ChatGPT uses to open a specific page in real time while answering someone. - **OAI-SearchBot** (OpenAI) — Search index: The crawler that builds the search index behind ChatGPT's search results. - **ClaudeBot** (Anthropic) — Training: Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. - **Applebot-Extended** (Apple) — Training: Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. --- ## FAQ ### What is GPTBot? OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models. GPTBot crawls publicly reachable pages across the open web and keeps them as candidate training data for OpenAI's foundation models. OpenAI states it filters out paywalled sources, content that violates its policies, and sources that gather personally identifiable information. It is a separate token from the two OpenAI crawlers that serve live traffic, so the training decision can be made on its own. ### Does GPTBot respect robots.txt? Yes — OpenAI documents that GPTBot honours `robots.txt`. A `Disallow` for the `GPTBot` token is respected. ### How do I block GPTBot? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: GPTBot Disallow: / ``` Use the `GPTBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking GPTBot hurt my traffic? Blocking GPTBot keeps your future pages out of OpenAI training runs. It does not remove you from ChatGPT: search results still come from OAI-SearchBot and live page reads still come from ChatGPT-User, each of which reads its own robots.txt token. This is a rights decision, not a visibility one.