# What is robots.txt?

**Concept**

**TL;DR:** robots.txt is a plain-text file at `yourdomain.com/robots.txt` that tells automated visitors (crawlers) which parts of your site they're allowed to access. It's a set of requests at the front door — not a lock.

---

## What it is

Imagine a sign just inside the entrance of a building that says "staff only" on a few doors. Most visitors respect it; a rude one could ignore it. That's `robots.txt`: a file listing `Allow` and `Disallow` rules for crawlers — Google's crawler, Bing's, and AI crawlers like GPTBot (ChatGPT), ClaudeBot, and PerplexityBot. It usually also points to your sitemap so crawlers can find your pages.

Well-behaved crawlers read and follow it. It's a convention, not enforcement — so it guides access, it doesn't secure anything.

---

## Why it matters

- **Steer crawl attention.** Keep bots out of low-value or duplicate areas so they spend their time on pages that matter.
- **Point the way.** Linking your sitemap in `robots.txt` helps engines discover new pages faster.
- **Decide who trains on you.** You can allow or block individual AI crawlers — the training-vs-citation tradeoff. Blocking an AI bot keeps it out; allowing it can help you be cited in answers.

---

## How MetaPilot handles it

MetaPilot generates a sensible `robots.txt` for you, lets you edit it, and serves it live at the edge. For AI crawlers specifically, it gives you a one-click Allow/Block per bot that safely rewrites only its managed section — leaving your own rules alone.

---

## FAQ

**Does blocking a page in robots.txt remove it from Google?**
No. `Disallow` stops crawling, not indexing — a blocked URL can still appear if it's linked elsewhere. To keep a page out of results, use a `noindex` tag instead.

**Is robots.txt the same as a sitemap?**
No. A sitemap lists pages you *want* found; robots.txt sets *access rules*. They complement each other.

**Can robots.txt stop AI from using my content?**
It can ask named AI crawlers not to fetch your pages, and well-behaved ones comply — but it isn't a guarantee against every bot.

**Where does robots.txt live?**
Always at the root: `yourdomain.com/robots.txt`.

---

## Related

- What is a sitemap?
- What are llms.txt and llms-full.txt?
- What is an LLM (and an answer engine)?