What is robots.txt?
TL;DR
robots.txt is a plain-text file at yourdomain.com/robots.txt that tells automated visitors (crawlers) which parts of your site they're allowed to access. It's a set of requests at the front door — not a lock.
What it is
Imagine a sign just inside the entrance of a building that says "staff only" on a few doors. Most visitors respect it; a rude one could ignore it. That's robots.txt: a file listing Allow and Disallow rules for crawlers — Google's crawler, Bing's, and AI crawlers like GPTBot (ChatGPT), ClaudeBot, and PerplexityBot. It usually also points to your sitemap so crawlers can find your pages.
Well-behaved crawlers read and follow it. It's a convention, not enforcement — so it guides access, it doesn't secure anything.
Why it matters
- Steer crawl attention. Keep bots out of low-value or duplicate areas so they spend their time on pages that matter.
- Point the way. Linking your sitemap in
robots.txthelps engines discover new pages faster. - Decide who trains on you. You can allow or block individual AI crawlers — the training-vs-citation tradeoff. Blocking an AI bot keeps it out; allowing it can help you be cited in answers.
How MetaPilot handles it
MetaPilot generates a sensible robots.txt for you, lets you edit it, and serves it live at the edge. For AI crawlers specifically, it gives you a one-click Allow/Block per bot that safely rewrites only its managed section — leaving your own rules alone. Manage your robots.txt in MetaPilot →
Related
FAQ
Does blocking a page in robots.txt remove it from Google?
No. Disallow stops crawling, not indexing — a blocked URL can still appear if it's linked elsewhere. To keep a page out of results, use a noindex tag instead.
Is robots.txt the same as a sitemap?
No. A sitemap lists pages you want found; robots.txt sets access rules. They complement each other. What is a sitemap?
Can robots.txt stop AI from using my content?
It can ask named AI crawlers not to fetch your pages, and well-behaved ones comply — but it isn't a guarantee against every bot.
Where does robots.txt live?
Always at the root: yourdomain.com/robots.txt.
Ask an AI to explain this