Monitor
5 min
See & control which AI bots crawl you
Answer engines can only cite what they read. This guide shows how to see which AI crawlers visit your site and decide, per bot, who gets in. New to AI crawlers? See What is an LLM (and an answer engine)? and What is robots.txt?
-
1
Open your dashboard
See which AI bots have crawled your site — GPTBot (ChatGPT), ClaudeBot, PerplexityBot, Google-Extended and more — how often, and which of your pages they read most. -
2
Read the training-vs-citation tradeoff
Blocking a bot keeps it out; allowing it can help you be cited in that engine's answers. MetaPilot flags what each bot is used for. -
3
Allow or block in one click
Open Agent Web → Robots & LLMs and use the per-bot Allow / Block toggle. MetaPilot rewrites only its managed block in your robots.txt — your own rules are untouched. -
4
It takes effect at the edge
The Cloudflare Worker serves the updated robots.txt on the bot's next crawl — no deploy.
See who is reading your site
Sign in and check your dashboard.
Stuck? Ask an AI to walk you through it