Meta-ExternalAgent
Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.
At a glance
- Operator
- Meta
- Category
-
Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
- User-Agent token
-
meta-externalagentMetaPilot also counts these user agents as Meta-ExternalAgent:
meta-externalagent,FacebookBot - robots.txt token
meta-externalagent- Respects robots.txt
- Documented as compliant
- Operator docs
- developers.facebook.com
What Meta-ExternalAgent does
Should you block Meta-ExternalAgent?
How to block Meta-ExternalAgent
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the meta-externalagent token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Meta-ExternalAgent
User-agent: meta-externalagent Disallow: /
Allow Meta-ExternalAgent explicitly
User-agent: meta-externalagent Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Meta-ExternalAgent
MetaPilot detects Meta-ExternalAgent in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Meta-ExternalAgent between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Meta-WebIndexer
Meta
The crawler that builds the search index Meta AI consults when it answers a question.
ClaudeBot
Anthropic
Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.
Applebot-Extended
Apple
Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.
Bytespider
ByteDance
ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).
FAQ
What is Meta-ExternalAgent?
Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.
Meta-ExternalAgent crawls the open web for what Meta describes as training foundation AI models and improving products by indexing content directly. It is the successor to FacebookBot, which MetaPilot counts under the same entry. Meta caches robots.txt for up to 24 hours, so a change here is not instant.
Does Meta-ExternalAgent respect robots.txt?
Yes — Meta documents that Meta-ExternalAgent honours robots.txt. A Disallow for the meta-externalagent token is respected.
How do I block Meta-ExternalAgent?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: meta-externalagentDisallow: /
Use the meta-externalagent token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Meta-ExternalAgent hurt my traffic?
Blocking it keeps your content out of Meta's model training. It does not remove you from Meta AI's answers — that index is built by Meta-WebIndexer, a separate token — and it does not affect link previews on Facebook, Instagram or WhatsApp, which use different agents again.
Ask an AI whether to block Meta-ExternalAgent