Meta-WebIndexer
The crawler that builds the search index Meta AI consults when it answers a question.
At a glance
- Operator
- Meta
- Category
-
Search index
Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.
- User-Agent token
-
Meta-WebIndexer - robots.txt token
Meta-WebIndexer- Respects robots.txt
- Documented as compliant
- Operator docs
- developers.facebook.com
What Meta-WebIndexer does
Should you block Meta-WebIndexer?
How to block Meta-WebIndexer
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Meta-WebIndexer token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Meta-WebIndexer
User-agent: Meta-WebIndexer Disallow: /
Allow Meta-WebIndexer explicitly
User-agent: Meta-WebIndexer Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Meta-WebIndexer
MetaPilot detects Meta-WebIndexer in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Meta-WebIndexer between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Meta-ExternalAgent
Meta
Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.
Amazonbot
Amazon
Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
Claude-SearchBot
Anthropic
Anthropic's indexing crawler, which builds the index Claude's web search draws on.
OAI-SearchBot
OpenAI
The crawler that builds the search index behind ChatGPT's search results.
FAQ
What is Meta-WebIndexer?
The crawler that builds the search index Meta AI consults when it answers a question.
Meta-WebIndexer crawls the web to improve the quality of Meta AI's search results. It is the discovery layer for Meta AI across Facebook, Instagram, WhatsApp and the standalone app — a very large audience that mostly never leaves those apps to search.
Does Meta-WebIndexer respect robots.txt?
Yes — Meta documents that Meta-WebIndexer honours robots.txt. A Disallow for the Meta-WebIndexer token is respected.
How do I block Meta-WebIndexer?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: Meta-WebIndexerDisallow: /
Use the Meta-WebIndexer token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Meta-WebIndexer hurt my traffic?
Meta's own guidance is that allowing Meta-WebIndexer helps Meta AI cite and link your content. Blocking it removes you from that index and from those citations, without doing anything about training — that is Meta-ExternalAgent's token.
Ask an AI whether to block Meta-WebIndexer