# Meta-WebIndexer

**Category:** Search index | **Operated by:** Meta

The crawler that builds the search index Meta AI consults when it answers a question.

---

## At a glance

| Property | Value |
|---|---|
| Operator | Meta |
| Category | Search index |
| User-Agent token | `Meta-WebIndexer` |
| robots.txt token | `Meta-WebIndexer` |
| Respects robots.txt | Documented as compliant |
| Operator docs | developers.facebook.com |

> **Search index:** Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.

---

## What Meta-WebIndexer does

Meta-WebIndexer crawls the web to improve the quality of Meta AI's search results. It is the discovery layer for Meta AI across Facebook, Instagram, WhatsApp and the standalone app — a very large audience that mostly never leaves those apps to search.

---

## Should you block Meta-WebIndexer?

Meta's own guidance is that allowing Meta-WebIndexer helps Meta AI cite and link your content. Blocking it removes you from that index and from those citations, without doing anything about training — that is Meta-ExternalAgent's token.

---

## How to block Meta-WebIndexer

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Meta-WebIndexer` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block Meta-WebIndexer:**

```
User-agent: Meta-WebIndexer
Disallow: /
```

**Allow Meta-WebIndexer explicitly:**

```
User-agent: Meta-WebIndexer
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles Meta-WebIndexer

MetaPilot detects Meta-WebIndexer in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Meta-WebIndexer between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related crawlers

- **Meta-ExternalAgent** (Meta) — Training: Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.
- **Amazonbot** (Amazon) — Search index: Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
- **Claude-SearchBot** (Anthropic) — Search index: Anthropic's indexing crawler, which builds the index Claude's web search draws on.
- **OAI-SearchBot** (OpenAI) — Search index: The crawler that builds the search index behind ChatGPT's search results.

---

## FAQ

### What is Meta-WebIndexer?
The crawler that builds the search index Meta AI consults when it answers a question. Meta-WebIndexer crawls the web to improve the quality of Meta AI's search results. It is the discovery layer for Meta AI across Facebook, Instagram, WhatsApp and the standalone app — a very large audience that mostly never leaves those apps to search.

### Does Meta-WebIndexer respect robots.txt?
Yes — Meta documents that Meta-WebIndexer honours `robots.txt`. A `Disallow` for the `Meta-WebIndexer` token is respected.

### How do I block Meta-WebIndexer?
Add these two lines to the `robots.txt` file at the root of your domain:

```
User-agent: Meta-WebIndexer
Disallow: /
```

Use the `Meta-WebIndexer` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

### Will blocking Meta-WebIndexer hurt my traffic?
Meta's own guidance is that allowing Meta-WebIndexer helps Meta AI cite and link your content. Blocking it removes you from that index and from those citations, without doing anything about training — that is Meta-ExternalAgent's token.