# Meta-ExternalAgent

**Operated by:** Meta | **Category:** Training

Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.

---

## At a glance

| Property | Value |
|---|---|
| Operator | Meta |
| Category | Training |
| User-Agent token | `meta-externalagent` (also counts `FacebookBot`) |
| robots.txt token | `meta-externalagent` |
| Respects robots.txt | Documented as compliant |
| Operator docs | developers.facebook.com |

---

## What Meta-ExternalAgent does

Meta-ExternalAgent crawls the open web for what Meta describes as training foundation AI models and improving products by indexing content directly. It is the successor to FacebookBot, which MetaPilot counts under the same entry. Meta caches robots.txt for up to 24 hours, so a change here is not instant.

---

## Should you block Meta-ExternalAgent?

Blocking it keeps your content out of Meta's model training. It does not remove you from Meta AI's answers — that index is built by Meta-WebIndexer, a separate token — and it does not affect link previews on Facebook, Instagram or WhatsApp, which use different agents again.

---

## How to block Meta-ExternalAgent

Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `meta-externalagent` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Block Meta-ExternalAgent:**
```
User-agent: meta-externalagent
Disallow: /
```

**Allow Meta-ExternalAgent explicitly:**
```
User-agent: meta-externalagent
Allow: /
```

> **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

---

## How MetaPilot handles Meta-ExternalAgent

MetaPilot detects Meta-ExternalAgent in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Meta-ExternalAgent between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules.

---

## Related crawlers

- **Meta-WebIndexer** (Meta) — The crawler that builds the search index Meta AI consults when it answers a question. *(Search index)*
- **ClaudeBot** (Anthropic) — Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. *(Training)*
- **Applebot-Extended** (Apple) — Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. *(Training)*
- **Bytespider** (ByteDance) — ByteDance's web crawler, gathering content to train the large language models behind its products. *(Training)*

---

## FAQ

**What is Meta-ExternalAgent?**
Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products. It is the successor to FacebookBot. Meta caches robots.txt for up to 24 hours, so changes are not instant.

**Does Meta-ExternalAgent respect robots.txt?**
Yes — Meta documents that Meta-ExternalAgent honours `robots.txt`. A `Disallow` for the `meta-externalagent` token is respected.

**How do I block Meta-ExternalAgent?**
Add these two lines to the `robots.txt` file at the root of your domain:
```
User-agent: meta-externalagent
Disallow: /
```
Use the `meta-externalagent` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

**Will blocking Meta-ExternalAgent hurt my traffic?**
Blocking it keeps your content out of Meta's model training. It does not remove you from Meta AI's answers (that index is built by Meta-WebIndexer, a separate token) and it does not affect link previews on Facebook, Instagram or WhatsApp, which use different agents.