# About MetaPilotBot

MetaPilotBot fetches pages on behalf of MetaPilot customers to generate schema.org JSON-LD and Open Graph metadata. If you're a site owner seeing this bot in your logs, here's everything you need to know.

## User-Agent string

```
MetaPilotBot/1.0 (+https://metapilot.io/bot)
```

## What it does

For each URL submitted by one of MetaPilot's customers (always one of their own pages), MetaPilotBot makes a single HTTP GET request to fetch the page's HTML. The cleaned content is sent to a large language model that returns structured data for that page, which the customer can publish back on their site.

- One request per page per generation. No crawling beyond the customer's submitted sitemap.
- Cached using `If-None-Match` and `If-Modified-Since`; returns `304 Not Modified` when nothing changed.
- Rate-limited to ~10 requests/second per host.
- Honors `robots.txt` for the `MetaPilotBot` user-agent.

## Block via robots.txt

To prevent MetaPilotBot from fetching pages on your site, add this to your `robots.txt`:

```
User-agent: MetaPilotBot
Disallow: /
```

Standard `User-agent: *` blocks are also honored. Note: if your own user has explicitly added your site to MetaPilot, blocking the bot will prevent them from generating metadata — they'll see "fetch failed" in their dashboard.

## Identifying requests

Every request from MetaPilotBot carries:

- `User-Agent: MetaPilotBot/1.0 (+https://metapilot.io/bot)`
- `X-MetaPilot-Org: <UUID>` — identifies which customer triggered the fetch
- `X-MetaPilot-User: <UUID>` — identifies the user that triggered the fetch (when applicable)

If a specific request from MetaPilot is causing problems, share the `X-MetaPilot-Org` header value and they can correlate it to a customer.

## Source IPs

Requests originate from MetaPilot's hosting infrastructure. MetaPilot does not publish a fixed IP list — IPs may change as they scale. Identification is via User-Agent.

## Contact

Bot abuse reports, sustained 4xx/5xx feedback, or anything else: contact@metapilot.io