MetaPilot · Bot identity

About MetaPilotBot

MetaPilotBot fetches pages on behalf of MetaPilot customers to generate schema.org JSON-LD and Open Graph metadata. If you're a site owner seeing this bot in your logs, here's everything you need to know.

User-Agent string

MetaPilotBot/1.0 (+https://metapilot.io/bot)

What it does

For each URL submitted by one of our customers (always one of their own pages), MetaPilotBot makes a single HTTP GET request to fetch the page's HTML. We send the cleaned content to a large language model that returns structured data for that page, which our customer can publish back on their site.

• One request per page per generation. No crawling beyond the customer's submitted sitemap.
• Cached using If-None-Match and If-Modified-Since; we'll see 304 Not Modified when nothing changed.
• Rate-limited to ~10 requests/second per host.
• Honors robots.txt for the MetaPilotBot user-agent.

Block via robots.txt

To prevent MetaPilotBot from fetching pages on your site, add this to your robots.txt:

User-agent: MetaPilotBot
Disallow: /

Standard User-agent: * blocks are also honored. Note: if your own user has explicitly added your site to MetaPilot, blocking the bot will prevent them from generating metadata. They'll see "fetch failed" in their dashboard.

Identifying requests

Every request from MetaPilotBot carries:

• User-Agent: MetaPilotBot/1.0 (+https://metapilot.io/bot)
• X-MetaPilot-Org: <UUID> — identifies which customer triggered the fetch
• X-MetaPilot-User: <UUID> — identifies the user that triggered the fetch (when applicable)

If a specific request from us is causing problems, share the X-MetaPilot-Org header value and we can correlate it to a customer.

Source IPs

Requests originate from MetaPilot's hosting infrastructure. We do not publish a fixed IP list — IPs may change as we scale. Identification is via User-Agent.

Contact

Bot abuse reports, sustained 4xx/5xx feedback, or anything else: contact@metapilot.io