# You optimized the pages. See which ones the engines actually read. **AI-crawler analytics** — MetaPilot MetaPilot already knows your pages, so it matches every crawler hit to the page it read. You get your own paths, ranked by how much answer-engine attention they earned, and a one-click allow or block for each bot. Requires the Cloudflare Worker channel. --- ## TL;DR A count of AI bots is a fact about the bot. Because MetaPilot already manages your pages, every crawler hit is matched to the page that was read, so the answer is a list of your own paths — and you can see which of the pages you optimized the engines keep coming back for. Each bot carries its category, and allowing or blocking one writes a managed rule into your robots.txt from the same screen. This feature needs the Cloudflare Worker: on Shopify and Wix nothing sits in front of your origin to see the hit. --- ## The Problem Counting AI bots is the easy half, and it is now close to free. A number saying GPTBot hit your site 400 times last week is a fact about GPTBot. It says nothing about you. The question you actually have is narrower and much harder to answer: *of the pages I spent the effort on, which ones are the engines coming back for?* Answering that needs the crawl and the page list in the same place. Almost nothing sits in both. --- ## How It Works 1. **The Worker sees every request** — Traffic to your site already passes through your Cloudflare Worker, so it can report bot traffic with no origin change, no tag to install and no JavaScript. 2. **We name the bot and the page** — Each hit is matched against a maintained roster of 26 AI and search crawlers, then against your own sitemap. Human traffic is dropped: this is not general analytics. 3. **You decide who gets in** — On **Robots & LLMs**, one click writes a MetaPilot-managed rule into your robots.txt, served at the edge. No redeploy, no CDN purge. --- ## What You Get - **Which of your pages got read, not just which bots showed up** — Every hit is keyed to the page it read, so the answer is a list of your own paths and who fetched each one. A bot counter tells you GPTBot came by. This tells you whether it came by the pages you actually optimized. - **Per-page crawler interest, where you manage pages** — An "AI crawls (7d)" column sits in your sitemap table, next to the auto-pilot switch. The page you just put on auto-pilot and the crawler traffic it earns are one row apart. - **One-click allow or block, per bot** — Toggle any crawler. MetaPilot rewrites a delimited managed block inside your `robots.txt` and leaves every hand-written rule you have exactly as you wrote it. - **The tradeoff, explained** — Every bot carries one of 4 labels: **model training**, **answer engine**, **AI search index** or **search engine**. Block a training bot at no cost to referral traffic. Block an answer engine and you may drop out of its cited answers. - **You cannot deindex yourself by accident** — Googlebot and Applebot are reported and never blockable. A one-click Disallow on Googlebot would remove you from Google, so MetaPilot refuses it on the server, not just in the button. - **Not human analytics** — MetaPilot records crawlers and drops everything else. Your visitors are not in this dataset. --- ## Plan Availability **This feature requires the Cloudflare Worker.** On Shopify or Wix there is nothing in the request path to see the hit, so MetaPilot says so rather than showing an empty table. Allow and block controls are on **every tier, including Free**. Consent should not be paywalled. Lookback windows by plan: - **Free** — 7 days - **Business** — up to 365 days - **Enterprise** — everything MetaPilot always records; only the window differs. --- ## Frequently Asked Questions **How is this different from counting AI bots?** A bot count is a number about the crawler. This is a number about your page. Because MetaPilot already manages your sitemap, every recorded hit is matched to the page it read, so you can ask: of the pages I optimized, which ones did the engines come back for? **Does this need Cloudflare?** Yes. Watching crawler traffic means being in front of the origin, which is what the edge Worker does and what a platform app cannot. On Shopify and Wix this feature is unavailable. **Does this track my human visitors?** No. MetaPilot records crawlers and drops everything else. It is not general web analytics and it does not see your users. **How does blocking a bot work?** One click adds a managed rule to your `robots.txt`, which the edge Worker serves. It applies on the next crawl, with no redeploy on your side. Your hand-written rules live outside the managed block and are never touched. **What are the bot categories?** - **Training** — collects pages to train foundation models; blocking keeps your content out of training at no cost to referral traffic. - **Answer engine** — fetches pages live to build cited answers; blocking can remove you from that assistant's answers. - **AI search index** — builds a searchable index that links back to you; usually worth allowing. - **Search engine** — classic crawler whose index also powers that company's AI answers; MetaPilot reports it but will not let you block it. **Which crawlers do you know about?** 26 crawlers, plus 3 training-policy tokens with no crawler of their own (shown as policy only). Full details in the AI bot directory. --- ## Keep Reading - The AI bot directory - Guide: monitor AI crawlers - What is robots.txt? - robots.txt and llms.txt, served at the edge