You optimized the pages. See which ones the engines actually read.
MetaPilot already knows your pages, so it matches every crawler hit to the page it read. You get your own paths, ranked by how much answer-engine attention they earned, and a one-click allow or block for each bot. On the Cloudflare channel, where something is in front of your origin to see it.
YOUR PAGE CRAWLERS HITS /pricing ChatGPT-User 14 · GPTBot 9 27 /guides/schema PerplexityBot 11 · CCBot 4 19 /blog/aeo-basics Claude-User 8 8 ROBOTS POLICY GPTBot training block ChatGPT-User answer engine allow Googlebot search engine always allowed Google-Extended policy only block
TL;DR
A count of AI bots is a fact about the bot. Because MetaPilot already manages your pages, every crawler hit is matched to the page that was read, so the answer is a list of your own paths and you can see which of the pages you optimized the engines keep coming back for. Each bot carries its category, and allowing or blocking one writes a managed rule into your robots.txt from the same screen. This feature needs the Cloudflare Worker: on Shopify and Wix nothing sits in front of your origin to see the hit.
The problem
Counting AI bots is the easy half, and it is now close to free. A number saying GPTBot hit your site 400 times last week is a fact about GPTBot. It says nothing about you.
The question you actually have is narrower and much harder to answer: of the pages I spent the effort on, which ones are the engines coming back for? Answering that needs the crawl and the page list in the same place. Almost nothing sits in both.
How it works
-
The Worker sees every request
Traffic to your site already passes through your Cloudflare Worker, so it can report bot traffic with no origin change, no tag to install and no JavaScript.
-
We name the bot and the page
Each hit is matched against a maintained roster of 26 AI and search crawlers, then against your own sitemap. Human traffic is dropped: this is not general analytics.
-
You decide who gets in
On Robots & LLMs, one click writes a MetaPilot-managed rule into your robots.txt, served at the edge. No redeploy, no CDN purge.
What you get
Which of your pages got read, not just which bots showed up
Every hit is keyed to the page it read, so the answer is a list of your own paths and who fetched each one. A bot counter tells you GPTBot came by. This tells you whether it came by the pages you actually optimized.
Per-page crawler interest, where you manage pages
An “AI crawls (7d)” column sits in your sitemap table, next to the auto-pilot switch. The page you just put on auto-pilot and the crawler traffic it earns are one row apart.
One-click allow or block, per bot
Toggle any crawler. MetaPilot rewrites a delimited managed block inside your robots.txt and leaves every hand-written rule you have exactly as you wrote it.
The tradeoff, explained
Every bot carries one of 4 labels: model training, answer engine, AI search index or search engine. Block a training bot at no cost to referral traffic. Block an answer engine and you may drop out of its cited answers. The label turns a nervous edit into an informed one.
You cannot deindex yourself by accident
Googlebot and Applebot are in the roster because their indexes are what Google and Apple generate AI answers from. They are reported and never blockable: a one-click Disallow on Googlebot would remove you from Google, so MetaPilot refuses it on the server, not just in the button.
Not human analytics
MetaPilot records crawlers and drops everything else. Your visitors are not in this dataset, so there is nothing here to reconcile with your privacy policy.
Plan availability
This one needs the Cloudflare Worker. Observing a crawler means sitting in front of the origin, and a platform app does not. On Shopify or Wix there is nothing in the request path to see the hit, so MetaPilot says so rather than showing you an empty table.
Allow and block controls are on every tier, including Free. Consent should not be paywalled.
What scales is how far back you can look, and it comes with your plan rather than being sold separately: 7 days on Free, up to 365 days on Business, and everything on Enterprise. We always record; only the window differs.
Frequently asked
A bot count is a number about the crawler. This is a number about your page. Because MetaPilot already manages your sitemap, every recorded hit is matched to the page it read, so you can ask the question that actually matters: of the pages I optimized, which ones did the engines come back for? Nothing sitting outside your page list can answer that.
Yes. Watching crawler traffic means being in front of the origin, which is what the edge Worker does and what a platform app cannot. On Shopify and Wix this feature is unavailable and MetaPilot tells you so in the product rather than leaving a table at zero.
No. MetaPilot records crawlers and drops everything else. It is not general web analytics and it does not see your users.
One click adds a managed rule to your robots.txt, which the edge Worker serves. It applies on the next crawl, with no redeploy on your side. Your hand-written rules live outside the managed block and are never touched. See the robots.txt guide.
A training bot collects pages to train foundation models, so blocking it keeps your content out of training at no cost to referral traffic. An answer engine fetches pages live to build cited answers, so blocking it can remove you from that assistant’s answers. An AI search index builds a searchable index that links back to you and is usually worth allowing. A search engine is the classic crawler, and its index is also what that company’s AI answers are generated from, which is why MetaPilot reports it and will not let you block it.
26 at the moment, plus 3 training-policy tokens that have no crawler of their own and are shown as policy only rather than as a hit count of zero. Browse them all, what each one does and how to allow or block it, in the AI bot directory.
Find out which pages are earning the crawls
Connect your zone and the first hits land the same day, each one against the page it read.
Ask an AI what per-page crawler data is worth