# AI crawler directory > Directory › AI crawler directory Every AI crawler MetaPilot detects in your logs — 29 of them — with what each one actually does, whether it honours `robots.txt`, and the exact lines to allow or block it. They are not all the same thing, and blocking the wrong one quietly costs you citations. --- ## Three kinds of crawler - **Training** — Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live. - **Answer engine** — Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open. - **Search index** — Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely. - **Search engine** — A classic search crawler whose index also feeds that company's AI answers. Blocking it removes you from both the blue links and the AI summaries above them. --- ## Amazon (1 crawler) ### Amazonbot - **Type:** Search index - **User-agent token:** `Amazonbot` - Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products. --- ## Anthropic (3 crawlers) ### Claude-SearchBot - **Type:** Search index - **User-agent token:** `Claude-SearchBot` - Anthropic's indexing crawler, which builds the index Claude's web search draws on. ### Claude-User - **Type:** Answer engine - **User-agent token:** `Claude-User` - The fetcher Claude uses to read a specific page while answering a user who asked about it. ### ClaudeBot - **Type:** Training - **User-agent token:** `ClaudeBot` - Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. --- ## Apple (2 crawlers) ### Applebot - **Type:** Search engine - **User-agent token:** `Applebot` - Apple's actual crawler — the one that does the fetching for Siri, Spotlight and Safari suggestions. ### Applebot-Extended - **Type:** Training - **User-agent token:** `Applebot-Extended` - Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. --- ## ByteDance (1 crawler) ### Bytespider - **Type:** Training - **User-agent token:** `Bytespider` - ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok). --- ## Common Crawl (1 crawler) ### CCBot (Common Crawl) - **Type:** Training - **User-agent token:** `CCBot` - The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on. --- ## Diffbot (1 crawler) ### Diffbot - **Type:** Training - **User-agent token:** `Diffbot` - Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph. --- ## DuckDuckGo (1 crawler) ### DuckAssistBot - **Type:** Answer engine - **User-agent token:** `DuckAssistBot` - DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results. --- ## Google (9 crawlers) ### Gemini-Deep-Research - **Type:** Answer engine - **User-agent token:** `Gemini-Deep-Research` - The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user. ### Google-CloudVertexBot - **Type:** Training - **User-agent token:** `Google-CloudVertexBot` - The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data. ### Google-Extended - **Type:** Training - **User-agent token:** `Google-Extended` - Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini. ### Google-InspectionTool - **Type:** Search engine - **User-agent token:** `Google-InspectionTool` - The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test. ### Googlebot - **Type:** Search engine - **User-agent token:** `Googlebot` - Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds. ### Googlebot-Image - **Type:** Search engine - **User-agent token:** `Googlebot-Image` - Googlebot's image-fetching counterpart, which populates Google Images. ### Googlebot-News - **Type:** Search engine - **User-agent token:** `Googlebot-News` - A robots.txt token for Google News, not a separate crawler — the fetching is done by Googlebot. ### Googlebot-Video - **Type:** Search engine - **User-agent token:** `Googlebot-Video` - Googlebot's video-fetching counterpart, which populates Google's video results. ### GoogleOther - **Type:** Search engine - **User-agent token:** `GoogleOther` - A shared token Google product teams use for one-off crawls and internal research, kept separate from Search. --- ## Meta (2 crawlers) ### Meta-ExternalAgent - **Type:** Training - **User-agent token:** `meta-externalagent` - Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products. ### Meta-WebIndexer - **Type:** Search index - **User-agent token:** `Meta-WebIndexer` - The crawler that builds the search index Meta AI consults when it answers a question. --- ## Mistral AI (1 crawler) ### MistralAI-User - **Type:** Answer engine - **User-agent token:** `MistralAI-User` - The fetcher Mistral's assistant uses to open a page when a user's question needs it. --- ## NICT (1 crawler) ### ICC-Crawler - **Type:** Training - **User-agent token:** `ICC-Crawler` - A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology. --- ## OpenAI (3 crawlers) ### ChatGPT-User - **Type:** Answer engine - **User-agent token:** `ChatGPT-User` - The fetcher ChatGPT uses to open a specific page in real time while answering someone. ### GPTBot - **Type:** Training - **User-agent token:** `GPTBot` - OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models. ### OAI-SearchBot - **Type:** Search index - **User-agent token:** `OAI-SearchBot` - The crawler that builds the search index behind ChatGPT's search results. --- ## Perplexity (2 crawlers) ### Perplexity-User - **Type:** Answer engine - **User-agent token:** `Perplexity-User` - Perplexity's live fetcher, which opens a page in the moment a user's question requires it. ### PerplexityBot - **Type:** Search index - **User-agent token:** `PerplexityBot` - Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers. --- ## Webz.io (1 crawler) ### Webz.io - **Type:** Training - **User-agent token:** `omgili` - Webz.io's crawler (historically "Omgilibot"), collecting public web and forum content for the datasets it sells. --- ## See these bots in your own logs A directory tells you what each crawler is. MetaPilot tells you which of them actually read your site, how often, and which pages they took — then lets you flip allow or deny per bot, live on the next crawl. - [AI-crawler analytics](https://metapilot.io) - [Guide: monitor & control AI crawlers](https://metapilot.io)