AI crawler directory

Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.

Anthropic's indexing crawler, which builds the index Claude's web search draws on.

The fetcher Claude uses to read a specific page while answering a user who asked about it.

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Apple's actual crawler — the one that does the fetching for Siri, Spotlight and Safari suggestions.

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.

Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph.

DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results.

The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.

The crawler Google Cloud customers use to pull a website into a Vertex AI agent as grounding data.

Google's AI opt-out token: it controls whether content Google already crawls may be used to train and ground Gemini.

The fetcher behind your own Search Console testing tools — URL Inspection and the Rich Results Test.

Google's main search crawler — and, indirectly, the crawler behind AI Overviews, because those are generated from the index it builds.

Googlebot's image-fetching counterpart, which populates Google Images.

A robots.txt token for Google News, not a separate crawler — the fetching is done by Googlebot.

Googlebot's video-fetching counterpart, which populates Google's video results.

A shared token Google product teams use for one-off crawls and internal research, kept separate from Search.

Meta's bulk crawler, collecting public web content to train its foundation models and index content for Meta products.

The crawler that builds the search index Meta AI consults when it answers a question.

The fetcher Mistral's assistant uses to open a page when a user's question needs it.

A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology.

The fetcher ChatGPT uses to open a specific page in real time while answering someone.

OpenAI's bulk web crawler, collecting public pages that may be used to train future OpenAI models.

The crawler that builds the search index behind ChatGPT's search results.

Perplexity's live fetcher, which opens a page in the moment a user's question requires it.

Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers.

Webz.io's crawler (historically "Omgilibot"), collecting public web and forum content for the datasets it sells.