# CCBot (Common Crawl) **Category:** Training | **Operated by:** Common Crawl The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on. --- ## At a glance | Property | Value | |---|---| | Operator | Common Crawl | | Category | Training | | User-Agent token | `CCBot` | | robots.txt token | `CCBot` | | Respects robots.txt | Documented as compliant | | Operator docs | commoncrawl.org | **Training category note:** Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether an AI assistant can cite you live. --- ## What CCBot (Common Crawl) does CCBot builds Common Crawl's open corpus: a periodic snapshot of the public web, published as free downloadable datasets by a non-profit. Common Crawl is not an AI company and sells nothing, but its archive is one of the most widely used training sources in the industry. A page CCBot collects can therefore end up inside many different models run by many different companies. --- ## Should you block CCBot (Common Crawl)? Blocking CCBot is the broadest single training opt-out available, because it cuts off one archive that feeds many downstream trainers at once. Two honest caveats: - It is **not retroactive** — crawls already published stay published. - It also removes you from a research and archival resource used well beyond AI. - It has **no effect** on any live answer engine. --- ## How to block CCBot (Common Crawl) Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `CCBot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block CCBot (Common Crawl):** ``` User-agent: CCBot Disallow: / ``` **Allow CCBot (Common Crawl) explicitly:** ``` User-agent: CCBot Allow: / ``` > **A note on what this does and doesn't do.** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles CCBot (Common Crawl) MetaPilot detects CCBot (Common Crawl) in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip CCBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related crawlers - **ClaudeBot** (Anthropic) — Training: Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. - **Applebot-Extended** (Apple) — Training: Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. - **Bytespider** (ByteDance) — Training: ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok). - **Diffbot** (Diffbot) — Training: Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph. --- ## FAQ ### What is CCBot (Common Crawl)? The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on. CCBot builds Common Crawl's open corpus: a periodic snapshot of the public web, published as free downloadable datasets by a non-profit. Common Crawl is not an AI company and sells nothing, but its archive is one of the most widely used training sources in the industry. A page CCBot collects can therefore end up inside many different models run by many different companies. ### Does CCBot (Common Crawl) respect robots.txt? Yes — Common Crawl documents that CCBot honours `robots.txt`. A `Disallow` for the `CCBot` token is respected. ### How do I block CCBot (Common Crawl)? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: CCBot Disallow: / ``` Use the `CCBot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking CCBot (Common Crawl) hurt my traffic? Blocking CCBot is the broadest single training opt-out available, because it cuts off one archive that feeds many downstream trainers at once. Two honest caveats: it is not retroactive — crawls already published stay published — and it also removes you from a research and archival resource used well beyond AI. It has no effect on any live answer engine.