What is CCBot (Common Crawl)? How to block it

At a glance

Operator: Common Crawl
Category: Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
User-Agent token: CCBot
robots.txt token: CCBot
Respects robots.txt: Documented as compliant
Operator docs: commoncrawl.org

What CCBot (Common Crawl) does

CCBot builds Common Crawl's open corpus: a periodic snapshot of the public web, published as free downloadable datasets by a non-profit. Common Crawl is not an AI company and sells nothing, but its archive is one of the most widely used training sources in the industry. A page CCBot collects can therefore end up inside many different models run by many different companies.

Should you block CCBot (Common Crawl)?

Blocking CCBot is the broadest single training opt-out available, because it cuts off one archive that feeds many downstream trainers at once. Two honest caveats: it is not retroactive — crawls already published stay published — and it also removes you from a research and archival resource used well beyond AI. It has no effect on any live answer engine.

How to block CCBot (Common Crawl)

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the CCBot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block CCBot (Common Crawl)

robots.txt

User-agent: CCBot
Disallow: /

Allow CCBot (Common Crawl) explicitly

robots.txt

User-agent: CCBot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles CCBot (Common Crawl)

MetaPilot detects CCBot (Common Crawl) in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip CCBot (Common Crawl) between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Applebot-Extended

Apple

Training

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

Bytespider

ByteDance

Training

ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

Diffbot

Training

Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph.

Browse every AI crawler

FAQ

What is CCBot (Common Crawl)?

The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.

CCBot builds Common Crawl's open corpus: a periodic snapshot of the public web, published as free downloadable datasets by a non-profit. Common Crawl is not an AI company and sells nothing, but its archive is one of the most widely used training sources in the industry. A page CCBot collects can therefore end up inside many different models run by many different companies.

Does CCBot (Common Crawl) respect robots.txt?

Yes — Common Crawl documents that CCBot (Common Crawl) honours robots.txt. A Disallow for the CCBot token is respected.

How do I block CCBot (Common Crawl)?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: CCBot
Disallow: /

Use the CCBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking CCBot (Common Crawl) hurt my traffic?

Blocking CCBot is the broadest single training opt-out available, because it cuts off one archive that feeds many downstream trainers at once. Two honest caveats: it is not retroactive — crawls already published stay published — and it also removes you from a research and archival resource used well beyond AI. It has no effect on any live answer engine.