What is Diffbot? How to allow or block it

At a glance

Operator: Diffbot
Category: Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
User-Agent token: Diffbot
robots.txt token: Diffbot
Respects robots.txt: Documented as compliant
Diffbot honours Disallow by default, but its documentation notes the setting can be overridden in specific cases — typically where its customer has a partnership or agreement with the site being crawled.
Operator docs: www.diffbot.com

What Diffbot does

Diffbot turns pages into entities and relationships — companies, people, products, articles — and sells access to the resulting graph. Diffbot states this general crawling is not used for AI model training; a separate Diffbot-User token covers fetches made on behalf of a human user of its products. Its customers include sales, research and AI teams that want structured data rather than raw text.

Should you block Diffbot?

Blocking keeps your data out of a commercial dataset that gets resold. Nothing a visitor can see changes, because there is no consumer answer engine on the other side. Plenty of businesses take the opposite view and want to be present and accurately represented in that graph.

How to block Diffbot

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the Diffbot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block Diffbot

robots.txt

User-agent: Diffbot
Disallow: /

Allow Diffbot explicitly

robots.txt

User-agent: Diffbot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles Diffbot

MetaPilot detects Diffbot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Diffbot between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Applebot-Extended

Apple

Training

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

Bytespider

ByteDance

Training

ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

CCBot (Common Crawl)

Common Crawl

Training

The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.

Browse every AI crawler

FAQ

What is Diffbot?

Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph.

Diffbot turns pages into entities and relationships — companies, people, products, articles — and sells access to the resulting graph. Diffbot states this general crawling is not used for AI model training; a separate Diffbot-User token covers fetches made on behalf of a human user of its products. Its customers include sales, research and AI teams that want structured data rather than raw text.

Does Diffbot respect robots.txt?

Yes — Diffbot documents that Diffbot honours robots.txt. A Disallow for the Diffbot token is respected.

Diffbot honours Disallow by default, but its documentation notes the setting can be overridden in specific cases — typically where its customer has a partnership or agreement with the site being crawled.

How do I block Diffbot?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: Diffbot
Disallow: /

Use the Diffbot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking Diffbot hurt my traffic?

Blocking keeps your data out of a commercial dataset that gets resold. Nothing a visitor can see changes, because there is no consumer answer engine on the other side. Plenty of businesses take the opposite view and want to be present and accurately represented in that graph.