What is Bytespider? (ByteDance's AI crawler)

At a glance

Operator: ByteDance
Category: Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
User-Agent token: Bytespider
robots.txt token: Bytespider
Respects robots.txt: Not reliably
Disputed, and undocumented. Site operators and researchers have repeatedly reported Bytespider requesting paths that robots.txt disallows, and ByteDance publishes nothing stating its policy either way. Treat a robots.txt entry here as a request that may be ignored.

What Bytespider does

Bytespider crawls broadly and at high volume, collecting page content that feeds ByteDance's model development. ByteDance publishes no crawler documentation, no verification method and no stated request-rate policy, so there is very little to go on beyond what turns up in your own logs. It is frequently among the heaviest bots on a site by request count.

Should you block Bytespider?

This is the least controversial block on the list: a training crawler with no answer surface that links back to you, often generating real bandwidth cost. The catch is that robots.txt alone may not stop it — if the traffic matters, add a firewall or CDN rule rather than trusting the file.

How to block Bytespider

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the Bytespider token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block Bytespider

robots.txt

User-agent: Bytespider
Disallow: /

Allow Bytespider explicitly

robots.txt

User-agent: Bytespider
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles Bytespider

MetaPilot detects Bytespider in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Bytespider between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

ClaudeBot

Anthropic

Training

Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.

Applebot-Extended

Apple

Training

Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.

CCBot (Common Crawl)

Common Crawl

Training

The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.

Diffbot

Training

Diffbot's crawler, which reads pages and extracts structured facts into a commercial Knowledge Graph.

Browse every AI crawler

FAQ

What is Bytespider?

ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).

Bytespider crawls broadly and at high volume, collecting page content that feeds ByteDance's model development. ByteDance publishes no crawler documentation, no verification method and no stated request-rate policy, so there is very little to go on beyond what turns up in your own logs. It is frequently among the heaviest bots on a site by request count.

Does Bytespider respect robots.txt?

Not reliably. A Disallow for Bytespider is worth adding, but it should not be treated as a guarantee.

Disputed, and undocumented. Site operators and researchers have repeatedly reported Bytespider requesting paths that robots.txt disallows, and ByteDance publishes nothing stating its policy either way. Treat a robots.txt entry here as a request that may be ignored.

How do I block Bytespider?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: Bytespider
Disallow: /

Use the Bytespider token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking Bytespider hurt my traffic?

This is the least controversial block on the list: a training crawler with no answer surface that links back to you, often generating real bandwidth cost. The catch is that robots.txt alone may not stop it — if the traffic matters, add a firewall or CDN rule rather than trusting the file.