ICC-Crawler
A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology.
At a glance
- Operator
- NICT
- Category
-
Training
Collects content used to train models. Blocking it keeps your work out of the training set, but does not affect whether the assistant can cite you live.
- User-Agent token
-
ICC-Crawler - robots.txt token
ICC-Crawler- Respects robots.txt
- Documented as compliant
- Operator docs
- ucri.nict.go.jp
What ICC-Crawler does
Should you block ICC-Crawler?
Crawl-delay, and publishes a contact address for cases where the directives do not appear to take effect.How to block ICC-Crawler
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the ICC-Crawler token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block ICC-Crawler
User-agent: ICC-Crawler Disallow: /
Allow ICC-Crawler explicitly
User-agent: ICC-Crawler Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles ICC-Crawler
MetaPilot detects ICC-Crawler in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip ICC-Crawler between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
ClaudeBot
Anthropic
Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude.
Applebot-Extended
Apple
Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models.
Bytespider
ByteDance
ByteDance's web crawler, gathering content to train the large language models behind its products (ByteDance also owns TikTok).
CCBot (Common Crawl)
Common Crawl
The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.
FAQ
What is ICC-Crawler?
A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology.
ICC-Crawler collects public web pages for NICT's research and development in multilingual translation, information analysis and artificial intelligence. NICT states collected data may also be shared with third parties for joint research and development. This is publicly funded research rather than a commercial product, and NICT documents that it monitors its own connection rate to avoid loading the hosts it visits.
Does ICC-Crawler respect robots.txt?
Yes — NICT documents that ICC-Crawler honours robots.txt. A Disallow for the ICC-Crawler token is respected.
How do I block ICC-Crawler?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: ICC-CrawlerDisallow: /
Use the ICC-Crawler token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking ICC-Crawler hurt my traffic?
Blocking removes you from a Japanese public-research corpus. There is no consumer-facing product on the other side, so nothing about your search or AI visibility changes. NICT documents that it honours robots.txt and Crawl-delay, and publishes a contact address for cases where the directives do not appear to take effect.
Ask an AI whether to block ICC-Crawler