# ICC-Crawler **Category:** Training | **Operator:** NICT A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology. --- ## At a glance | Property | Value | |---|---| | Operator | NICT | | Category | Training | | User-Agent token | `ICC-Crawler` | | robots.txt token | `ICC-Crawler` | | Respects robots.txt | Documented as compliant | | Operator docs | ucri.nict.go.jp | > **Training crawlers** collect content used to train models. Blocking one keeps your work out of the training set, but does not affect whether an AI assistant can cite you live. --- ## What ICC-Crawler does ICC-Crawler collects public web pages for NICT's research and development in multilingual translation, information analysis and artificial intelligence. NICT states collected data may also be shared with third parties for joint research and development. This is publicly funded research rather than a commercial product, and NICT documents that it monitors its own connection rate to avoid loading the hosts it visits. --- ## Should you block ICC-Crawler? Blocking removes you from a Japanese public-research corpus. There is no consumer-facing product on the other side, so nothing about your search or AI visibility changes. NICT documents that it honours `robots.txt` and `Crawl-delay`, and publishes a contact address for cases where the directives do not appear to take effect. --- ## How to block ICC-Crawler Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `ICC-Crawler` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block ICC-Crawler:** ``` User-agent: ICC-Crawler Disallow: / ``` **Allow ICC-Crawler explicitly:** ``` User-agent: ICC-Crawler Allow: / ``` > **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles ICC-Crawler MetaPilot detects ICC-Crawler in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip ICC-Crawler between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## FAQ **What is ICC-Crawler?** A research crawler operated by NICT, Japan's National Institute of Information and Communications Technology. It collects public web pages for NICT's research and development in multilingual translation, information analysis and artificial intelligence. Collected data may also be shared with third parties for joint research and development. **Does ICC-Crawler respect robots.txt?** Yes — NICT documents that ICC-Crawler honours `robots.txt`. A `Disallow` for the `ICC-Crawler` token is respected. **How do I block ICC-Crawler?** Add these two lines to `robots.txt` at the root of your domain: ``` User-agent: ICC-Crawler Disallow: / ``` Use the `ICC-Crawler` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Will blocking ICC-Crawler hurt my traffic?** Blocking removes you from a Japanese public-research corpus. There is no consumer-facing product on the other side, so nothing about your search or AI visibility changes. --- ## Related crawlers - **ClaudeBot** (Anthropic) — Training: Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. - **Applebot-Extended** (Apple) — Training: Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. - **Bytespider** (ByteDance) — Training: ByteDance's web crawler, gathering content to train the large language models behind its products. - **CCBot / Common Crawl** (Common Crawl) — Training: The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on.