# Applebot-Extended **Category:** Training | **Operated by:** Apple Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. --- ## At a Glance | Property | Value | |---|---| | Operator | Apple | | Category | Training | | User-Agent token | None — this is a robots.txt control token, not a crawler | | robots.txt token | `Applebot-Extended` | | Respects robots.txt | Yes — documented as compliant | **User-Agent note:** Apple fetches with its own crawler (Applebot) and reads this token only to decide what it may then do with the result. No request ever carries the name `Applebot-Extended`, so it will never appear in your access logs no matter how often your content is used. --- ## What Applebot-Extended Does Applebot-Extended does not crawl anything. Applebot does the fetching, for Siri, Spotlight and Safari suggestions; the extended token is the separate switch controlling whether that same content may also train Apple's generative models. Apple designed it as a directive rather than as a second crawler precisely so publishers would not have to choose between search and training. --- ## Should You Block Applebot-Extended? Disallowing Applebot-Extended keeps your content out of Apple's model training while leaving Applebot free to index you for Siri and Spotlight. That makes it a cheap opt-out. Blocking Applebot itself is the much heavier move, and that is the one that removes you from Apple's search surfaces. > **Our take:** This is the low-risk opt-out. Because Applebot-Extended governs what may be done with content Apple has already fetched — rather than whether it may fetch at all — disallowing it keeps you in search while keeping you out of model training. That's the trade the token exists to let you make. --- ## How to Block Applebot-Extended Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Applebot-Extended` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block Applebot-Extended:** ``` User-agent: Applebot-Extended Disallow: / ``` **Allow Applebot-Extended explicitly:** ``` User-agent: Applebot-Extended Allow: / ``` > **Note:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot Handles Applebot-Extended MetaPilot detects Applebot-Extended in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Applebot-Extended between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related Crawlers - **Applebot** (Apple) — Search engine: Apple's actual crawler — the one that does the fetching for Siri, Spotlight and Safari suggestions. - **ClaudeBot** (Anthropic) — Training: Anthropic's bulk crawler, collecting public web content that may be used to develop and train Claude. - **Bytespider** (ByteDance) — Training: ByteDance's web crawler, gathering content to train the large language models behind its products. - **CCBot / Common Crawl** (Common Crawl) — Training: The crawler behind Common Crawl, the free public web archive that a great many AI models are trained on. --- ## FAQ ### What is Applebot-Extended? Apple's AI opt-out token, telling Apple not to use content Applebot already crawled to train its foundation models. Applebot-Extended does not crawl anything; Applebot does the fetching for Siri, Spotlight and Safari suggestions, while the extended token is the separate switch controlling whether that same content may also train Apple's generative models. ### Does Applebot-Extended respect robots.txt? Yes — Apple documents that Applebot-Extended honours `robots.txt`. A `Disallow` for the `Applebot-Extended` token is respected. Because it is a robots.txt directive rather than a separate crawler, it has no user-agent of its own and will not show up as a distinct hit in your logs. ### How do I block Applebot-Extended? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: Applebot-Extended Disallow: / ``` Use the `Applebot-Extended` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking Applebot-Extended hurt my traffic? No. Disallowing Applebot-Extended keeps your content out of Apple's model training while leaving Applebot free to index you for Siri and Spotlight. Blocking Applebot itself is the much heavier move that removes you from Apple's search surfaces.