# Amazonbot **Category:** Search index | **Operated by:** Amazon Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products. --- ## At a glance | Property | Value | |---|---| | Operator | Amazon | | Category | Search index | | User-Agent token | `Amazonbot` | | robots.txt token | `Amazonbot` | | Respects robots.txt | Documented as compliant | | Operator docs | developer.amazon.com | > **Note on robots.txt caching:** Amazon documents that it may use a copy of your robots.txt cached within the last 30 days, so a change to your rules can take considerably longer to take effect here than with most crawlers. --- ## What Amazonbot does Amazonbot crawls public pages so that Amazon services — Alexa among them — can find and use them. Amazon documents that it honours the Robots Exclusion Protocol, the `rel="nofollow"` attribute and page-level robots meta tags, though not `Crawl-delay`. Amazon also runs a separate `Amzn-SearchBot` token which it states is not used for generative AI model training. --- ## Should you block Amazonbot? Blocking Amazonbot removes your content from what Amazon's assistants can draw on. Because the same token covers both product improvement and AI training, a blanket block is a coarser instrument than it looks — you lose the Alexa surface along with the training opt-out. --- ## How to block Amazonbot Add the rule below to the `robots.txt` file at the root of your domain (`yourdomain.com/robots.txt`). Use the `Amazonbot` token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. **Block Amazonbot:** ``` User-agent: Amazonbot Disallow: / ``` **Allow Amazonbot explicitly:** ``` User-agent: Amazonbot Allow: / ``` > **Important:** `robots.txt` is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well. --- ## How MetaPilot handles Amazonbot MetaPilot detects Amazonbot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Amazonbot between allow and deny with one click — MetaPilot rewrites only its own managed section of your `robots.txt` and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. --- ## Related crawlers - **Claude-SearchBot** (Anthropic) — Anthropic's indexing crawler, which builds the index Claude's web search draws on. - **Meta-WebIndexer** (Meta) — The crawler that builds the search index Meta AI consults when it answers a question. - **OAI-SearchBot** (OpenAI) — The crawler that builds the search index behind ChatGPT's search results. - **PerplexityBot** (Perplexity) — Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers. --- ## FAQ ### What is Amazonbot? Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products. Amazonbot crawls public pages so that Amazon services — Alexa among them — can find and use them. Amazon documents that it honours the Robots Exclusion Protocol, the `rel="nofollow"` attribute and page-level robots meta tags, though not `Crawl-delay`. Amazon also runs a separate `Amzn-SearchBot` token which it states is not used for generative AI model training. ### Does Amazonbot respect robots.txt? Yes — Amazon documents that Amazonbot honours `robots.txt`. A `Disallow` for the `Amazonbot` token is respected. Amazon documents that it may use a copy of your robots.txt cached within the last 30 days, so a change to your rules can take considerably longer to take effect here than with most crawlers. ### How do I block Amazonbot? Add these two lines to the `robots.txt` file at the root of your domain: ``` User-agent: Amazonbot Disallow: / ``` Use the `Amazonbot` token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered. ### Will blocking Amazonbot hurt my traffic? Blocking Amazonbot removes your content from what Amazon's assistants can draw on. Because the same token covers both product improvement and AI training, a blanket block is a coarser instrument than it looks — you lose the Alexa surface along with the training opt-out.