Amazonbot
Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
At a glance
- Operator
- Amazon
- Category
-
Search index
Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.
- User-Agent token
-
Amazonbot - robots.txt token
Amazonbot- Respects robots.txt
-
Documented as compliant
Amazon documents that it may use a copy of your robots.txt cached within the last 30 days, so a change to your rules can take considerably longer to take effect here than with most crawlers.
- Operator docs
- developer.amazon.com
What Amazonbot does
rel="nofollow" attribute and page-level robots meta tags, though not Crawl-delay. Amazon also runs a separate Amzn-SearchBot token which it states is not used for generative AI model training.Should you block Amazonbot?
How to block Amazonbot
Add the rule below to the robots.txt file at the root of your domain
(yourdomain.com/robots.txt). Use the Amazonbot token rather than the raw
User-Agent string, and repeat the rule on every subdomain you want covered.
New to the file? Read what robots.txt is first.
Block Amazonbot
User-agent: Amazonbot Disallow: /
Allow Amazonbot explicitly
User-agent: Amazonbot Allow: /
A note on what this does and doesn't do. robots.txt is a request, not a lock —
it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already
fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.
How MetaPilot handles Amazonbot
MetaPilot detects Amazonbot in your edge logs, so you see its real hits on your
real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to
week. From the same screen you flip Amazonbot between allow and deny with one click —
MetaPilot rewrites only its own managed section of your
robots.txt and serves it at the edge, so the change is live on the crawler's next visit with
no deploy and no touching your own rules.
See AI-crawler analytics or follow the guide to
monitor and control AI crawlers.
Related crawlers
Claude-SearchBot
Anthropic
Anthropic's indexing crawler, which builds the index Claude's web search draws on.
Meta-WebIndexer
Meta
The crawler that builds the search index Meta AI consults when it answers a question.
OAI-SearchBot
OpenAI
The crawler that builds the search index behind ChatGPT's search results.
PerplexityBot
Perplexity
Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers.
FAQ
What is Amazonbot?
Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.
Amazonbot crawls public pages so that Amazon services — Alexa among them — can find and use them. Amazon documents that it honours the Robots Exclusion Protocol, the rel="nofollow" attribute and page-level robots meta tags, though not Crawl-delay. Amazon also runs a separate Amzn-SearchBot token which it states is not used for generative AI model training.
Does Amazonbot respect robots.txt?
Yes — Amazon documents that Amazonbot honours robots.txt. A Disallow for the Amazonbot token is respected.
Amazon documents that it may use a copy of your robots.txt cached within the last 30 days, so a change to your rules can take considerably longer to take effect here than with most crawlers.
How do I block Amazonbot?
Add these two lines to the robots.txt file at the root of your domain:
User-agent: AmazonbotDisallow: /
Use the Amazonbot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.
Will blocking Amazonbot hurt my traffic?
Blocking Amazonbot removes your content from what Amazon's assistants can draw on. Because the same token covers both product improvement and AI training, a blanket block is a coarser instrument than it looks — you lose the Alexa surface along with the training opt-out.
Ask an AI whether to block Amazonbot