What is PerplexityBot? (Perplexity's AI crawler)

At a glance

Operator: Perplexity
Category: Search index
Builds the search index the assistant queries before answering. Blocking it takes you out of that index entirely.
User-Agent token: PerplexityBot
robots.txt token: PerplexityBot
Respects robots.txt: Documented as compliant
Disputed. Perplexity documents that PerplexityBot follows robots.txt, but in August 2025 Cloudflare reported that Perplexity traffic continued to reach pages after site owners had blocked both its declared user agents, apparently via undeclared browser-like agents, and removed Perplexity from its Verified Bots programme. Perplexity rejects the characterisation, arguing a user-driven agent is not a crawler. The disagreement is not settled.
Operator docs: docs.perplexity.ai

What PerplexityBot does

PerplexityBot crawls and indexes pages so that Perplexity can find and link them when it answers. Perplexity presents itself as an answer engine that cites its sources on every response, so pages in this index are eligible to appear as a numbered citation. Perplexity explicitly recommends allowing this token if you want to appear in its results.

Should you block PerplexityBot?

Blocking PerplexityBot removes you from the index Perplexity searches, and therefore from its citation list. Perplexity is one of the more generous engines about linking out, so this is usually a straightforward loss of referral traffic with no offsetting benefit.

How to block PerplexityBot

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the PerplexityBot token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block PerplexityBot

robots.txt

User-agent: PerplexityBot
Disallow: /

Allow PerplexityBot explicitly

robots.txt

User-agent: PerplexityBot
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles PerplexityBot

MetaPilot detects PerplexityBot in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip PerplexityBot between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

Perplexity-User

Perplexity

Answer engine

Perplexity's live fetcher, which opens a page in the moment a user's question requires it.

Amazonbot

Amazon

Search index

Amazon's crawler, used to improve Alexa's answers and Amazon's own search and AI products.

Claude-SearchBot

Anthropic

Search index

Anthropic's indexing crawler, which builds the index Claude's web search draws on.

Meta-WebIndexer

FAQ

What is PerplexityBot?

Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers.

PerplexityBot crawls and indexes pages so that Perplexity can find and link them when it answers. Perplexity presents itself as an answer engine that cites its sources on every response, so pages in this index are eligible to appear as a numbered citation. Perplexity explicitly recommends allowing this token if you want to appear in its results.

Does PerplexityBot respect robots.txt?

Yes — Perplexity documents that PerplexityBot honours robots.txt. A Disallow for the PerplexityBot token is respected.

Disputed. Perplexity documents that PerplexityBot follows robots.txt, but in August 2025 Cloudflare reported that Perplexity traffic continued to reach pages after site owners had blocked both its declared user agents, apparently via undeclared browser-like agents, and removed Perplexity from its Verified Bots programme. Perplexity rejects the characterisation, arguing a user-driven agent is not a crawler. The disagreement is not settled.

How do I block PerplexityBot?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: PerplexityBot
Disallow: /

Use the PerplexityBot token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking PerplexityBot hurt my traffic?

Blocking PerplexityBot removes you from the index Perplexity searches, and therefore from its citation list. Perplexity is one of the more generous engines about linking out, so this is usually a straightforward loss of referral traffic with no offsetting benefit.