What is Perplexity-User? (Perplexity crawler)

At a glance

Operator: Perplexity
Category: Answer engine
Fetches your page in real time to answer a user who is asking right now. Blocking it removes you from those answers — this is the one to leave open.
User-Agent token: Perplexity-User
robots.txt token: Perplexity-User
Respects robots.txt: Not reliably
Perplexity's own documentation states that because a user requested the fetch, "this fetcher generally ignores robots.txt rules". That is a deliberate design position, not an accident — and it is the same position Cloudflare disputed publicly in August 2025.
Operator docs: docs.perplexity.ai

What Perplexity-User does

Perplexity-User retrieves a specific URL while a user is waiting on an answer, rather than crawling on a schedule. Perplexity treats it as an extension of the user rather than as a crawler, which is also how it justifies its robots.txt behaviour. Pages it fetches can be quoted and linked in the response.

Should you block Perplexity-User?

Blocking it means Perplexity stops opening your pages for the person who asked — though because of the robots.txt position below, a Disallow may not actually stop it. If you want a reliable block you need a firewall or CDN rule, and you should be sure you want to be absent from those answers.

Our take

Leave it open. Answer-engine fetchers only run because a person just asked a question, and the page they read is the page that gets cited. Being the source is the entire goal — blocking here buys you nothing and costs you the mention.

How to block Perplexity-User

Add the rule below to the robots.txt file at the root of your domain (yourdomain.com/robots.txt). Use the Perplexity-User token rather than the raw User-Agent string, and repeat the rule on every subdomain you want covered. New to the file? Read what robots.txt is first.

Block Perplexity-User

robots.txt

User-agent: Perplexity-User
Disallow: /

Allow Perplexity-User explicitly

robots.txt

User-agent: Perplexity-User
Allow: /

A note on what this does and doesn't do. robots.txt is a request, not a lock — it works because well-behaved crawlers choose to read it. It also isn't retroactive: anything already fetched stays fetched. If you need a hard stop, block at your firewall or CDN as well.

How MetaPilot handles Perplexity-User

MetaPilot detects Perplexity-User in your edge logs, so you see its real hits on your real pages rather than a guess: how often it visits, which URLs it takes, and how that trend moves week to week. From the same screen you flip Perplexity-User between allow and deny with one click — MetaPilot rewrites only its own managed section of your robots.txt and serves it at the edge, so the change is live on the crawler's next visit with no deploy and no touching your own rules. See AI-crawler analytics or follow the guide to monitor and control AI crawlers.

Related crawlers

PerplexityBot

Perplexity

Search index

Perplexity's indexing crawler — the one that decides whether your pages can be surfaced as a source in Perplexity answers.

Claude-User

Anthropic

Answer engine

The fetcher Claude uses to read a specific page while answering a user who asked about it.

DuckAssistBot

DuckDuckGo

Answer engine

DuckDuckGo's real-time fetcher for DuckAssist, the AI-assisted answers that appear above its search results.

Gemini-Deep-Research

Google

Answer engine

The agent behind Gemini's Deep Research mode, which reads pages as it works through a multi-step research task for one user.

Browse every AI crawler

FAQ

What is Perplexity-User?

Perplexity's live fetcher, which opens a page in the moment a user's question requires it.

Perplexity-User retrieves a specific URL while a user is waiting on an answer, rather than crawling on a schedule. Perplexity treats it as an extension of the user rather than as a crawler, which is also how it justifies its robots.txt behaviour. Pages it fetches can be quoted and linked in the response.

Does Perplexity-User respect robots.txt?

Not reliably. A Disallow for Perplexity-User is worth adding, but it should not be treated as a guarantee.

Perplexity's own documentation states that because a user requested the fetch, "this fetcher generally ignores robots.txt rules". That is a deliberate design position, not an accident — and it is the same position Cloudflare disputed publicly in August 2025.

How do I block Perplexity-User?

Add these two lines to the robots.txt file at the root of your domain:

User-agent: Perplexity-User
Disallow: /

Use the Perplexity-User token, not the raw User-Agent string, and repeat the rule on every subdomain you want covered.

Will blocking Perplexity-User hurt my traffic?

Blocking it means Perplexity stops opening your pages for the person who asked — though because of the robots.txt position below, a Disallow may not actually stop it. If you want a reliable block you need a firewall or CDN rule, and you should be sure you want to be absent from those answers.