TopGEOClaim Wiki
PerplexityBot and Perplexity-User explained
PerplexityBot and Perplexity-User are two different agents with two different jobs, and the distinction matters for robots.txt decisions. Perplexity documents PerplexityBot as the agent "designed to surface and link websites in search results on Perplexity", not for crawling content to train AI foundation models — and recommends allowing it in robots.txt, plus allowing its published IP ranges, if you want your site to appear in Perplexity search results (Perplexity docs). Perplexity-User is the fetcher behind user actions: when someone asks Perplexity a question, it may visit a web page to answer and include a link — and "since a user requested the fetch, this fetcher generally ignores robots.txt rules" (Perplexity docs).
The two lanes
In other words: the automatic lane (PerplexityBot) decides whether your pages can be discovered and linked in Perplexity's search results, and it respects robots.txt. The user-triggered lane (Perplexity-User) runs only when a person's question prompts a page fetch, and robots.txt is, per the documentation, generally ignored for those fetches. Neither agent is documented as a trainer of foundation models. If you use a web application firewall, Perplexity also publishes WAF configuration guidance so its bots are not blocked by challenge pages (Perplexity docs).
Robots.txt basics under the protocol
The underlying mechanism is the Robots Exclusion Protocol, formalised in RFC 9309 (an IETF standards-track document, published 2022, extending the method Martijn Koster defined in 1994). A robots.txt file groups rules by user agent — User-agent followed by Disallow or Allow paths — and crawlers are requested to honour those rules; the RFC states plainly that "these rules are not a form of access authorization" (RFC 9309). Two practical consequences follow: a robots.txt allow is not a contract with any vendor, and a robots.txt block stops well-behaved crawlers but is not a security control.
Checking your logs, and what logs cannot prove
Server or CDN logs can show requests from each agent — the documented user-agent strings are PerplexityBot/1.0 and Perplexity-User/1.0 (with the note that version numbers may change) (Perplexity docs). A log entry proves a fetch happened; it does not prove that an answer used your content, cited you, or will do so again. Keep that distinction in any measurement: crawl presence is a weak, indirect signal, and treating it as an outcome invites bad decisions.
What is not documented
Perplexity's page does not document how sources are chosen, ranked or quoted in answers, how quickly robots.txt changes take effect, or whether these behaviours apply to any other product. Treat all of that as unknown. Vendor pages also change; the statements above reflect the documentation as read on 9 November 2026.
Deciding what to allow is a business decision — visibility in an answer engine versus control of automated crawling — and it is genuinely yours, not ours. If you want an independent read of which agents reach your site and what your current robots.txt actually permits, TopGEOClaim runs a scoped diagnosis, applies fixes you approve, and retests to record the result, with no promises about mentions or rankings: GEO & SEO services · contact.
Frequently asked questions
Does robots.txt stop Perplexity from fetching my page?
It stops PerplexityBot, the search-results crawler, if you disallow it. For Perplexity-User — fetches triggered by a user's question — Perplexity's documentation says the fetcher "generally ignores robots.txt rules" because a user requested the fetch (Perplexity docs).
Is PerplexityBot used to train AI models?
Perplexity states that it is not: the agent surfaces and links websites in search results and "is not used to crawl content for AI foundation models" (Perplexity docs). As with any vendor statement, this reflects the documentation as written, not an independent audit.
Can I verify Perplexity agents visited my site?
Look for the documented user-agent strings in server or CDN logs, and cross-check against Perplexity's published IP ranges. A log entry proves a fetch, not an appearance in an answer.
Is robots.txt an access-control mechanism?
No. RFC 9309, which specifies the protocol, states the rules "are not a form of access authorization" (RFC 9309). It is a courtesy protocol that compliant crawlers honour.
Sources
- docs.perplexity.ai/docs/resources/perplexity-crawlers (read 9 November 2026)
- rfc-editor.org/rfc/rfc9309 (read 9 November 2026)
Related
GEO & SEO optimisation service · Make an enquiry · More GEO and AI search answers
Owned editorial content by TopGEOClaim (ACN International Pte. Ltd., Singapore). Drafted with AI assistance, reviewed, and checked against the linked sources. It describes public documentation, not customer results, and does not promise any search or AI placement. Vendor behaviour may change; check the linked sources.