TopGEOClaim Wiki
Anthropic's crawlers and robots.txt
Anthropic currently documents three robots, each with a distinct job and a distinct robots.txt story: ClaudeBot (collects web content that may contribute to model training), Claude-User (fetches pages when an individual asks Claude a question), and Claude-SearchBot (crawls to improve search result quality). Anthropic states that its bots respect robots.txt directives, and gives a per-agent pattern for limiting each one (Anthropic help centre). So robots.txt can control all three by name — unlike some other vendors' user-triggered fetchers (see our note on OpenAI's bots at the end).
What each bot does, per the documentation
ClaudeBot— gathers public web content that could contribute to training datasets. If a site restricts it, "it signals that the site's future materials should be excluded from our AI model training datasets" (Anthropic).Claude-User— supports user-initiated actions: when a person asks Claude a question, Claude may access the site with this agent. Disabling it prevents retrieval in response to user queries, "which may reduce your site's visibility" in user-directed search (Anthropic).Claude-SearchBot— analyses online content to improve the relevance and accuracy of search responses. Disabling it prevents indexing for that purpose, again with a stated visibility trade-off (Anthropic).
How to block or limit each one
Anthropic's examples follow the Robots Exclusion Protocol (specified in RFC 9309):
User-agent: ClaudeBot
Disallow: /That blocks ClaudeBot site-wide, and Anthropic notes you must repeat the opt-out in the robots.txt of every subdomain you want excluded. It also supports the non-standard Crawl-delay extension to slow crawling rather than stop it. Anthropic cautions against relying on IP blocking: it "may not work correctly or persistently", because blocking the IPs can prevent the crawler from reading your robots.txt at all (Anthropic).
A newer signal: machine-readable no-AI-training
As of 30 October 2026, Anthropic says it recognises a machine-readable signal letting site owners express a preference that content not be used for AI training, and that where robots.txt explicitly names ClaudeBot, "we follow that rule, even if your general setting says something different" (Anthropic). That is a recent, opt-in mechanism; how other vendors treat similar signals varies and is largely undocumented, so read each vendor's own pages before relying on it.
How this compares with OpenAI's bots
OpenAI documents its own set of agents with a similar split — most notably, ChatGPT-User for user-initiated fetches, for which OpenAI says robots.txt rules "may not apply", and OAI-SearchBot for search inclusion (OpenAI bots overview). We keep OpenAI's controls in a separate existing wiki answer rather than repeating them here. The general lesson across both vendors: name each agent explicitly in robots.txt, because per-agent rules are honoured and generic rules are weaker than specific ones.
What remains unknown
Anthropic's page does not describe how sources are selected or ranked in answers, how quickly a robots.txt change takes effect, or whether these patterns extend to future agents. Blocking a fetcher also has a documented visibility trade-off for user-facing features — a genuine business trade-off, not a technical error. Vendor pages change; the above reflects Anthropic's documentation as read on 9 November 2026, not an independent test.
If you want a plain, checked read of what your current robots.txt says to each vendor's agents — with a diagnosis, a fix you approve, and a retest record — see our GEO & SEO services or write to us at contact.
Frequently asked questions
Can robots.txt stop Anthropic from using my site for training?
Anthropic states that restricting ClaudeBot signals your future materials should be excluded from training datasets, and that its bots honour robots.txt directives (Anthropic). This is a vendor statement about its own behaviour, not a legal guarantee.
Should I block Claude-User and Claude-SearchBot too?
That depends on whether you value visibility in Claude's user-directed answers and search. Anthropic documents that disabling them may reduce that visibility. Decide per agent; the two serve different purposes.
Is blocking Anthropic's IPs a safer option?
No. Anthropic explicitly cautions that IP blocking may not work correctly or persistently guarantee an opt-out, because it can stop the crawler from reading your robots.txt (Anthropic).
Does one robots.txt rule cover all my subdomains?
No — each subdomain serves its own robots.txt, and the opt-out must be repeated for each one you want excluded (Anthropic).
Sources
- support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (read 9 November 2026)
- rfc-editor.org/rfc/rfc9309 (read 9 November 2026)
- developers.openai.com/api/docs/bots (read 9 November 2026)
Related
GEO & SEO optimisation service · Make an enquiry · More GEO and AI search answers
Owned editorial content by TopGEOClaim (ACN International Pte. Ltd., Singapore). Drafted with AI assistance, reviewed, and checked against the linked sources. It describes public documentation, not customer results, and does not promise any search or AI placement. Vendor behaviour may change; check the linked sources.