Does ClaudeBot respect robots.txt?

Anthropic documents that ClaudeBot honors robots.txt (and has documented crawl-delay support). Use User-agent: ClaudeBot with explicit Allow or Disallow rules.

Is ClaudeBot the same as Claude-User or Claude-SearchBot?

No. ClaudeBot is training-oriented collection; Claude-User is live user fetches; Claude-SearchBot is search indexing. Anthropic treats them as independent controls.

Does blocking ClaudeBot affect Google rankings?

No. ClaudeBot is Anthropic’s agent, not Googlebot. Google Search visibility is controlled separately.

If I block ClaudeBot, can Claude still cite me?

Search and user-fetch agents may still access your site if allowed. Citations also come from third-party sources. Blocking training crawl is not a total “remove from Claude” switch.

What is a common robots.txt pattern for Claude bots?

Some sites Disallow ClaudeBot for training opt-out while Allowing Claude-SearchBot and Claude-User for visibility and live accuracy. Choose based on policy, not defaults alone.

Where is Anthropic’s official crawler documentation?

See Anthropic’s help article on web crawling and blocking crawlers (linked as official docs on this page). Prefer that source over secondary blogs when policies change.

ClaudeBot

Anthropic · Collects web content that may contribute to Claude model training and safety

What is ClaudeBot?

ClaudeBot is Anthropic’s crawler for collecting public web content that may contribute to model training and related improvements for Claude. Anthropic documents multiple user-agents: ClaudeBot (training-oriented collection), Claude-User (user-initiated fetches), and Claude-SearchBot (search indexing). Each token is controlled independently in robots.txt; blocking ClaudeBot alone does not block the others. Anthropic states that its crawlers honor robots.txt (and has documented crawl-delay support). Operators who want granular policy—e.g., allow search visibility but block training crawl—should set separate groups for each Claude agent.

How to block ClaudeBot

Add this group to robots.txt to disallow ClaudeBot. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: ClaudeBot
Disallow: /

How to allow ClaudeBot

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: ClaudeBot
Allow: /

Should you block ClaudeBot?

If your goal is AI visibility inside Claude-related products, allowing training and search agents is the more open stance; if your goal is content control for model training, block ClaudeBot specifically and review Claude-SearchBot and Claude-User separately. Anthropic’s updated docs emphasize independent control per bot—use that granularity rather than a single blanket assumption. As with other training crawlers, robots.txt is a preference signal, not encryption. Document your policy so legal, SEO, and engineering teams share the same intent.

Category: Training

Official documentation (Anthropic)

Last verified: 2026-07-18. Re-check operator docs after major crawler announcements.

ClaudeBot FAQ

Anthropic documents that ClaudeBot honors robots.txt (and has documented crawl-delay support). Use User-agent: ClaudeBot with explicit Allow or Disallow rules.