Does Claude-User respect robots.txt?

Anthropic documents that Claude-User honors robots.txt. That differs from some vendors’ user-fetch agents—still verify the official article after updates.

Is Claude-User a training crawler?

No. Claude-User supports fetching pages when Claude users need live web content. Training-oriented collection is associated with ClaudeBot.

Does blocking Claude-User change Google SEO?

No direct impact on Google rankings. It only affects Anthropic’s user-triggered fetches under that token.

Why would I allow Claude-User but block ClaudeBot?

To keep conversational page opens working while limiting training crawl. Implement two robots.txt groups so the intents stay separate.

How do I detect Claude-User in logs?

Filter access logs for the Claude-User user-agent string. Volume often correlates with real questions rather than continuous bulk crawl.

Can Claude-User bypass authentication?

robots.txt does not grant access to private URLs. If a page requires login, unauthenticated bots—including Claude-User—should not see protected content.

Claude-User

Anthropic · Fetches pages when people use Claude and need live web content

What is Claude-User?

Claude-User supports Claude AI users: when someone asks Claude a question that requires fetching a website, Anthropic may access the page with the Claude-User agent. It is a live-browsing / user-triggered fetch agent, not the training crawler ClaudeBot. Anthropic documents that Claude-User honors robots.txt. Site owners often allow Claude-User so documentation and product pages remain readable when users ask Claude to open them, while optionally restricting ClaudeBot for training preferences. Confirm current wording on Anthropic’s official crawler help article before publishing policy.

How to block Claude-User

Add this group to robots.txt to disallow Claude-User. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: Claude-User
Disallow: /

How to allow Claude-User

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: Claude-User
Allow: /

Should you block Claude-User?

Allowing Claude-User is typically low risk for public sites: volume follows real user demand, and blocking can break helpful “open this page” workflows. If you must restrict Claude entirely, block Claude-User, ClaudeBot, and Claude-SearchBot with separate groups. A common pattern is allow Claude-User and Claude-SearchBot while blocking ClaudeBot for training—only if that matches your legal and product goals. Re-test after Anthropic documentation updates.

Category: Live browsing

Official documentation (Anthropic)

Last verified: 2026-07-18. Re-check operator docs after major crawler announcements.

Claude-User FAQ

Anthropic documents that Claude-User honors robots.txt. That differs from some vendors’ user-fetch agents—still verify the official article after updates.