Does Claude-SearchBot respect robots.txt?

Anthropic documents robots.txt compliance for its crawlers, including Claude-SearchBot. Use a dedicated User-agent: Claude-SearchBot group.

Is Claude-SearchBot the same as ClaudeBot?

No. Claude-SearchBot indexes content for Claude search relevance; ClaudeBot is oriented toward training data collection. Blocking one does not block the other.

Does blocking Claude-SearchBot hurt Google rankings?

No. It may reduce visibility in Claude search contexts, not classic Google organic rankings.

Should GEO-focused sites allow Claude-SearchBot?

Often yes, if Claude search visibility matters. Anthropic has noted tradeoffs when search crawlers are blocked. Balance that against any content-control requirements.

How do I allow Claude-SearchBot under a strict wildcard?

Add an explicit Claude-SearchBot group with Allow: / after or alongside your User-agent: * rules so specific agents are not left only on the catch-all Disallow.

Does Claude-SearchBot replace sitemap submission?

No. Sitemaps help discovery broadly; Claude-SearchBot is Anthropic’s own indexing agent. Keep sitemaps for search engines and use robots.txt for AI agent policy.

Claude-SearchBot

Anthropic · Indexes content to improve Claude search result relevance

What is Claude-SearchBot?

Claude-SearchBot crawls the public web to build and improve indexes used for Claude’s search results and citations. Anthropic separates this from ClaudeBot (training collection) and Claude-User (on-demand user fetches). Blocking Claude-SearchBot can reduce how your pages appear in Claude search contexts without by itself stopping training collection (ClaudeBot) or live user fetches (Claude-User). Anthropic documents robots.txt compliance for its crawlers; use an explicit User-agent: Claude-SearchBot group rather than relying only on a wildcard.

How to block Claude-SearchBot

Add this group to robots.txt to disallow Claude-SearchBot. Path rules can be narrowed if you only need to protect parts of the site.

User-agent: Claude-SearchBot
Disallow: /

How to allow Claude-SearchBot

For most public marketing and documentation sites, allowing well-behaved AI agents is the default recommended stance for AI visibility—while remaining a factual robots.txt Allow rule you can reverse later.

User-agent: Claude-SearchBot
Allow: /

Should you block Claude-SearchBot?

If you want to be findable when Claude search surfaces web results, allow Claude-SearchBot. Anthropic has noted visibility tradeoffs when search crawlers are blocked. Training policy is a separate decision via ClaudeBot. For GEO-oriented sites, search-indexing agents are often allowed even when training crawl is restricted—only if that split matches your compliance needs. Keep NAP, schema, and content quality strong so indexed pages remain useful.

Category: Search indexing

Official documentation (Anthropic)

Last verified: 2026-07-18. Re-check operator docs after major crawler announcements.