ClaudeBot
Definition
ClaudeBot is Anthropic’s crawler for collecting public web content that may support Claude model training and related improvements.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
Anthropic documents separate agents such as ClaudeBot (training-oriented collection), Claude-User (user-initiated fetches), and Claude-SearchBot (search indexing). Each can be allowed or blocked independently in robots.txt.
A common granular pattern is to allow search and user-fetch agents while restricting training crawl—only when that split matches legal and product goals. As with other crawlers, robots.txt is a preference signal, not a security control.
Verify current behavior against Anthropic’s official crawler documentation before publishing policy. See AI Crawler and GPTBot for multi-agent policy comparison.