robots.txt
Definition
robots.txt is a text file at a site’s root that suggests which crawlers may fetch which paths, using User-agent and Allow/Disallow rules.
By Vinespire Editorial Team, Editorial ·
This term is part of the full AI search glossary.
Full definition
Well-behaved crawlers read https://example.com/robots.txt before requesting pages. Rules are voluntary preferences, not a security boundary—authentication, authorization, and network controls still protect private content.
AI companies publish distinct user-agents (GPTBot, ClaudeBot, Google-Extended, and others). A catch-all User-agent: * Disallow: / can block training, search-indexing, and live-browsing bots unintentionally; operators who want granular policy set separate groups per agent.
robots.txt cannot erase prior training data or third-party copies. Pair crawl policy with noindex, auth, and licensing where needed. See User-Agent and AI Crawler.
Example
User-agent: GPTBot Allow: / User-agent: * Disallow: /admin