Allow Directive (robots.txt)
Definition
Allow is a robots.txt rule that permits a crawler to fetch a path, often used to carve exceptions inside a broader Disallow.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
Within a User-agent group, Allow: /path tells compliant bots that a URL prefix may be crawled even when a less specific Disallow would otherwise block a parent path. Specificity and rule-ordering behavior can differ by crawler; Google’s documented longest-prefix matching is a common reference, but operators should verify each bot’s rules.
Allow does not force a bot to crawl or index a URL—it only removes a robots.txt fetch restriction. Authentication, noindex, and network controls still apply.
AI visibility work often uses Allow to keep documentation open to GPTBot or OAI-SearchBot while Disallowing private app routes. See Disallow Directive, robots.txt, and User-Agent.
Example
User-agent: GPTBot Disallow: / Allow: /docs/