Disallow Directive (robots.txt)
Definition
Disallow is a robots.txt rule that asks matching crawlers not to fetch URLs under a given path prefix.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
A Disallow: /admin line in a User-agent group is a crawl preference for well-behaved bots, not a security lock. Empty Disallow (Disallow: with no path) typically means “allow all” for that group. Disallow: / blocks the whole site for that agent.
Paths blocked in robots.txt may never be fetched, so meta robots noindex on those URLs may never be seen. Use authentication for secrets; use Disallow for crawl hygiene and policy.
Blocking AI training agents while allowing search-oriented agents is a common GEO policy split—implemented with separate User-agent groups and explicit Disallow/Allow lines. See Allow Directive, AI Training Opt-Out, and GPTBot.
Example
User-agent: CCBot Disallow: /