Crawl-delay
Definition
Crawl-delay is an unofficial robots.txt directive that asks a crawler to wait a stated number of seconds between successive requests.
By Vinespire Editorial Team, Editorial ·
Crawlers & botsAI search glossary
This term is part of the full AI search glossary.
Full definition
Some crawlers historically honored Crawl-delay: N inside a User-agent group as a politeness hint. It is not part of a single universal robots.txt standard, and major engines such as Google ignore it in favor of their own crawl-rate controls.
Where supported (certain bots document support), Crawl-delay can reduce load on fragile origins. Overly large delays may slow discovery of new content. Prefer CDN capacity, caching, and clean URL spaces before relying on Crawl-delay.
AI crawlers vary: check each operator’s docs rather than assuming Crawl-delay works sitewide. See robots.txt, Crawl Budget, and Bot Management.
Example
User-agent: ClaudeBot Crawl-delay: 10