Crawl Budget
Definition
Crawl budget is the practical limit on how many URLs a search engine will crawl on a site in a given period, shaped by demand and capacity.
By Vinespire Editorial Team, Editorial ·
This term is part of the full AI search glossary.
Full definition
Large or frequently changing sites cannot be fully recrawled every day. Engines allocate crawl activity based on factors such as site popularity, server health, and how useful prior crawls were. Wasting fetches on infinite filters, soft errors, or low-value parameters can delay discovery of important updates.
Improving crawl efficiency means cleaner architecture: sensible sitemaps, strong internal links, stable status codes, fewer near-duplicates, and robots rules that exclude pure junk without hiding critical sections.
AI training and live-browsing crawlers have their own rate and policy limits; treating crawl access as infinite is unsafe. Prioritize paths that carry entity facts, docs, and commercial truth. See robots.txt, Faceted Navigation, and Log File Analysis.