engines
Crawl budget
The number of pages a crawler will fetch from your site in a given time window — a hard ceiling on indexation depth.
Definition
Crawl budget is the total number of URLs a search/AI engine bot will fetch from your domain per crawl session. Determined by your site's authority + responsiveness + size. Large sites with thin pages waste budget on low-value URLs and starve high-value pages of crawl attention. AI engine crawlers (GPTBot, ClaudeBot) typically have smaller crawl budgets than Googlebot, making prioritization more important.
Example
A 10K-page ecommerce site where 8K are sold-out PDPs. Googlebot crawls 2K/day → 5 days to refresh. Pruning sold-out URLs lifts refresh to 1 day for live SKUs.How to optimize
Block low-value URLs via robots.txt. Use noindex on thin pages. Submit XML sitemaps prioritizing high-value pages. Maintain fast TTFB so crawler can complete more requests per session.