Crawl Budget Guide for large sites
This guide is designed to help you optimize how Google crawls large, frequently updated websites. If your site doesn’t have many pages that change often or if your pages are typically crawled the same day they’re published, you probably don’t need to read this guide. Simply keep your sitemap updated and regularly check your index coverage—that should be enough. However, if you have pages that have been around for a while but still aren’t indexed, that’s a different issue. In that case, use the URL Inspection tool to find out why those pages aren’t being indexed.
This guide is written for:
- Large sites (1 million+ unique pages) with content that changes moderately often (once a week)
- Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)
- Sites with a large portion of their total URLs classified by Search Console as Discovered – currently not indexed
* Above numbers are estimates & not thresholds.
Understanding How Google Crawls Websites
The web is so vast that it’s impossible for Google to explore and index every URL. Because of this, Googlebot (the tool Google uses to crawl websites) has to limit how much time it spends on each site. The amount of time and resources Google allocates to crawling a site is called the site’s “crawl budget.” However, just because a page is crawled doesn’t mean it will be indexed. Each page is evaluated before Google decides to include it in its index.
The crawl budget is influenced by two key factors: crawl capacity limit and crawl demand.
Crawl Capacity Limit
Googlebot aims to crawl your site without overwhelming your servers. To do this, it calculates a crawl capacity limit—this is the maximum number of simultaneous connections Googlebot can use to crawl your site and the time delay between fetches. This ensures that Google can cover your important content without causing server issues.
The crawl capacity limit can fluctuate based on:
- Crawl health: If your site responds quickly, the limit increases, allowing Googlebot to make more connections. If your site slows down or has server errors, the limit decreases, and Googlebot crawls less.
- Google’s resource limits: While Google has many machines, they aren’t infinite, so choices must be made about how resources are used.
Crawl Demand
Crawl demand refers to how much time Googlebot spends crawling your site based on factors like size, how often content is updated, page quality, and relevance compared to other sites.
Key factors that influence crawl demand include:
- Perceived inventory: Googlebot will attempt to crawl most URLs it knows about on your site. If many of these URLs are duplicates or unnecessary, it wastes Google’s crawling time. This is where you have the most control.
- Popularity: URLs that are more popular tend to be crawled more often to keep them updated in Google’s index.
- Staleness: Google wants to recrawl pages frequently enough to catch any changes.
- Site-wide events: Events like site moves can trigger a spike in crawl demand as Google reindexes content under new URLs.
Google’s crawl budget for your site is determined by a combination of crawl capacity and crawl demand. If demand is low, Googlebot may crawl your site less, even if the crawl capacity limit isn’t reached. Understanding and managing these factors can help you optimize how Googlebot interacts with your site.
Best Practices for Optimizing Crawl Efficiency
Google Says:
Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time. Don’t use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don’t want Google to crawl at all. Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.
Manage Your URLs Wisely:
Use the right tools to guide Google on which pages to crawl and which to skip. If Google spends too much time on irrelevant URLs, it might overlook other important parts of your site.Eliminate Duplicate Content:
Focus Google’s attention on unique content by removing duplicates. This ensures that Googlebot crawls valuable pages instead of wasting time on different URLs with the same content.Use robots.txt to Block Unwanted Crawling:
Some pages are important for users but don’t need to appear in search results, like pages with infinite scrolling or different versions of the same content. If you can’t consolidate these, block them with robots.txt to prevent Google from indexing them.Avoid Using noindex for Crawl Management:
Google will still request pages marked with a noindex tag, which wastes crawling time. Instead of using noindex or robots.txt to temporarily manage crawl budget, only block pages that shouldn’t be crawled at all.Return a 404 or 410 for Removed Pages:
When pages are permanently removed, return a 404 or 410 status code. This tells Google not to crawl that URL again, unlike blocked URLs, which stay in the crawl queue longer.Fix Soft 404 Errors:
Soft 404 pages waste your crawl budget because they continue to be crawled. Check the Index Coverage report regularly to identify and fix these errors.Keep Sitemaps Updated:
Ensure your sitemap is always up to date with the content you want Google to crawl. If you have content that frequently changes, use the<lastmod>tag to indicate the latest update.Avoid Long Redirect Chains:
Long redirect chains can slow down crawling and should be minimized.Optimize Page Loading Speed:
Faster page loading means Google can crawl more content in less time. Optimize your pages to load efficiently.Monitor Your Crawl Activity:
Regularly check for any availability issues during crawling and seek opportunities to improve crawl efficiency.
To optimize Google’s crawling of your site, focus on managing URLs, eliminating duplicate content, and using robots.txt to block unnecessary pages from being crawled. Avoid using noindex tags for crawl management; instead, return a 404 or 410 status for permanently removed pages. Regularly update your sitemap, fix soft 404 errors, and minimize long redirect chains. Speed up page loading to allow Google to crawl more content efficiently. Lastly, monitor your site’s crawl activity to identify and resolve any issues, ensuring a more effective and efficient crawling process.
