Crawl Budget SEO: What It Is and How to Diagnose It

In SEO, crawl budget is the number of URLs Google can and wants to crawl on your site in a given period. It is set by two things: the crawl capacity limit (how much crawling your server handles without slowing down) and crawl demand (how much Google wants to crawl, based on how many URLs it knows about, how popular they are and how often they change). Most stores never hit it, but large or fast-changing catalogs can, and the signs show up in Search Console long before rankings suffer.

This guide is the definition-and-diagnosis piece for e-commerce and Shopify teams. You'll learn what crawl budget is and isn't, Google's own thresholds for when it matters, how a modest catalog turns into tens of thousands of crawlable URLs, the symptoms of a real crawl budget problem and a step-by-step way to confirm one. When you're ready to fix it, our companion guide to crawl budget optimization covers the remedies, including AI crawlers.

What is crawl budget in SEO?

Google defines crawl budget as the set of URLs that Googlebot can and wants to crawl. Two forces decide it.

Crawl capacity limit

This is the maximum crawling Google will do without overloading your server. According to Google's crawl budget guide, the limit rises when your site responds quickly and consistently, and falls when responses slow down or return 5xx errors or HTTP 429. Google's own crawling resources are finite too, so it has to prioritize across the whole web.

Crawl demand

This is how much Google wants to crawl. Google names three factors:

• Perceived inventory. Without guidance, Googlebot tries to crawl most of the URLs it knows about on your site. Google calls this the factor you can control most. Duplicate and unwanted URLs waste crawl time here.
• Popularity. URLs that are more popular on the web tend to be crawled more often.
• Staleness. Google tries to recrawl often enough to pick up changes.

Two details catch people out. First, Google treats each hostname as a separate site, so shop.example.com and www.example.com have separate crawl budgets. Second, any URL Googlebot fetches counts, including CSS, JavaScript and API calls that a page needs to render. Google said so in its 2017 explainer on what crawl budget means for Googlebot, and it is still true: a theme that loads dozens of script files spends crawl requests on every page view Googlebot renders.

Crawling, crawl budget and indexing compared

Most "crawl budget problems" turn out to be something else. Before you diagnose anything, be clear about which stage is failing.

The key line is between crawling and indexing. If Google is fetching your pages and then declining to index them, more crawl budget won't help. That is a quality or duplication issue, and our guide to index bloat covers it.

Does crawl budget matter for your store?

Google is direct about who its crawl budget guide is written for, and it calls these numbers rough estimates rather than hard cutoffs.

If your pages seem to be crawled the same day they are published, Google says you don't need its crawl budget guide. For many Shopify stores with a few hundred products, that is the honest answer.

The catch for e-commerce is what "pages" means. Google counts URLs, not products. A store with 2,000 products can expose many times that number of crawlable URLs once you add collection paths, filters, sort orders, pagination and variant parameters. And the "daily changes" threshold is easy for a store to hit: prices, stock levels, sale badges and new arrivals change every day.

Ask yourself three questions:

• Are more than about 10,000 URLs crawlable on the store, counting filtered and parameter URLs, not just products?
• Do prices, stock or product details change daily, and do you need Google to show the current version?
• Is the "Discovered, currently not indexed" count in Search Console large or growing?

If you answered yes to any of them, the diagnosis steps below are worth an afternoon.

How a Shopify store multiplies its URLs

Here is a worked example. Picture a store with 2,000 products and 100 collections. Its "real" inventory, the URLs you want in search, is about 2,100 pages. Now count what a crawler can actually reach.

In this example the store exposes about 18,600 crawlable URLs, nearly nine for every page it wants indexed. The numbers depend entirely on the assumptions in the chart note, but the pattern is real: duplicates outnumber originals, and all of them compete for the same crawl capacity.

What Shopify already handles

Shopify does more of this cleanup by default than most platforms. On live Shopify stores we checked in October 2026 (allbirds.com and ballerinafarm.com), we saw the following:

• robots.txt blocks the worst offenders. The default rules disallow collection URLs with sort_by, tag combinations joined with + and URLs with two or more filter parameters, plus cart, checkout and account paths.
• Canonical tags consolidate duplicates. A collection-aware product path such as /collections/all/products/handle and a ?variant= URL both canonicalized to /products/handle. A single-filter collection URL canonicalized to the unfiltered collection, and page 2 of a collection was self-canonical.
• Sitemaps include lastmod. The auto-generated sitemap.xml is an index split into products, collections, pages, blogs and articles, and product entries carry a <lastmod> date, which Google's guide recommends.

Canonical tags are output by the theme, so check yours rather than assuming. And canonicals don't stop crawling: Google still has to fetch a duplicate to read its canonical. That is why single-filter URLs and collection-aware product links can still eat crawl budget even when they are handled correctly for indexing. Our guide to faceted navigation SEO goes deeper on filters.

Signs you have a crawl budget problem

Crawl budget problems have a recognizable set of symptoms. Use this grid to match what you see to where to look.

One symptom alone isn't proof. A slow first crawl for a brand-new store, or a single day of 5xx errors during a sale, is normal. You're looking for patterns that persist over weeks.

How to diagnose crawl budget step by step

This process works in Search Console alone, with server logs as an optional final check. It needs a Domain property or a root-level URL-prefix property, because the Crawl Stats report is only available for root-level properties.

1. Count your intended inventory. Add up products, collections, pages and articles you want indexed. Your sitemap index is a fast way to get this; the free sitemap checker lists what's in it.
2. Compare it with what Google knows. In the Page indexing report, add indexed and not indexed URLs together. If Google knows of several times more URLs than you intend, you have an inventory problem.
3. Read the "not indexed" reasons. "Discovered, currently not indexed" points to crawling. "Crawled, currently not indexed" and duplicate statuses point to quality or canonicals, not crawl budget. Open the examples and look for patterns such as ?filter, ?page= or /collections/*/products/.
4. Open Crawl Stats. Find it under Settings in Search Console. Note total crawl requests, average response time and host status for the last 90 days.
5. Break crawl requests down. Check the split by response code, by file type and by purpose. A high share of 404s, redirects or JavaScript files means wasted fetches. A low share of discovery requests on a growing catalog means Google isn't finding new URLs quickly.
6. Spot-check key pages. Run URL Inspection on your top five products and newest five products. Compare "Last crawl" dates with when they last changed.
7. Test your robots.txt. Confirm the rules block what you think they block with the robots.txt tester, especially if you've edited robots.txt.liquid.
8. Confirm with logs if you can. Search Console shows samples. Server or CDN logs show every Googlebot request, which tells you exactly which URL patterns absorb crawling. Our guide to log file analysis walks through it.

How to read the Crawl Stats report

The Crawl Stats report is the closest thing to a crawl budget dashboard Google provides. Here's what each part tells you.

Totals over time

Total crawl requests, total download size and average response time. Google's help page defines average response time across all resources fetched, so it includes images and scripts, not just HTML. A sudden drop in requests alongside a rise in response time is the classic sign of the capacity limit being lowered.

Host status

Google tracks three availability checks: robots.txt fetching, DNS resolution and server connectivity. Google flags an issue when a failure rate crosses the threshold on the chart, and shows whether a significant issue happened in the last week or earlier in the past 90 days. Robots.txt failures matter most: if Google can't fetch the file, it stops crawling for the first 12 hours.

Breakdowns

• By response: the share of 200, 301, 404 and 5xx responses. Rising redirects or 404s mean internal links point to dead or moved URLs. The redirect checker helps trace chains, and our guide to crawl errors covers fixes.
• By file type: HTML versus JavaScript, CSS, images and other files.
• By purpose: discovery (URLs never crawled before) versus refresh (recrawls of known URLs).
• By Googlebot type: smartphone, desktop, image, page resource load, AdsBot, StoreBot and others.

Turning requests into a recrawl cycle

A useful back-of-envelope check is to divide your crawlable URLs by your daily HTML crawl requests. It shows roughly how long Google would need to get through everything once. Google doesn't crawl evenly, so treat it as a rough gauge, not a forecast.

Same server, same crawl rate, and the store's important pages wait roughly nine times longer between visits because duplicates are in the queue. That is the real cost of crawl waste on a store: stale prices and slow pickup of new products, not a ranking penalty.

Crawl budget myths to ignore

A lot of crawl budget advice is out of date or never applied to Google at all.

Google's 2017 explainer also ranked the URL types that drain crawling, in order of significance: faceted navigation and session identifiers, on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low-quality or spam content. For a store, the first two on that list are almost always the place to start.

What to do once you've confirmed a problem

If your diagnosis shows a bloated inventory, a slow server or important pages going stale, the fixes follow directly: cut low-value URLs, keep key pages close to the homepage in your internal links, fix errors and redirect chains, and keep response times fast. Our crawl budget optimization guide covers each of those in order, plus how GPTBot, ClaudeBot and other AI crawlers change the picture.

If the diagnosis shows Google is crawling fine but not indexing, switch to quality and duplication instead. And if you'd like a second pair of eyes on your Crawl Stats and Page indexing reports, our technical SEO team can help. Get in touch.