Crawl Budget Optimization for Ecommerce and AI Crawlers

Crawl budget optimization means making sure search engine and AI crawlers spend their limited visits on the pages you want found, not on duplicate, filtered, redirected or broken URLs. You do it by cutting low-value URLs, keeping important pages a few clicks from the homepage, fixing errors and redirects, and keeping your server fast. With GPTBot, ClaudeBot and PerplexityBot now crawling alongside Googlebot, a clean URL structure matters more than ever, because AI crawlers waste far more requests than Google does.

This guide is for e-commerce and Shopify teams with growing catalogs. You'll learn how Google defines crawl budget and whether your store needs to worry about it, how AI crawlers behave differently, where Shopify stores waste crawls, a step-by-step optimization process and how to check your progress in Search Console and server logs.

What is crawl budget optimization?

Google defines crawl budget as the set of URLs that Google can and wants to crawl on your site. It has two parts:

• Crawl capacity limit. How much crawling your server can handle without slowing down. Google raises it when your site responds quickly and lowers it when responses slow down or return 5xx errors or HTTP 429.
• Crawl demand. How much Google wants to crawl, based on how many URLs it knows about, how popular they are and how often they change. Site moves can raise demand for a while.

Optimization works on both sides. You raise capacity by keeping the server fast and error free, and you focus demand by removing the junk URLs that compete with your products and collections for attention. Google also notes that the capacity limit is shared across all its crawlers, so wasted Googlebot requests on one section can slow discovery everywhere else.

Crawl budget versus indexing

Crawling is Google fetching a page. Indexing is Google deciding to store it and show it in search. A page has to be crawled before it can be indexed, but being crawled doesn't guarantee indexing. If your problem is that Google crawls pages and then declines to index them, you have a quality or duplication problem, not a crawl budget problem. Our guide to index bloat covers that side.

Does your store need to worry about crawl budget?

Most small stores don't. Google says its crawl budget guide is for sites with roughly 1 million or more unique pages that change about weekly, sites with 10,000 or more pages that change daily, and sites with a large share of URLs stuck in "Discovered, currently not indexed." If new pages are usually crawled the day you publish them, Google says you don't need to manage crawl budget.

The catch for e-commerce is that "unique pages" means URLs, not products. A 3,000-product store with filters, sort options, variant parameters and collection-aware product paths can expose many times that number of crawlable URLs. That is why mid-size stores often show crawl symptoms long before they reach Google's thresholds.

Even if you're in the low group, the steps below are worth doing. Crawl hygiene improves how fast new products get picked up, and it determines what AI crawlers see when they visit.

How AI crawlers change crawl budget

Google's crawl budget is about Googlebot. But your server now answers a second wave of crawlers from AI companies, and they behave differently. Vercel measured crawler traffic across its network for a month in late 2024. OpenAI's GPTBot made 569 million requests and Anthropic's Claude crawler 370 million. Together with AppleBot and PerplexityBot, the AI crawlers it tracked added up to about 28% of Googlebot's 4.5 billion requests.

Cloudflare's 2025 Year in Review shows the trend has continued. From January 1 to December 2, 2025, AI bots made 4.2% of HTML requests across Cloudflare's network on average, close to Googlebot's 4.5%. Googlebot was still the most active verified bot at over 28% of verified bot traffic, with GPTBot next at about 7.5%.

AI crawlers waste far more requests

The bigger issue is efficiency. In Vercel's data, about 35% of ChatGPT's and 34% of Claude's fetches hit 404 pages, compared with 8% for Googlebot. ChatGPT also spent another 14% of fetches following redirects. AI crawlers are requesting old, broken and guessed URLs at a high rate, so every dead link, expired product URL and redirect you leave in place costs more crawling than it used to.

Most AI crawlers don't run JavaScript

Vercel also found that none of the major AI crawlers it tested render JavaScript, including OpenAI's crawlers, ClaudeBot and PerplexityBot. They may download JavaScript files but don't execute them. Googlebot, Gemini and AppleBot do render. For a Shopify store, the core of a Liquid theme is server rendered, but content injected by apps after the page loads, such as some review widgets, FAQ blocks or size guides, may be invisible to AI crawlers. Check by viewing your page source, not the rendered page.

User-triggered fetchers such as ChatGPT-User and Claude-User only visit when a person asks a question. OpenAI notes robots.txt rules may not apply to ChatGPT-User because a user starts the request. For the full breakdown of OpenAI's bots, see GPTBot vs OAI-SearchBot.

Where Shopify stores waste crawl budget

Most crawl waste on e-commerce sites comes from a handful of URL patterns that multiply. Shopify handles some of them by default, but not all.

These defaults were checked against a live Shopify store's robots.txt and pages, so confirm them on your own store; themes and apps can change them. Shopify says its default robots.txt works for most stores and that editing robots.txt.liquid is an unsupported customization where mistakes can cost you traffic. Our faceted navigation guide covers filter URLs in depth, and canonical tags for SEO explains the product path duplicates.

How to structure URLs for search and AI crawlers

Both Google and AI crawlers find pages mainly by following links and reading sitemaps. Vercel's advice for being crawled by AI is the same as good SEO architecture: keep redirects maintained, sitemaps current and URL patterns consistent. Here is what that means for a store.

One URL per page, used everywhere

Pick one URL format for products and collections, and use it in navigation, internal links, sitemaps and structured data. Every alternative path you link to is another URL for crawlers to fetch.

Keep priority pages close to the homepage

Pages linked from your main navigation, homepage and top collections get crawled more often than pages buried behind several layers of pagination. Make sure best sellers and key collections are reachable within a few clicks, and add links from related products, buying guides and blog posts to the pages you want crawled most.

Group by topic

A clear hierarchy (category, subcategory, product) with descriptive, stable handles helps crawlers understand how pages relate. Avoid renaming handles once pages are live; every change creates a redirect that crawlers keep requesting.

Keep content in the HTML

Put product descriptions, specs, prices, availability and FAQs in the server-rendered HTML so AI crawlers that don't run JavaScript can read them.

How to optimize crawl budget step by step

1. Measure your baseline. Open the Crawl Stats report in Google Search Console (it needs a Domain property or a root-level URL prefix property). Note total requests, average response time and the breakdown by response code, file type and purpose (discovery versus refresh).
2. Read your server logs. Search Console only shows Google. Your logs or CDN analytics show what GPTBot, ClaudeBot, PerplexityBot and others request. Our log file analysis guide walks through the process.
3. Cut duplicate and low-value URLs. Use robots.txt to block URL patterns you never want crawled, such as extra sort orders or infinite filter combinations, as Google recommends. Test changes with our robots.txt tester before you publish them.
4. Return the right status codes. Google recommends 404 or 410 for permanently removed pages and eliminating soft 404s, which keep getting crawled. Don't serve a 200 "no products found" page for empty filters.
5. Fix redirect chains. Update internal links to point to the final URL and collapse chains to a single hop. Our redirect checker shows every hop.
6. Clean your sitemap. List only canonical, indexable URLs that return 200, and keep lastmod accurate for updated content. Check it with the sitemap checker.
7. Speed up the server. Faster, stable responses raise Google's crawl capacity. Support HTTP caching so Google can get 304 (Not Modified) responses for unchanged pages.
8. Decide your AI crawler policy. Allow the search crawlers you want to be cited by, such as OAI-SearchBot and Claude-SearchBot, and make a deliberate choice about training crawlers. Anthropic supports a Crawl-delay directive for its bots if load is a problem. See should you block GPTBot before you block anything.
9. Re-check monthly. Compare crawl stats and log data with your baseline, and watch the share of requests that hit 404s and redirects.

Crawl budget mistakes that waste time

Some common tactics don't do what people expect. These come straight from Google's crawl budget guidance.

How to measure crawl efficiency

Track a few numbers each month and compare them with your baseline:

• Share of crawl requests returning 200. From Crawl Stats in Search Console and your logs. Rising 404 and 3xx shares mean old URLs are eating requests.
• Average response time. From Crawl Stats. Spikes usually mean server strain, which lowers crawl capacity.
• Time to first crawl for new products. Check new product URLs with URL Inspection a few days after launch.
• "Discovered, currently not indexed" count. From the Page indexing report. A falling number means Google is reaching more of the pages you care about.
• AI crawler requests by status code. From your logs. Watch how many GPTBot, ClaudeBot and PerplexityBot requests hit real product and collection pages versus 404s and redirects.

Crawl Stats also shows host status for the past 90 days: whether Google could fetch your robots.txt, resolve your DNS and connect to your server. Any failures there come before every other fix. For a full review of technical issues, use our technical SEO audit checklist.

Where to start this week

Open Crawl Stats, export a week of server logs and answer two questions: what share of bot requests hit pages you want found, and which URL patterns take the rest. For most stores, the answer points to a short list of fixes: filter and sort URLs, old product URLs, redirect chains and a messy sitemap. Fix those first and the pages that make you money get crawled sooner, by Google and by AI tools.

If you'd rather have experts do it, our technical SEO team audits crawl data and logs for Shopify and e-commerce stores, or you can contact us for a free plan.