Crawl Errors SEO: What to Fix First (and What to Ignore)

Crawl errors are the failures Googlebot hits when it tries to fetch your URLs: server errors, 404s, soft 404s, redirect errors and blocked requests. For SEO, fix them in order of what they cost you: errors on pages that earn traffic or revenue first, sitewide problems such as server errors and robots.txt failures next, and leave expected exclusions like removed pages, duplicates and intentionally blocked URLs alone.

This guide is for e-commerce and Shopify teams staring at a Search Console report full of red and gray. You will learn where Google reports crawl errors today, what each Page indexing reason actually means, a seven-step triage you can run in an afternoon, the exact fixes for each error type, and the Shopify quirks that create most of the noise on store sites.

What are crawl errors in SEO?

A crawl error happens when Googlebot requests a URL and does not get a usable page back. The request might time out, the server might return a 5xx status, the URL might not exist, a redirect might loop, or a rule might block Googlebot from fetching it at all. Google can only index and rank what it can fetch, so an error on an important URL is an error on your revenue.

The old "Crawl Errors" report in Search Console no longer exists. Google now spreads the same information across three tools:

• Page indexing report. Lists every URL Google knows about as indexed or not indexed, with a reason for each non-indexed group, such as "Server error (5xx)" or "Not found (404)".
• Crawl stats report (under Settings). Shows how many requests Googlebot made, the response codes it got and a 90-day host status for robots.txt fetching, DNS resolution and server connectivity. It is only available for root-level properties.
• URL Inspection tool. Shows the last crawl of one URL and lets you run a live test after a fix.

Google's own help page adds a useful reality check: if your site has fewer than 500 pages, you probably do not need the Page indexing report at all, and site: searches for your key pages will tell you enough. Most e-commerce stores pass that size quickly once collections, filters and variants are counted, which is why the report fills up.

Which crawl errors actually hurt SEO?

The single most important line in Google's Page indexing documentation is that "not indexed" is not necessarily bad. A URL can be unindexed for the right reason: a robots.txt rule you meant to add, a noindex tag on a thank-you page, a duplicate filtered URL, or a 404 for a product you discontinued with no replacement.

Crawl errors hurt when they hit one of three things:

1. URLs that should rank. A 5xx or 404 on a product, collection or guide that earns clicks. Google says 4xx URLs that were indexed get removed from the index, and URLs that keep returning server errors are eventually dropped.
2. The whole site's crawl rate. Google's HTTP status documentation says 5xx and 429 responses make Googlebot temporarily slow down crawling. A flaky server makes every page, including new products, get crawled less.
3. Your crawl budget on large sites. Soft 404s keep getting crawled, and long redirect chains waste fetches. Google's crawl budget guide is written for sites with roughly 1 million or more unique pages, or 10,000 or more pages that change daily, but catalogs with filter URLs reach that territory faster than you would expect.

Everything else is housekeeping. The Page indexing report includes a Source column for each reason, and Google notes that you can generally only fix issues where the source is "Website".

Page indexing report reasons, decoded

These are the reasons you will see most on store sites, what Google says each one means and whether it usually needs action. Use it to sort the report into "fix now", "check" and "ignore" before you touch anything.

Two of these trip people up. "Indexed, though blocked by robots.txt" is a warning, not an error: Google indexed the URL from links without fetching it. If you want it out of results, remove the robots.txt block and use noindex, because Google cannot see a noindex tag on a page it is not allowed to crawl. And the 404 list only shows URLs that returned 404 in the past month, so an old spike will age out on its own.

How to prioritize crawl errors in 7 steps

Working through the report top to bottom wastes days. Sort by impact instead. This is the triage order we recommend for store sites.

1. Check host status first. Open Settings, then Crawl stats. If robots.txt fetching, DNS or server connectivity shows a recent issue, fix that before anything else, because it affects every URL on the site.
2. Export each error reason. In the Page indexing report, open each reason and export the examples. Google caps the example list at 1,000 URLs and says it is not guaranteed to show every affected URL, so treat it as a sample of a pattern, not a complete list.
3. Join the export to your traffic data. Match the URLs against a Search Console Performance export (clicks and impressions) and your analytics revenue by landing page. Any error URL with clicks, revenue or backlinks goes to the top. The Search Console Quick Wins Analyzer helps you read a queries export quickly.
4. Group by pattern, not by URL. Most crawl errors come from one template, app or rule. Five thousand 404s from one deleted collection need one redirect rule, not five thousand fixes.
5. Check your sitemap and internal links. An error URL listed in your XML sitemap or linked from navigation is a self-inflicted problem. Run your sitemap through the XML Sitemap Checker to flag non-200 URLs.
6. Fix the cause, then spot-check. Use URL Inspection's live test on a few examples to confirm Googlebot now gets the right response.
7. Validate the fix. Click Validate fix on the reason. Google says validation typically takes up to about two weeks and stops early if it finds a single remaining instance, so fix every instance of the pattern before you start.

How to fix server errors and timeouts

Server errors are the crawl errors that cost the most, because Google responds to them by crawling the whole site more slowly. Google's documentation says the slowdown for 500 errors is proportional to how many URLs return them, and crawling speeds back up gradually once your server returns 2xx responses again.

Find the cause

• Overloaded or slow server. Look for spikes in average response time in Crawl stats, and compare them with sales, deploys or bot traffic. Google's Page indexing help points out that parameter-heavy dynamic URLs can take too long to respond and cause timeouts.
• Firewall or bot protection blocking Googlebot. Google calls this out directly: protection systems often treat Googlebot's request volume as an attack. Check your CDN or WAF logs for blocked requests from verified Googlebot IPs.
• 429 rate limiting. Google treats 429 like a server error. Fine as a short-term brake, but a permanent 429 for Googlebot means you are throttling your own indexing.
• DNS failures. In Crawl stats, Google gives a daily DNS failure rate above 5% as an example of an issue worth fixing.

Treat robots.txt as critical infrastructure

If your robots.txt file returns a 5xx or 429, Google does not crawl around it. Per the Crawl stats documentation, Google stops crawling the site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days. A 404 for robots.txt, by contrast, is treated as "no rules" and crawling continues. Test your file with the Robots.txt Tester after every change, and read our robots.txt best practices before editing it.

How to fix 404s, soft 404s and redirect errors

These are the errors store owners see most, because products sell out, collections get renamed and apps add and remove URLs. Google is relaxed about genuine 404s: its help page says a 404 is not necessarily a problem if the page was removed without a replacement, and recommends fixing only the 404s you link to yourself or list in a sitemap.

Google's 404 help page is blunt: many 404 errors are not worth fixing, because 404s do not harm your site's indexing or ranking. Google also treats 410 (Gone) the same as 404. What it warns against is the opposite habit: returning a 200 or redirecting to the homepage for pages that no longer exist, which creates soft 404s.

Googlebot follows up to 10 redirect hops by default, but every hop is an extra fetch and a place for something to break. Check chains with the Redirect Checker, and when you have a list of broken URLs to map, the 404 Redirect Mapper matches each one to the closest live page and exports an import-ready file. Our guides to 404 errors and redirect chains go further.

For pages you have removed for good, Google's crawl budget guide recommends returning 404 or 410, calling a 404 a strong signal not to crawl that URL again. Redirect only when a close equivalent exists, such as a replacement product or the parent collection.

Crawl errors on Shopify stores

Shopify hosts your store, so server-level fixes are mostly out of your hands. The crawl errors you can control usually come from the catalog itself:

• Deleted products and changed handles. Shopify's help center says that when you change a URL you can create a URL redirect so customers still find the page, and that after deleting a product you can set up a redirect to a similar one. You manage them under Content, then Menus, then URL redirects. Do it for any page that earned traffic or links.
• Filtered and sorted collection URLs. Storefront filters add parameters such as filter.v.option.color=black. These usually show up as duplicates or "blocked by robots.txt", which is expected. Our guide to faceted navigation SEO covers which ones to keep.
• Empty collections. An automated collection whose rules match nothing still returns 200, which is a classic soft 404.
• App and theme leftovers. Uninstalled apps can leave links in theme code that now 404. Search your theme for the app's path after uninstalling.
• Market and language subfolders. With Shopify Markets, every error can appear once per subfolder, so group by pattern before you count.

Shopify's default robots.txt blocks cart, checkout, account and sorted or multi-filter collection URLs. You can customize it with the robots.txt.liquid theme template, and Shopify recommends saving a copy of your current file and keeping the default rules rather than replacing them with plain text.

Crawl errors and crawl budget

For most stores, crawl budget is not the bottleneck, and fixing a few hundred 404s will not change rankings. It becomes relevant when Search Console shows large numbers of "Discovered, currently not indexed" URLs, which Google lists as one of the signs its crawl budget guide applies to you.

The bigger crawl problem is usually not errors at all, but URL spaces that never end. When Google's Gary Illyes discussed the team's 2025 year-end review of crawl issue reports, faceted navigation alone accounted for close to half.

Action parameters are URLs such as ?add-to-cart=123 that trigger an action instead of showing different content. Irrelevant parameters include tracking codes and session IDs. The lesson for stores is that filters, cart links and tracking parameters create far more crawl waste than the occasional 404. Our guides to crawl budget and index bloat show how to find it, and log file analysis shows exactly where Googlebot spends its time.

A monthly crawl error routine

Crawl errors are never finished, because your catalog never stops changing. A short monthly check keeps them from building up:

• Check Crawl stats host status and response codes for anything new.
• Look at the Page indexing chart for jumps in any "Website" source reason, and match each jump to a deploy, app install or catalog change.
• Re-run your sitemap through a checker so only 200, indexable, canonical URLs are listed.
• Redirect any newly deleted product that had clicks or links.
• Re-inspect your top 20 revenue pages with URL Inspection.

For a full review of everything Google sees on your store, start with our technical SEO audit checklist or run a quick scan with the Website SEO Checker. If the report is too big to untangle alone, our technical SEO team can help you sort what matters, or you can get in touch.