Sitemap vs Crawl Comparator

An XML sitemap is a list of the URLs you want search engines to index. That is the entire purpose, and it explains why the two common failure modes are both breaches of the same rule.

A sitemap containing URLs you do not want indexed sends a contradiction — you have explicitly listed a page as important while also telling crawlers to ignore it. A sitemap missing pages you do want indexed removes a discovery route, which matters most for new pages and for pages buried deep in the structure.

Neither problem prevents indexing outright. Both make it slower and less predictable, and on large sites that difference compounds.

URLs listed in the sitemap that return a redirect, a 404, or carry a noindex tag or a canonical pointing elsewhere. Each one wastes crawl budget and muddies the signal. Redirects are the most common: a URL changed, the sitemap was regenerated from a stale source, and the old entry survived.

Pages that return 200, are indexable, and are not listed. Usually a generation problem — a page type excluded from the sitemap logic, or content created outside the normal publishing flow. New pages are the most affected, since links pointing at them are still sparse.

URLs in the sitemap with no internal links pointing at them. The sitemap tells search engines the page matters while the site structure says nothing leads there. That contradiction is worth resolving, and the resolution is usually a link rather than a sitemap edit.

Faceted navigation and tracking parameters generating sitemap entries at scale. Common on ecommerce platforms and almost always unintentional.

Include: canonical, indexable pages that return 200 and that you would be happy to see in search results. Products, collections, articles, key landing pages.

Exclude:

• Redirected URLs — list the destination instead
• 404s and error pages
• Noindexed pages
• Non-canonical duplicates
• Parameter and filtered URLs
• Paginated series beyond page one, in most cases
• Internal search results
• Thank-you and checkout pages

The test is simple: would you want this URL to appear in search results? If not, it does not belong in the sitemap.

Shopify generates sitemaps automatically at sitemap.xml, split into child sitemaps for products, collections, pages and blogs. That automation is mostly helpful and has some specific characteristics.

• You cannot edit it directly. Inclusion is driven by product and page status, not by manual selection.
• Unavailable products can persist in the sitemap while their pages behave differently, which produces mismatches.
• Collections with no products may be listed while offering nothing to index.
• Tag-based filtered URLs are the most common source of unwanted entries.

Because you cannot hand-edit the file, the fix is almost always upstream: change product status, remove empty collections, or adjust how filtered URLs are handled.

A single sitemap file can hold up to 50,000 URLs and must stay under 50MB uncompressed. Beyond that, split into multiple files referenced by a sitemap index.

Splitting by content type rather than arbitrarily is worth doing even below the limit. Separate sitemaps for products, collections and articles let you see indexing rates per type in Search Console, which turns a single unhelpful coverage number into a diagnosis. If products index at 95% and articles at 40%, you know where to look.

1. Fetch your sitemap at yoursite.com/sitemap.xml and follow any child sitemaps.
2. Crawl the site and export URLs with status codes, indexability and inlink counts.
3. Compare the two lists in both directions — in sitemap but not indexable, and indexable but not in sitemap.
4. Group findings by cause, not by URL. Almost all of them share a handful of causes.
5. Fix the generation logic where possible rather than the output.
6. Resubmit in Search Console and monitor the coverage report over the following weeks.

The sitemap comparison tells you what you submitted. Search Console tells you what Google did with it, and the gap between the two is where the useful diagnosis lives.

Google crawled the URL and chose not to index it. Common reasons are thin content, near-duplication of another page, or a canonical pointing elsewhere. This is a content or structure signal, not a sitemap problem — removing the URL from the sitemap hides the symptom rather than addressing it.

Google knows the URL exists but has not fetched it. Usually a crawl budget or priority signal, and it appears most often on large sites with many low-value URLs. Reducing the number of thin pages competing for attention helps more than resubmitting.

Google found and indexed pages your sitemap does not list. Not harmful, but it means your sitemap is not a complete statement of what you want indexed, which weakens it as a signal.

The most useful number to track is the proportion of submitted URLs that end up indexed, measured per content type rather than sitewide.

A products sitemap indexing at 95% and an articles sitemap at 40% is a specific, actionable finding. The same two figures averaged into a single 78% tells you nothing. This is the main practical argument for splitting sitemaps by type even when you are well below the 50,000 URL limit — it converts one vague metric into several precise ones.