How to Scan All Pages of a Website (and What to Fix)

To scan all pages of a website, combine three sources: crawl the site with an SEO crawler that follows every internal link, pull every URL from the XML sitemap, and export the indexed and non-indexed URLs from Google Search Console. No single source finds everything, so the pages that appear in one list but not the others are where most SEO problems hide.

This guide is for store owners and marketers who want a complete inventory of their site and a clear list of what to fix. You'll get the methods compared side by side, the free crawler limits, a step-by-step scanning process, what to check on every page, the Shopify specifics, and how to turn a crawl export into a prioritized fix list.

Why you should scan every page of your website

Google can only rank pages it can find, crawl and understand. Most sites have pages that fail one of those steps without anyone noticing: a product that lost its last internal link, a collection that started redirecting after a menu change, a blog template that shipped with a noindex tag. A full scan is how you catch these before they cost traffic.

Google's own tools won't give you the full picture on their own. Google's documentation for the site: search operator says the list of URLs it returns is not always exhaustive, and bigger sites shouldn't expect to see all their URLs. The Search Console Page indexing report limits its example URL lists to 1,000 items and doesn't guarantee to show every URL in a given status. To see every page, you need your own crawl.

What a scan tells you that Google won't

• Pages Google can't reach. URLs in your sitemap or catalog with no internal links pointing to them (orphan pages).
• Pages you didn't know existed. Parameter URLs, old campaign pages, test templates and duplicate paths that crawlers can still find.
• Why a page isn't indexed. The status code, canonical, robots directives and content of every URL, side by side.
• How deep pages sit. How many clicks each page is from the homepage, which shapes how often it gets crawled.

Five ways to find all pages on a website

Each method sees the site from a different angle. Use at least the first three for a complete scan, and add the others when you need to dig into a specific problem.

The site:yourdomain.com search is fine for a rough sanity check, but don't use its result count as a page count. Google says it doesn't necessarily return every indexed URL.

How to scan all pages of a website, step by step

This process works for any site, from a 50-page store to a catalog with tens of thousands of products. Budget an afternoon for a mid-sized store the first time.

1. Check robots.txt first. Open yourdomain.com/robots.txt and note which paths are blocked. Run a few key URLs through our Robots.txt and AI Crawler Tester so you know whether Googlebot can reach them. A crawler set to obey robots.txt will skip blocked areas, which is what you want for a Google's-eye view.
2. Collect your sitemap URLs. Find the sitemap index (usually /sitemap.xml, or the Sitemap line in robots.txt) and validate it with our XML Sitemap Checker, which also spot-checks that listed URLs return 200 and are indexable.
3. Configure the crawler. Start from the homepage, choose a Googlebot smartphone user agent, turn on JavaScript rendering if your theme builds links with JavaScript, and add your sitemap so the tool crawls sitemap URLs too.
4. Set a polite speed. Screaming Frog crawls at 5 threads by default "to not overload servers," and warns that very high thread counts can crash a server. Keep the default unless your host can clearly handle more.
5. Run the crawl and export everything. Export all internal HTML URLs with status code, indexability, canonical, meta robots, title, H1, word count, crawl depth and inlinks.
6. Export Search Console data. From the Page indexing report, export each "Why pages aren't indexed" reason. From the Performance report, export pages with clicks and impressions for the last three months.
7. Merge the lists. Put crawl, sitemap and Search Console URLs in one sheet and flag where each URL appears. URLs in the sitemap but not the crawl are likely orphans. URLs in the crawl but not the sitemap may be duplicates or pages you forgot to include.
8. Triage and fix. Sort issues by the revenue of the pages affected, fix the template-level problems first, then recrawl to confirm.

How long a full crawl takes

Crawl time depends on how many URLs your site exposes and how fast the server answers. The URL count is often much higher than your page count, because filters, sorting and pagination create extra URLs. Here is the simple math for a site with 20,000 crawlable URLs:

Free and paid crawlers for scanning a site

Free plans are enough for small stores and for spot checks. Once a site exposes more than a few thousand URLs, you'll need a paid crawler or a verified free account with higher limits. These are the limits as each vendor publishes them:

Ahrefs says its credits are only used on HTML pages that return a 200 status, so redirects, broken URLs and non-HTML files don't count. For a quick inventory of a small site, our XML Sitemap Generator crawls up to 300 pages and gives you a downloadable URL list.

What to check on every page you scan

A crawl export has dozens of columns. These are the ones that decide whether a page can rank, grouped by the question each one answers.

Can Google reach it?

• Status code. Pages you want ranked should return 200. Fix 4xx errors on linked pages and update internal links that point to redirects.
• Redirect chains. A link that hops through two or more redirects slows crawling. Point links straight at the final URL. Our Redirect Checker traces every hop.
• Robots.txt. Make sure no rule blocks pages, CSS or JavaScript that you need indexed and rendered. See our robots.txt best practices for what to block.
• Crawl depth and inlinks. Important products and collections should be a few clicks from the homepage and have several internal links.

Will Google index it?

• Meta robots and X-Robots-Tag. Look for unexpected noindex values, especially on templates.
• Canonical tag. Each indexable page should usually point to itself. A canonical pointing elsewhere tells Google to index the other URL instead.
• Duplicates. Filter, sort and tracking parameters often create copies of the same page. Google's Page indexing report lists these as "Duplicate without user-selected canonical" when you haven't chosen a preferred URL.

Is it worth ranking?

• Title and H1. Missing, duplicated or truncated titles are among the most common crawl findings and the easiest to fix in bulk.
• Thin content. Sort by word count to find empty collections, placeholder pages and products with one-line descriptions.
• Soft 404s. Pages that say "not found" or "sold out" but return 200. Google recommends returning a real 404 for pages that no longer exist.

For context on how common these elements are across the web, the 2025 Web Almanac found that 68% of desktop pages have a canonical tag and the median desktop page has 43 internal links:

How to scan every page of a Shopify store

Shopify makes some of this easier and some of it harder. Shopify's help center says every store gets an automatically generated sitemap.xml at the root of its domain that links to child sitemaps for products, collections, pages and blog posts, and that it updates as you add content. In the live sitemaps we checked on October 27, 2026, Shopify also listed article and metaobject page sitemaps, and split large catalogs across several numbered product sitemaps. The sitemap can't be edited directly.

Shopify quirks to account for

• No server logs. Shopify is hosted, so log-file analysis is not available. Rely on Search Console's Crawl Stats report to see how Googlebot crawls the store.
• Duplicate product paths. Products can be reached at /products/handle and through collection paths such as /collections/shoes/products/handle. Your crawl will find both, so check that each duplicate's canonical points to the single URL you want indexed.
• Filter and sort URLs. Shopify's default robots.txt blocks collection URLs with sort_by, so a crawler that obeys robots.txt will skip them. Filtered URLs from apps may still be open. Our guide to faceted navigation SEO covers how to handle them.
• Your catalog is a URL list. A product export from the Shopify admin gives you every product handle, including drafts and archived items. Compare it to the crawl to catch products that are live but unlinked.
• Password pages. Shopify says a store in private mode blocks search engines from the sitemap. Make sure the password page is off before you crawl a live store.

Turn your crawl into a prioritized fix list

A crawl of a mid-sized store can flag thousands of issues. Most are the same few template problems repeated across many URLs, so group by cause before you count. Then fix in this order:

For broken URLs you've decided to redirect, our 404 Redirect Mapper matches each one to the closest live page and exports a redirect map you can import. If the crawl shows Google wasting time on parameter URLs, read our guide to crawl budget optimization next. And for the full list of checks beyond a crawl, use our technical SEO audit checklist.

How often to scan your site

Google's crawl budget guide is written for large sites with over 1 million pages, or over 10,000 pages that change daily, and says those numbers are rough estimates. Most stores are smaller than that, so crawl budget is rarely the issue. Broken templates, orphaned products and stray noindex tags are. A practical schedule:

• Monthly: a full crawl compared against last month's export, plus a look at the Page indexing report.
• After every theme change, migration, app install or menu change: a crawl of the affected templates before and after.
• Weekly for large catalogs: a sitemap check to confirm new products return 200 and are indexable.

Keep each export with its date so you can see when an issue first appeared. That history is often the fastest way to connect a traffic drop to the change that caused it. If you'd like help reading a crawl or turning it into a plan, our technical SEO team does this work every day, or you can tell us about your site.