Log File Analysis for SEO: Googlebot, AI Bots, Shopify

Log file analysis for SEO means reading your web server's access logs to see exactly which URLs Googlebot and other crawlers request, how often, and what status codes they get back. It is the only first-hand record of crawler behavior, so it reveals crawl waste, errors and ignored pages that sampled reports can miss.

This guide is for technical SEOs and e-commerce teams. You will learn what a log line contains, how to verify Googlebot and AI crawlers such as GPTBot and ClaudeBot, a step-by-step analysis you can run with free command-line tools, what to fix, and what to do on Shopify, which does not give merchants raw server logs.

What is log file analysis in SEO?

Every time a browser or bot requests a file from your site, the server (or the CDN in front of it) writes a line to an access log. Log file analysis filters those lines to search engine and AI crawlers, then groups them by URL, status code, response time and date. The result answers questions no crawler tool can answer on its own:

• Which pages does Googlebot actually crawl, and which does it ignore?
• How much crawling goes to filter, sort and parameter URLs that should never rank?
• Do bots hit 404s, redirect chains or 5xx errors you did not know about?
• How quickly does Google come back to new or updated product pages?
• How much traffic comes from AI crawlers, and are they real?

Logs matter most on large or fast-changing sites. Google's crawl budget guide is written for sites with more than a million pages that change weekly, sites with 10,000 or more pages that change daily, and sites with many URLs stuck in "Discovered, currently not indexed." Google adds that these are not exact thresholds. A catalog with faceted navigation can generate that many URLs quickly. For the theory behind crawl budget, see our guide to crawl budget optimization.

What a server log line tells you

Most servers write the Combined Log Format or something close to it. Here is a single Googlebot request, with Google's smartphone user agent:

66.249.66.1 - - [27/Oct/2026:08:14:03 +0000] "GET /products/merino-runner?variant=4211 HTTP/1.1" 200 51234 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Some servers and CDNs also log response time, which is worth adding if you can. Google's crawl capacity limit falls when your server slows down or returns 5xx or 429 responses, so slow responses to bots are a crawl problem, not just a user experience problem.

Can you get server logs on Shopify?

No, not raw ones. Shopify is a hosted platform and does not provide merchants with access to its web server logs. Merchants and agencies regularly ask for them in the Shopify Community, and the answer is consistently that they are not available, including on Shopify Plus. Apps and Shopify Analytics track human sessions, not bot requests.

That does not mean you are blind. Use these alternatives, roughly in order of usefulness:

The Cloudflare route is real but not trivial. Cloudflare documents an orange-to-orange setup for Shopify, where your own Cloudflare zone proxies the store before Shopify's: a proxied CNAME to shops.myshopify.com, with a warning not to enable Always Use HTTPS because it breaks Shopify's certificate renewals. Which log datasets you can export depends on the products on your Cloudflare account, so check before you plan around it.

Getting the most from Crawl Stats

Open Search Console, go to Settings, then Crawl stats. The report is available only for Domain properties and root-level URL-prefix properties. Look at three things:

• By response. A rising share of 404s or 301s means bots are following stale links. Click through to see example URLs.
• By purpose. Discovery is crawling of new URLs, refresh is recrawling known ones. A spike in discovery after a theme or app change often means new parameter URLs have appeared.
• Host status. Google checks robots.txt fetching, DNS resolution and server connectivity over the last 90 days. A 429 or 5xx on robots.txt counts as a failure. Google gives one threshold: DNS failing on more than 5% of requests on a given day is an issue.

How to verify Googlebot and AI crawlers

Anyone can put "Googlebot" in a user agent. Scrapers do it all the time to slip past bot rules. Before you draw conclusions, verify the IPs.

Verify Googlebot

Google's verification guide gives two methods. The manual one is a reverse DNS lookup on the IP, a check that the name ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup on that name to confirm it returns the same IP:

host 66.249.66.1
# returns a name like crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# must return 66.249.66.1 again

For bulk checks, match IPs against Google's published JSON range files: common-crawlers.json (Googlebot), special-crawlers.json (such as AdsBot), and the user-triggered fetcher and agent files, all under developers.google.com/static/crawling/ipranges/. Common crawlers reverse-resolve to names like crawl-***-***-***-***.googlebot.com.

Verify AI crawlers

OpenAI, Anthropic and Perplexity each publish user agents and IP lists. They also run separate bots for training, search indexing and user-triggered fetches, and those behave differently.

Sources: OpenAI crawler docs, Anthropic's crawler help article and Perplexity's bot docs. Anthropic advises against blocking its IPs as an opt-out, because that stops its bots from reading your robots.txt. For the difference between OpenAI's bots, see GPTBot vs OAI-SearchBot.

How to run a log file analysis step by step

You do not need expensive software to start. A month of logs and a terminal will answer most questions; dedicated log analyzers help once files run into gigabytes.

1. Get at least 30 days of logs. Ask your host, developer or CDN provider for access logs covering every hostname (www, bare domain, subdomains). Make sure they are not already filtered to remove bots.
2. Filter to crawlers. Pull lines whose user agent contains Googlebot, bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
3. Verify the IPs. Match each IP against the owners' published lists, or run DNS checks on a sample. Drop anything that fails. Scrapers often fake bot user agents, so unverified lines can skew every count that follows.
4. Group by status code. Count 200, 301, 302, 304, 404, 410, 429 and 5xx responses for each bot.
5. Group by URL pattern. Bucket URLs into products, collections, blog, filters and parameters, search, cart and account, and assets. This shows where crawl activity goes.
6. Join with your own URL list. Compare crawled URLs with your sitemap and a site crawl. Pages in your sitemap that bots never request need better internal links; URLs bots request that are not in your sitemap may be orphans or traps.
7. Check freshness. For your top products and newest pages, find the last Googlebot visit. Long gaps on important pages are a priority.
8. Fix, then repeat. Make changes, wait a few weeks and pull a new sample to confirm the shift.

These one-liners work on a Combined Log Format file, where field 7 is the URL and field 9 is the status code. Run them only after verifying the IPs:

# Googlebot requests by status code
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# The 50 most crawled URLs
grep "Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50

# Share of Googlebot hits on parameter URLs
grep "Googlebot" access.log | awk '$7 ~ /\?/' | wc -l

What to look for in your logs

Crawl waste on low-value URLs

In e-commerce, the usual offenders are faceted filters, sort orders, internal search results, tag pages and tracking parameters. If a large share of verified Googlebot requests lands on URLs you never want indexed, your important pages compete for attention. Google's guide says noindex does not save crawl, because Google still has to request the page to see the tag. Use robots.txt for URLs that should never be crawled, and fix the internal links that create them.

Errors and redirects

Google says a 404 is a strong signal not to crawl a URL again, so occasional 404s are normal. Soft 404s, which return 200 for an empty page, waste crawl because Google keeps coming back. Redirect chains force extra requests. And 5xx or 429 responses directly reduce how much Google is willing to crawl. AI crawlers make broken URLs more expensive still: in Vercel's network data, roughly a third of ChatGPT and Claude crawler fetches hit 404 pages.

Pages bots never reach

Products that appear in your sitemap but never in your logs usually have weak internal links, sit deep in pagination or are only reachable through JavaScript. Vercel's same study found that none of the major AI crawlers rendered JavaScript at the time, so content that only appears after scripts run may be invisible to them even when Googlebot sees it.

AI crawlers in your server logs

AI crawlers now show up in most logs, and the volumes can be large. Vercel published one month of crawler requests across its network in December 2024. Googlebot still led by a wide margin, but OpenAI's and Anthropic's crawlers together made close to a billion requests.

The bigger question is what you get back. Cloudflare's Radar 2025 Year in Review, covering January 1 to December 2, 2025, measured crawl-to-refer ratios: pages crawled for every visit sent back. Anthropic's peaked around 500,000 to 1, OpenAI's peaked at 3,700 to 1 in March, Perplexity's stayed below 200 to 1 from September, and Google's started the year just over 3 to 1. The same report put AI bots at 4.2% of HTML requests on average, slightly behind Googlebot at 4.5%.

Use your own logs to decide per bot. Search and user-triggered bots such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User fetch pages that can be cited in answers, so most stores want them allowed. Training crawlers such as GPTBot and ClaudeBot are a business choice. Our post on whether to block GPTBot walks through the trade-offs, and the AI Crawler Tester shows what your current robots.txt allows.

Turning log findings into fixes

A log analysis is only useful if it changes something. Map each finding to a fix, then re-sample to confirm.

Our Redirect Checker traces chains hop by hop, and the 404 Redirect Helper maps broken URLs to replacements. For a broader checklist, see our technical SEO audit checklist.

Where to start this week

If you have server or CDN logs, pull 30 days, verify Googlebot, and run the status code and top URL counts above. That alone usually surfaces the biggest crawl leak. If you are on Shopify, open Crawl Stats, check the response and purpose breakdowns and host status, test your robots.txt and sitemap, and decide whether an orange-to-orange Cloudflare setup is worth it for edge data.

Log work pays off most on large catalogs and during migrations. If you want help reading the data or fixing what it shows, our technical SEO team can help, or get in touch for a free plan.