Robots.txt Best Practices for SEO and AI Crawlers

Robots.txt best practices come down to three rules: block only the URLs that waste crawling (cart, checkout, account, internal search, sort and filter combinations), never block pages or resources you want ranked, and use noindex, not Disallow, when you want a page out of search results. Then decide on purpose which AI crawlers may read your store, and test every change before it goes live, because one bad line can stop Google from crawling your whole site.

This guide is for Shopify and e-commerce teams who own their technical SEO. You'll get the rules Google and RFC 9309 actually follow, a block and allow table for a typical store, copy-ready robots.txt examples, a plain map of the AI crawlers, the mistakes that cost traffic, and a step-by-step way to ship changes safely.

What robots.txt does and what it can't do

A robots.txt file is a plain text file at the root of your domain (for example https://example.com/robots.txt) that tells crawlers which URLs they may request. Google describes its main job as managing crawler traffic so bots don't overload your site. It is the first file most well-behaved crawlers fetch, and it only covers the exact host, protocol and port it sits on, so shop.example.com and example.com each need their own file.

Robots.txt controls crawling, not indexing

This is the distinction that trips up most stores. Google's robots.txt introduction says the file is not a mechanism for keeping a web page out of Google. A disallowed URL can still be indexed if other pages link to it, and it can appear in results without a description. To keep a page out of search, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header. Google's noindex documentation is explicit that the rule only works if the page is not blocked by robots.txt, because a blocked crawler never sees the tag.

Robots.txt is not security

RFC 9309, the IETF standard for the Robots Exclusion Protocol published in September 2022, states that robots.txt rules are not a form of access authorization. Compliance is voluntary, and anyone can read your file. Listing a private folder in robots.txt can actually advertise it. Protect staging sites, customer data and wholesale price lists with a login, not a Disallow line.

Robots.txt syntax and the rules Google follows

A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by the Allow and Disallow rules for those crawlers. Sitemap lines can go anywhere. Here is a minimal, valid file for a store:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://example.com/sitemap.xml

How matching works

• Longest match wins. When an Allow and a Disallow rule both match a URL, Google uses the most specific rule, meaning the longest path. If they are equally long, the least restrictive rule (Allow) wins. RFC 9309 says the same: the match with the most octets wins, and Allow breaks ties.
• Paths are case sensitive. Disallow: /Cart does not block /cart. Field names and user agent values are not case sensitive.
• Two wildcards. * matches any sequence of characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf, and Disallow: /*?q= blocks any URL containing ?q=.
• Only one group applies. Google follows the group with the most specific matching user agent and ignores the rest. If you add a User-agent: Googlebot group, Googlebot stops reading your User-agent: * rules entirely, so repeat any rules you still need. Multiple groups for the same crawler are combined.

The limits that matter

Google reads only the first 500 KiB of the file, caches it for up to 24 hours (longer if it can't refetch), and follows at least five redirect hops before treating the file as missing. Server errors are the dangerous case: if robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours while it retries, then falls back to the last cached copy for up to 30 days. A 404 or other 4xx status (except 429) is treated as if no robots.txt exists, which means no restrictions at all.

Robots.txt best practices for e-commerce stores

Most stores don't need a long file. They need a short one that blocks the right things and nothing else. These are the practices we'd apply to any Shopify or e-commerce store:

1. Serve one file per host, at the root, with a 200 status. Check www, the bare domain and any market or language subdomains separately.
2. Block crawl traps, not content. Cart, checkout, account, order pages, internal search results and endless sort or filter combinations are the usual candidates. Products, collections, blog posts and pages should stay open.
3. Never block CSS, JavaScript or image files your pages need to render. Google says you can block non-essential resources only if the page still makes sense without them.
4. Use noindex for pages that should be crawlable but not indexed, such as thin tag pages, and leave them unblocked so Google can see the tag.
5. List your sitemap index with a full absolute URL in a Sitemap line.
6. Keep rules simple and documented. Add # comments explaining why each custom rule exists and who added it.
7. Treat specific user agent groups carefully, because they replace your * rules for that crawler instead of adding to them.
8. Don't use robots.txt to shift crawl budget temporarily. Google's crawl budget guide says it won't move freed-up budget to other pages unless your site is already hitting its crawl capacity limit.
9. Make an explicit AI crawler decision instead of inheriting whatever your platform ships.
10. Test every change before and after it goes live.

How big sites actually use robots.txt

The HTTP Archive's 2025 Web Almanac SEO chapter, published January 15, 2026, gives a picture of real files across millions of sites. About 85% of robots.txt requests returned a 200 status, up from 84% in 2024, and 13% returned a 404. A 404 is not an error for crawling (Google treats it as "crawl everything"), but it also means you have no sitemap line and no control over AI crawlers.

What to block and allow on a Shopify store

Shopify generates a robots.txt file for every store. We checked the live files of four large Shopify stores on October 27, 2026 (Allbirds, Kith, Gymshark and Ruggable). All four disallowed cart, checkout and account paths and collection URLs containing sort_by. Two of the four also disallowed /search, and none had a group for GPTBot or any other AI crawler. Two files contained Shopify's newer rule that blocks collection URLs combining two or more filters (/collections/*filter*&*filter*). Shopify does not publish the full default file in its docs, so check your own /robots.txt before you change anything.

How to edit robots.txt on Shopify

Shopify's developer docs explain that you customize the file by adding a robots.txt.liquid template to your theme's Templates folder. Adding the template replaces the generated file, so Shopify recommends keeping its Liquid objects (robots.default_groups, group.rules, group.sitemap) so you still inherit future default rules. This pattern, based on Shopify's documented example, keeps every default and adds one rule for all crawlers:

{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /*?q=*' }}
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}

New user agent groups, such as an AI crawler rule, go as plain text after the loop. Shopify's docs also warn that the rendered output doesn't always match the generated file exactly, so save a copy of your current /robots.txt first and compare the two after you publish.

How to manage AI crawlers in robots.txt

AI companies now run several crawlers each, and they do different jobs. Blocking a training crawler keeps your content out of future model training. Blocking a search crawler can remove you from AI answers and the referral traffic they send. You can make these choices independently. OpenAI's documentation says each of its settings is independent of the others, and that it can take about 24 hours for its search systems to reflect a robots.txt change.

For most stores, the commercial case is to stay visible in AI search while deciding separately about training. This example opts out of OpenAI and Anthropic training while keeping both companies' search crawlers welcome:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account

Notice the last group repeats the cart, checkout and account rules. Because a named group replaces the * group for that crawler, leaving them out would open those paths to the search bots. Our guide on whether you should block GPTBot walks through the trade-off in detail, and GPTBot vs OAI-SearchBot explains the two OpenAI crawlers.

How many sites name AI crawlers

Explicit AI rules are still a minority practice. In the 2025 Web Almanac, GPTBot was the most-named AI crawler, appearing in 4.5% of desktop robots.txt files, while the catch-all * group appeared in about 77%. That means most sites, and most Shopify stores using the default file, let AI crawlers in by default without ever deciding to.

Robots.txt is also not the same as llms.txt. Robots.txt sets crawl permissions that major crawlers honor. llms.txt is a proposed file that lists your important pages for AI tools, and Google says it ignores it. See llms.txt vs robots.txt for what each file does.

Common robots.txt mistakes that cost traffic

Most robots.txt damage comes from a handful of repeat errors. Use this grid when traffic drops after a theme change, migration or app install.

Two more to watch. Rules are case sensitive, so a rule written for /Collections/ won't match Shopify's lowercase paths. And remember the 500 KiB limit: the Web Almanac found very few files near it, but auto-generated files that list thousands of individual URLs can get there, and anything past the limit is ignored.

How to test and roll out robots.txt changes

Search Console's robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, when it last crawled them, and any warnings or errors. It also lets you request a recrawl of the file in an emergency. Use it together with a tester that checks specific URLs against specific user agents. Follow this sequence for every change:

1. Save the current file. Copy /robots.txt from every host into your change log with today's date.
2. List the URLs that must stay crawlable. Include your homepage, top collections, top products, blog posts and a CSS and JavaScript file from your theme.
3. List the URLs you mean to block. Add real examples of each pattern, such as a sorted collection URL and a search URL.
4. Test the draft. Paste it into our Robots.txt and AI Crawler Tester and check both lists against Googlebot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot.
5. Publish. On Shopify, edit robots.txt.liquid; on other platforms, upload the file to the root of each host.
6. Read the live file. Open /robots.txt in a browser, confirm a 200 status and check that every rule sits on its own line.
7. Check Search Console. Open the robots.txt report, request a recrawl of the file if the change is urgent (the report works only for domain-level properties), and use URL Inspection on a few key pages.
8. Watch for a week. Compare the Crawl Stats report and the Page indexing report against the week before the change.

It also helps to confirm that your sitemap and robots.txt agree. Every URL in your sitemap should be crawlable; a sitemap URL that robots.txt blocks is a mixed signal. Our XML Sitemap Checker fetches your sitemap and spot-checks that its URLs return 200 and are indexable.

A robots.txt routine you can keep

Robots.txt rarely needs weekly attention, but it needs an owner. Review it whenever you change themes, migrate platforms, add a market or subdomain, install an app that creates new URL patterns, or change your stance on AI crawlers. Once a quarter, open the live file, compare it to your last saved copy, and run your must-crawl list through a tester.

If the file and your crawl data disagree, the problem is usually bigger than robots.txt: duplicate URLs from filters, redirect chains or orphaned products. Our guides to crawl budget optimization and faceted navigation SEO cover those fixes. If you'd rather have someone audit the whole crawl path for you, our technical SEO team can help, or you can tell us about your store.