llms.txt vs robots.txt: What Each File Does in 2026

robots.txt tells crawlers which URLs they may fetch, and compliant bots such as Googlebot, GPTBot and ClaudeBot follow it. llms.txt is a Markdown reading list that points AI tools to your best content, but it cannot allow or block anything, and in Ahrefs' study of 137,210 sites, 97% of llms.txt files got zero requests in a month.

This guide is for Shopify and e-commerce teams deciding how to handle AI crawlers. You'll get a side-by-side comparison of llms.txt vs robots.txt (and the sitemap), the current data on who actually reads each file, a bot-by-bot table of what blocking costs you, a setup order for Shopify, and the mistakes that quietly cut stores out of ChatGPT and Perplexity answers.

llms.txt vs robots.txt: the short answer

The difference comes down to one word: robots.txt restricts, llms.txt recommends. robots.txt is a formal standard that crawlers read before they fetch your pages. llms.txt is an optional proposal that a tool may read when it wants a quick map of your site.

• robots.txt sits at the root of your domain and lists rules per user agent: which paths a crawler may or may not fetch, plus optional sitemap URLs. It is the only one of the two files that changes crawler behavior.
• llms.txt is a Markdown file, usually at /llms.txt, with your site name, a short summary and curated lists of links. It describes. It never permits or forbids.
• sitemap.xml is the third file in the set: a complete list of the URLs you want search engines to discover.

If you only remember one thing: blocking an AI crawler is a robots.txt job, and no amount of llms.txt content will undo a Disallow rule.

What robots.txt controls (and what it doesn't)

robots.txt has been around since 1994 and became an official internet standard in September 2022 as RFC 9309, the Robots Exclusion Protocol. The file must live in the top-level directory of a host, such as https://yourstore.com/robots.txt, and its rules apply only to that host, protocol and port.

The rules it supports

Google's robots.txt documentation supports four fields: user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, are ignored by Google, though some crawlers (Anthropic says its bots do) support crawl-delay as an extension. A typical rule to keep a training crawler out looks like this:

User-agent: GPTBot
Disallow: /

Three limits every store owner should know

• It controls crawling, not indexing. Google says a disallowed URL can still be indexed and shown without a snippet if other pages link to it. Use a noindex tag (on a page Google can crawl) to keep a page out of results.
• It is not security. RFC 9309 states that the rules "are not a form of access authorization." Protect private areas with logins, not Disallow lines.
• It only works on bots that choose to comply. OpenAI, Anthropic, Google and Perplexity document that their crawlers respect it. Scrapers that don't care will ignore it.

How fast changes take effect

Google generally caches your robots.txt for up to 24 hours, and RFC 9309 says crawlers should not use a cached copy for longer than that unless the file is unreachable. OpenAI says it can take about 24 hours for its search systems to adjust after you update robots.txt, and Perplexity gives the same 24 hour window. Status codes matter too: Google treats most 4xx responses as "no robots.txt, crawl everything," while a 5xx error makes it pause crawling the site. Google also stops reading after 500 KiB, which almost no store will hit.

What llms.txt is and why it exists

llms.txt was proposed by Jeremy Howard on September 3, 2024, and the llmstxt.org spec was revised in 2026. The idea is simple: HTML pages full of navigation, scripts and pop-ups are a poor way for a language model to learn what a site offers, so you publish a clean Markdown summary with links to the pages that matter.

The format

1. An H1 with the name of the site. This is the only required part.
2. A blockquote with a short summary of what the site or brand is.
3. Optional paragraphs with more detail.
4. H2 sections, each a list of links in the form [name](url): notes.
5. By convention, an "Optional" H2 for secondary links a tool can skip when it has less room.

The current spec also lets you place llms.txt files at subpaths (for example /docs/llms.txt) and suggests offering clean Markdown versions of important pages at the same URL with .md added.

How the spec itself compares the two files

llmstxt.org is explicit that robots.txt and llms.txt have different purposes. robots.txt governs which automated access is acceptable. llms.txt is meant to be read on demand, when an AI tool or agent needs information about your site at the moment it is answering a question. The spec also notes that sitemaps list every indexable page, which is usually far more than fits in a model's context window, while llms.txt is a short, curated subset.

Who actually reads each file

robots.txt is read by essentially every legitimate crawler, because the standard tells them to check it before fetching anything. llms.txt is a different story. The best public data comes from Ahrefs' June 2026 study, which looked at 137,210 domains using Ahrefs Web Analytics that received traffic in May 2026.

28% of those domains published an llms.txt file, yet 97% of the files received no requests at all during the month. The 3% that were fetched received about 22,000 requests in total, and 96% of those came from bots rather than people.

The more telling part is which bots. Most were not AI tools. SEO audit tools were the biggest single group, and AI retrieval bots, the ones that fetch pages to cite in live answers, made up just 1.1% of requests. Ahrefs found that 77% of the bots fetching llms.txt were not AI tools at all. GPTBot was the most active named AI bot, at about 4.5% of requests, and OAI-SearchBot was the top retrieval bot at under 1%.

Two other signals point the same way. Google's AI optimization guide (updated July 10, 2026) says you don't need AI text files to appear in Google Search, and that creating them "will neither harm nor help" because Google Search ignores them. And none of the crawler documentation from OpenAI, Anthropic or Perplexity says their bots use llms.txt; the only mentions are links to their own documentation indexes.

That does not make llms.txt worthless. Coding assistants and agents do use these files on documentation sites, and the cost of a good file is low. It does mean llms.txt is a small, optional extra, while robots.txt decides whether AI crawlers see your store at all. We cover the full decision in do I need llms.txt.

Which AI crawlers to allow in robots.txt

This is the decision that actually affects AI visibility, and it is more nuanced than "block AI" or "allow AI." Each major AI company now runs separate bots for training, search indexing and user-triggered fetches, and you can treat each one differently.

Sources: OpenAI crawler docs, Anthropic's help center article on its crawlers, and Perplexity's bot docs. Perplexity says its user-triggered fetcher, Perplexity-User, generally ignores robots.txt for the same reason as ChatGPT-User. For a deeper look at the OpenAI pair, see GPTBot vs OAI-SearchBot.

What other sites are doing

Zyte's State of Web Access 2026 report checked 11,100 top landing pages, 8,419 of which had a valid robots.txt. 12.7% of sites named at least one AI crawler in a Disallow rule. Training crawlers were blocked far more often than search bots.

Zyte also found that search-linked bots had the most explicit Allow rules relative to blocks. OAI-SearchBot, for example, was disallowed by 2.9% of sites and explicitly allowed by 1.7%, while GPTBot was blocked five times as often as it was allowed. In other words, sites that think about this tend to block training and keep search.

A sensible default for most stores

For a store that sells products, being recommended in ChatGPT, Claude and Perplexity answers is free distribution. Our default for e-commerce clients is to allow the search and user-triggered bots, and make a deliberate business decision about training crawlers. Some brands block GPTBot and ClaudeBot to protect original content; others allow them on the theory that models trained on accurate product information describe the brand more accurately. There is no published evidence that settles which choice wins, so decide on your own priorities, then check the result with our robots.txt and AI crawler tester.

How llms.txt compares to your sitemap

People often confuse llms.txt with sitemap.xml because both are lists of links. They do different jobs.

• A sitemap is exhaustive. It lists every URL you want indexed: every product, collection, blog post and page. On Shopify it is generated automatically at /sitemap.xml and updates as you add products.
• llms.txt is selective. It is a short, human-written summary with a few dozen links at most, ordered by what matters, with a one-line note on each.
• A sitemap carries metadata for crawlers. Google may use an accurate lastmod date. llms.txt carries meaning for readers: what the page is for and when to use it.

A useful way to think about it: your sitemap is the full catalog, robots.txt marks the staff-only shelves, and llms.txt is the staff picks list by the register. You can check that your sitemap is valid and reachable with our XML sitemap checker, and robots.txt is the right place to point crawlers to it with a Sitemap: line.

How to set up both files on Shopify

Shopify generates both files for you, so the work is mostly reviewing and deciding, not writing from scratch. Here is the order we use.

1. Read your live robots.txt. Open yourstore.com/robots.txt. Shopify's default file is organized into groups of rules per user agent and works for most stores. Save a copy before you change anything.
2. Decide your AI bot policy. Use the table above. Write down which bots you allow and which you block, and why. Most stores should allow OAI-SearchBot, Claude-SearchBot and PerplexityBot.
3. Customize only if you need to. To change rules, create a robots.txt.liquid template in your theme, as described in Shopify's robots.txt docs. Build on the robots.default_groups object rather than replacing everything with plain text, which Shopify strongly advises against.
4. Compare before you publish. Shopify warns the rendered template doesn't always match its generated file exactly. Diff the new output against your saved copy and test key URLs (a product, a collection, the homepage) for each bot.
5. Check your llms.txt. Open yourstore.com/llms.txt. Shopify serves it as an alternate URL for /agents.md, its managed agent-discovery file, on your primary domain.
6. Keep Shopify's default unless you have a reason. Shopify's llms.txt template docs recommend the managed default for most stores. To customize agent content, edit agents.md.liquid, which covers /agents.md, /llms.txt and /llms-full.txt together. Add a separate llms.txt.liquid only if /llms.txt must differ.
7. Recheck in 24 to 48 hours. Allow time for crawlers to pick up the new robots.txt, then confirm with a tester that the bots you want can reach your key pages.

Common mistakes with llms.txt and robots.txt

Most problems we see come from treating the two files as one system and assuming one can fix the other.

A note on pointing to llms.txt from robots.txt

Some guides suggest adding a line to robots.txt that points to llms.txt. There is no such directive in RFC 9309, and Google ignores fields it doesn't support. A comment line (starting with #) is harmless, but we have found no evidence from any AI company that it helps discovery. Ahrefs found that of the requests for llms.txt files that didn't exist, none came from AI bots, which suggests the bots that matter are not hunting for the file.

What to do this week

Treat robots.txt as the decision that matters and llms.txt as housekeeping. Open your robots.txt, confirm the AI search bots you want are allowed, and decide on training crawlers on purpose rather than by accident. Then leave Shopify's managed llms.txt in place, or make it accurate if you customize it, and check how ready your pages are to be quoted with our AEO checker.

The bigger lever for AI visibility is still the content those bots find: clear product and collection pages, answers to real buyer questions and structured data. If you want help with that side, our AI search optimization and technical SEO teams do this work for Shopify brands every week, or you can tell us about your store.