Should You Block GPTBot? Robots.txt Guide for Stores

Block GPTBot if you don't want OpenAI to use your content to train its models. Blocking it will not remove you from ChatGPT search, because OpenAI crawls for search with a separate bot, OAI-SearchBot, which you can keep allowed. For most e-commerce brands, the right setup is a deliberate choice on GPTBot and an explicit allow for OAI-SearchBot, so shoppers can still find your products in ChatGPT.

This guide is for Shopify and e-commerce teams deciding what to do about AI crawlers. You'll get what each OpenAI bot does, how much GPTBot actually crawls, a decision table by business type, copy and paste robots.txt rules (including for Shopify), the equivalent controls at Anthropic, Google and Perplexity, and a checklist to confirm your policy is live.

Should you block GPTBot? The short answer

Blocking GPTBot is a policy decision about training, not a traffic decision. Disallowing it tells OpenAI your content should not be used to train its generative AI foundation models. It does not affect whether ChatGPT search can find, cite or link to your pages, because that is OAI-SearchBot's job.

• Block GPTBot if you sell or license original content, have contracts that restrict reuse, or your leadership has not agreed on AI training yet.
• Allow GPTBot if your public pages are marketing and product information you are happy for AI models to learn from.
• Either way, allow OAI-SearchBot if you want to appear in ChatGPT search answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Which OpenAI bot are you blocking?

OpenAI's crawler documentation lists four user agents, each with its own job. A blanket "block OpenAI" rule treats them as one, which is where most mistakes start.

OpenAI states that "each setting is independent of the others," and gives the example of allowing OAI-SearchBot while disallowing GPTBot. It also notes that if you allow both, it may use one crawl for both purposes to avoid duplicate crawling, so separate policies do not always mean separate requests in your logs. For a deeper comparison of the two main bots, see GPTBot vs OAI-SearchBot.

How much does GPTBot actually crawl?

Enough to show up in your server logs. Cloudflare's analysis of traffic across its network found that GPTBot's share of AI and search crawler traffic grew from 2.2% in May 2024 to 7.7% in May 2025, with raw requests up 305%. Among crawlers that serve AI only, GPTBot went from 5% to 30% of traffic and became the largest.

Across all of 2025, Cloudflare's Radar Year in Review says GPTBot originated about 7.5% of verified bot traffic. Cloudflare also found that crawling for AI training made up the large majority of AI crawler traffic, peaking at 7 to 8 times the volume of AI search crawling.

The other side of the ledger is referrals. Cloudflare tracks a crawl-to-refer ratio, the number of pages a platform crawls for every visit it sends back. In 2025, OpenAI's ratio was spiky and peaked at 3,700 to 1 in March, while Google Search ranged from about 3 to 1 up to 30 to 1. That gap is a big reason many publishers decided training crawls were not worth the load, even while they kept search crawlers allowed.

Many large sites have already made a call. When Cloudflare checked the top 10,000 domains in June 2025, GPTBot was both the most disallowed and the most explicitly allowed bot.

When to block GPTBot, and when to allow it

Start with what your content is and what you want from AI platforms. A Shopify store's product pages, a brand's original research and a member area should not share one blanket rule.

Whatever you choose, make it on purpose and write down why. Don't allow GPTBot because someone promised training data will turn into citations or sales; OpenAI's documentation makes no such link. And don't block it expecting a traffic gain, since GPTBot is not the bot behind ChatGPT search.

For most e-commerce brands we work with, the practical middle ground is to decide GPTBot based on how you feel about training, and keep OAI-SearchBot allowed. Product discovery in AI answers is a real channel, and our guide on how to appear in ChatGPT covers what earns those mentions once access is in place.

How to block GPTBot in robots.txt

For the common "block training, allow search" policy, add these two groups to your existing robots.txt file:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

To block training on one section only, such as a research hub or recipe library, use a path rule instead:

User-agent: GPTBot
Disallow: /blogs/research/

Follow these steps so the change does what you expect:

1. Save a copy of the current file. Note the date, the reason for the change and the hosts it affects, so you can roll back.
2. Merge, do not replace. Keep existing rules for other crawlers and your sitemap line. Pasting a new file over the old one is the most common way to accidentally unblock checkout or cart URLs.
3. Repeat path rules in named groups. Under the Robots Exclusion Protocol, RFC 9309, a crawler follows the most specific group that matches its name and ignores the User-agent: * group. If your wildcard group blocks /checkout and you add User-agent: OAI-SearchBot with Allow: /, OAI-SearchBot may now crawl /checkout unless you repeat that rule in its group.
4. Cover every host. The file must be served as plain text at /robots.txt on each host and subdomain you want covered, such as www and shop.
5. Deploy and fetch the live file. Open yourdomain.com/robots.txt in a private window and confirm your new groups are there.

Blocking GPTBot on Shopify

Shopify generates robots.txt for you, so you can't upload a file. To change it, add a robots.txt.liquid template to your theme (Online Store, Themes, Edit code, Add a new template, robots). Keep Shopify's default rules by leaving the default loop in place, then add your GPTBot and OAI-SearchBot groups after it. Shopify's defaults already block cart, checkout and account paths for all crawlers, and you want to keep them.

Because a named group ignores the wildcard rules, consider repeating the key Shopify disallows (such as /cart/, /checkout and /account) inside your OAI-SearchBot group. Our guide to robots.txt best practices covers syntax in more depth.

Should you block other AI crawlers too?

A GPTBot rule only covers OpenAI. If your goal is a consistent training policy, set rules for the other major providers. Each documents its own controls, and the names do not line up neatly.

Sources: Anthropic's crawler help article, Google's common crawlers documentation and Perplexity's crawler guide. Bot names change, so recheck these pages each time you review your policy.

Blocking Google-Extended is the one that worries store owners most, and Google's documentation is clear that it does not change how you rank in Google Search. It is a separate control from Googlebot.

How to verify your GPTBot block is live

A saved edit is not proof the policy works. Run these checks after every change:

1. Test each bot against the live file. Paste your live robots.txt into our robots.txt and AI crawler tester and check GPTBot and OAI-SearchBot separately, on a product URL, a collection URL and a blog URL. The tester evaluates the rules you give it. It does not contact OpenAI or prove a crawler visited.
2. Check your CDN and firewall. Bot-management settings can block or challenge OAI-SearchBot even when robots.txt allows it. Cloudflare, for example, now lets you set separate policies for AI training, search and agent bots. Make sure a default or a broad rule is not blocking the bot you meant to allow, and scope any exception narrowly.
3. Review verified requests in your logs. Look at paths, status codes and source IPs. Match IPs against OpenAI's published ranges at openai.com/gptbot.json and openai.com/searchbot.json, because user-agent strings are easy to fake.
4. Allow time. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. It does not publish a timing for GPTBot.
5. Recheck after theme or platform changes. A new Shopify theme, a migration or a CDN change can silently replace your rules.

Our guide on how to check if ChatGPT can crawl your website walks through each check in more detail.

What blocking GPTBot does not do

• It is not a deletion request. OpenAI documents a GPTBot disallow as a signal that content should not be used in training. It does not promise to remove content already collected or reverse past training.
• It does not cover other companies. Other AI providers, scrapers and sites that copy your content follow their own rules, if any.
• It is not security. RFC 9309 says robots.txt rules are not a form of access authorization, and listing a private path can even advertise it. Protect private content with logins and server-side access controls.
• It does not stop ChatGPT-User. When a user asks ChatGPT to open a page, OpenAI says robots.txt rules may not apply.
• It is unrelated to llms.txt. The llms.txt proposal is an optional guide for AI agents. It does not allow or block anything. See do you need an llms.txt file for more.

Measure ChatGPT visibility, not just access

Treat crawler rules as a policy, not a performance metric. Allowing OAI-SearchBot makes you eligible for ChatGPT search; it does not guarantee you get cited or recommended. To see whether ChatGPT is working for you:

• Track referrals. In GA4, look for sessions and purchases from chatgpt.com. Links from ChatGPT answers often carry a utm_source=chatgpt.com parameter.
• Log a fixed set of prompts. Keep a dated record of 20 or so shopper questions in your category, the answers ChatGPT gives and which URLs it cites. Answers vary between runs, so look for patterns over weeks.
• Check citation readiness. Once access is right, run key product, collection and guide pages through our AEO checker to see whether they answer questions clearly enough to quote.

Decide on training once, keep search open, verify the rules are live and then put your effort into pages worth citing. If you want a second pair of eyes on your crawler policy or your AI visibility, our ChatGPT SEO team can help, or you can contact us.