GPTBot vs OAI-SearchBot: Training vs Search Crawlers

GPTBot vs OAI-SearchBot comes down to one difference: GPTBot crawls content that may be used to train OpenAI's models, while OAI-SearchBot crawls so your pages can appear in ChatGPT search answers. You control each one separately in robots.txt, so you can block training and still stay visible in ChatGPT search.

This guide is for e-commerce and Shopify teams deciding what to do with OpenAI's crawlers. You'll get a side-by-side comparison of all four OpenAI user agents, what Cloudflare's network data says about how much they crawl, a decision table for the common setups, copy-ready robots.txt rules (including the Shopify way to add them), and a checklist for confirming in your logs that the right bots are actually getting through.

GPTBot vs OAI-SearchBot at a glance

OpenAI's crawler documentation lists four user agents, each with its own job and its own robots.txt setting. OpenAI says each setting is independent of the others, which is what makes a "block training, allow search" setup possible.

What is GPTBot?

GPTBot is OpenAI's training crawler. OpenAI describes it as crawling content that may be used to train its generative AI foundation models, and says disallowing it signals that a site's content should not be used for training. It identifies itself with a user-agent string containing GPTBot and a link to openai.com/gptbot. OpenAI shows a version number in its example string and notes that the version may change, so match on the token, not the full string.

A few things GPTBot is not:

• It is not a search indexer. It does not decide whether you show up in ChatGPT search results.
• It is not the bot that fetches a page when someone asks ChatGPT a question. That is ChatGPT-User.
• It does not affect your Google rankings. Google ranks pages with its own crawlers and systems.

OpenAI does not publish a crawl schedule, frequency or volume for GPTBot. Treat any blog post that tells you how often it visits, or that it crawls in "training cycles," as a guess. Your own server or CDN logs are the only reliable record for your site.

What blocking GPTBot does and does not do

Blocking GPTBot is a forward-looking signal. It tells OpenAI not to use what it crawls from your site for training from now on. OpenAI's documentation does not say that blocking GPTBot removes content from models that were already trained, so don't expect it to change what ChatGPT already knows about your brand. It also has no documented effect on ChatGPT search, which runs on OAI-SearchBot.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's search crawler. It crawls the web so pages can be surfaced and linked in ChatGPT's search features. OpenAI recommends allowing it, and allowing requests from its published IP ranges, if you want your pages to appear in ChatGPT search answers. Its user-agent string contains OAI-SearchBot and a link to openai.com/searchbot.

A common mistake, repeated in many older articles, is that OAI-SearchBot fetches pages live each time a user asks a question. OpenAI's documentation does not describe it that way. OAI-SearchBot is the crawler that makes your pages eligible for search. When a user asks ChatGPT to open a specific page, that visit comes from ChatGPT-User instead.

Three details from OpenAI's docs matter for store owners:

• Changes are not instant. OpenAI says robots.txt changes can take about 24 hours to affect search.
• Opting out is not total invisibility. Sites that block OAI-SearchBot won't appear in ChatGPT search answers, but they may still show as navigational links.
• Eligible is not the same as cited. Allowing the bot makes your pages eligible. Whether ChatGPT quotes or links your product page depends on the page itself.

Can one crawl count for both bots?

Yes. OpenAI says that if a site allows both OAI-SearchBot and GPTBot, it may use the results of a single crawl for both purposes to avoid crawling twice. So separate rules do not always mean separate requests in your logs. If you allow both, you might see fewer requests than you expect from one of them. If you block GPTBot and allow OAI-SearchBot, you should expect search crawling without training use.

ChatGPT-User and OAI-AdsBot explained

Most comparisons stop at two bots. Two more OpenAI user agents can show up in your logs, and confusing them with OAI-SearchBot leads to bad decisions.

ChatGPT-User: visits a person asked for

ChatGPT-User visits pages when a user asks ChatGPT or a Custom GPT a question, or uses GPT Actions. OpenAI says it is not used for automatic crawling, and that robots.txt rules may not apply because a user triggered the visit. OpenAI also says ChatGPT-User does not determine whether you appear in search. That is OAI-SearchBot's job.

For a store, ChatGPT-User traffic usually means a shopper or researcher pasted or clicked a link to your page inside ChatGPT. Blocking it at your firewall would stop ChatGPT from reading the page that person asked about, which is rarely what a brand wants.

OAI-AdsBot: only for ChatGPT ads

OAI-AdsBot checks landing pages submitted as ads on ChatGPT for policy compliance, and may use their content to decide when to show an ad. OpenAI says the data it collects is not used to train generative AI foundation models. If you don't advertise on ChatGPT, you can ignore it.

The robots.txt marker

OpenAI says its crawlers may add a "robots.txt" marker to their user-agent strings when they fetch your robots.txt file. That lets you tell rule checks apart from page crawls in your logs, which is useful when you're confirming that a new rule was picked up.

How much do OpenAI's bots crawl?

OpenAI doesn't publish crawl volumes, but Cloudflare, which sits in front of a large share of websites, does publish what it sees. In its 2025 crawler report, GPTBot became the most active AI crawler on Cloudflare's network, rising from 5% of AI crawler traffic in May 2024 to 30% in May 2025.

A few more figures from the same Cloudflare report put that in context:

• Across a wider set of AI and search crawlers, GPTBot's share rose from 2.2% to 7.7%, and its raw requests grew 305%. Googlebot still led at 50%.
• ChatGPT-User grew from 0.1% to 1.3% of that combined crawler traffic, with raw requests up 2,825%. That is the user-triggered agent, so the growth reflects more people using ChatGPT to look at web pages.
• Cloudflare did not report a separate traffic share for OAI-SearchBot.

The other side of the ledger is referrals. In an August 2025 analysis, Cloudflare reported that training accounted for nearly 80% of AI bot crawling in July 2025. For the first week of August 2025, it measured OpenAI's crawl-to-refer ratio at 887 to 1, meaning roughly 887 crawls for every visit sent back. Perplexity's ratio was 118 to 1 and Anthropic's was nearly 50,000 to 1.

What this means for you: most OpenAI crawling is for training, and training crawls send no visitors. Search crawling is the part tied to being cited and clicked. That is the economic case for treating the two bots differently. Keep in mind that Cloudflare's numbers describe its own network, not the whole web, and they are 2025 snapshots.

Should you block GPTBot, OAI-SearchBot or both?

For most e-commerce brands, the answer is to allow OAI-SearchBot, because ChatGPT search is where shoppers ask for product recommendations, and decide on GPTBot based on how you feel about training. Here is how the four realistic setups compare:

You won't be alone in whichever you choose. Cloudflare checked robots.txt files across the top 10,000 domains in June 2025. Of the 3,816 domains with a robots.txt file, 546 had rules naming AI bots, and GPTBot was the most blocked bot of all.

Those are mostly large publishers protecting their articles. A store's situation is different. Product pages, collection pages and buying guides exist to be found, and the content on them (prices, specs, sizing, policies) is the kind of thing you want ChatGPT to repeat accurately. For a deeper look at the training side of the decision, read should you block GPTBot.

How to set robots.txt for OpenAI's bots

The most common setup for brands that want to opt out of training is "block GPTBot, allow OAI-SearchBot." These are the groups to add to your existing robots.txt file:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /checkout
Disallow: /cart
Disallow: /account

The path rules in the OAI-SearchBot group are examples. Copy whatever your wildcard group already blocks.

1. Save your current file. Open yourdomain.com/robots.txt and keep a copy, so you can compare and roll back.
2. Decide your setup from the table above. If you want to allow both bots, you don't need any OpenAI-specific groups at all: a bot that isn't named falls under your User-agent: * rules.
3. Add named groups for GPTBot and OAI-SearchBot. Keep the bot names exactly as OpenAI writes them.
4. Repeat your path rules inside each named group (see the warning below).
5. On Shopify, use the theme template. In the theme code editor, create templates/robots.txt.liquid. Keep Shopify's default groups by looping over robots.default_groups, and add your OpenAI groups outside that loop. Shopify's robots.txt docs recommend using the provided Liquid objects and comparing the rendered output with your saved copy before publishing.
6. Test the live file. Paste it into our robots.txt and AI crawler tester and check how GPTBot, OAI-SearchBot and ChatGPT-User are treated on a product URL, a collection URL and your checkout.
7. Wait, then confirm in logs. Give it about a day, which is OpenAI's stated time for search changes, then check your logs as described in the next section.

What Shopify's default file does

Shopify generates a robots.txt for every store. In the live files we checked on Shopify stores in October 2026, the default had no groups that name OpenAI's bots, which means GPTBot and OAI-SearchBot both fall under the wildcard rules and are allowed on product and collection pages. If you're happy allowing both, you don't need to edit anything.

How to verify OpenAI bots in your logs

A permissive robots.txt does not guarantee access. CDN bot management, web application firewalls and rate limits can block or challenge OAI-SearchBot before it ever reads your file. And because user-agent strings are easy to fake, anything claiming to be GPTBot should be checked before you trust it or block it.

OpenAI publishes IP ranges for each bot as JSON files: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json. Match the source IP of each request against the right file.

Our guide on how to check if ChatGPT can crawl your website walks through these checks in more detail, including what to look for in common CDN dashboards.

Make your store worth citing in ChatGPT

Letting OAI-SearchBot in is step one. After that, the page has to be easy to read and worth quoting. For product and collection pages, that means:

• Put key content in the HTML. Product names, prices, specs, sizing, shipping and return policies should be in the page source. Content that only appears after JavaScript runs, such as some review widgets or tabbed spec panels, may be missed by crawlers that don't render scripts.
• Answer the shopper's question directly. Lead with the answer, then add detail. A short "who this is for" line, a comparison table and a real FAQ make a product or guide page easy to extract. Our AEO checker scores a page for answer-friendly structure.
• Keep structured data accurate. Product, Organization and breadcrumb markup help machines understand your pages. Validate it with our schema validator. Markup alone does not guarantee a citation.
• Be consistent everywhere. If your price, shipping terms or product claims differ between your site, marketplaces and review sites, AI answers may repeat the wrong version.

For the full playbook on getting products recommended, see how to appear in ChatGPT.

Your next step

Open your live robots.txt today and answer three questions: Is OAI-SearchBot allowed on your product and collection pages? Is GPTBot set the way you actually want? Do your named groups repeat the path rules from your wildcard group? Then spend ten minutes in your CDN logs confirming that OAI-SearchBot requests come from OpenAI's published IPs and get a 200 response.

If you'd like a second pair of eyes on crawler access and AI visibility for your store, our ChatGPT SEO team can help, or you can get in touch directly.