How to Check If ChatGPT Can Crawl Your Website

To check if ChatGPT can crawl your website, open your live robots.txt and confirm it does not disallow OAI-SearchBot (the crawler behind ChatGPT search), then confirm your firewall or CDN is not blocking OpenAI's published IP ranges and that the page returns real HTML. The only proof that a crawl actually happened is a request in your server or CDN logs from an OpenAI IP address that received a 200 response with your content.

This guide is for e-commerce and Shopify teams who suspect ChatGPT is not seeing their products, or who want to confirm it before investing in AI search. You'll get a five-minute check, the difference between OpenAI's crawlers, how to test a page the way a crawler sees it, how to verify real requests in your logs, the Shopify specifics, and a symptom, cause and fix table for every result you might get.

Can ChatGPT crawl my website? The 5-minute check

Run these checks in order on one important public URL, such as your best-selling product page. Each step rules out one layer, so stop at the first one that fails and fix it before moving on.

1. Open yourdomain.com/robots.txt in a browser. Look for a group that names OAI-SearchBot, and for any User-agent: * group with Disallow: /. If either blocks the path you care about, ChatGPT search is told to stay out.
2. Paste the file into a robots.txt tester. Our free robots.txt tester shows how the rules apply to OAI-SearchBot, GPTBot and ChatGPT-User for a specific path, including inherited wildcard rules.
3. Check your bot and firewall settings. If you run your own CDN or web application firewall, look for "block AI bots" rules, bot fight modes or country blocks that would catch OpenAI's traffic.
4. Fetch the page with a crawler user agent. Run curl with the OAI-SearchBot user agent (shown below) and confirm you get a 200 status and your product name, price and description in the raw HTML.
5. Look for real OpenAI requests in your logs. Filter for the user agent, then confirm the source IP is in openai.com/searchbot.json. A verified 200 response is the strongest evidence you can get.

Which OpenAI crawler are you checking?

"ChatGPT crawling" is not one switch. OpenAI's crawler documentation lists separate user agents with different jobs, and says each robots.txt setting is independent of the others. Before you change anything, decide which behavior you are actually checking.

For most stores, the question that matters is OAI-SearchBot, because that crawler decides whether your pages can show up when a shopper asks ChatGPT for a product recommendation. If you want to stay out of model training but stay in search, you can block GPTBot and allow OAI-SearchBot. Our guide to GPTBot vs OAI-SearchBot covers that decision in depth, and should I block GPTBot walks through the tradeoffs.

OpenAI also notes that if you allow both OAI-SearchBot and GPTBot, it may use one crawl for both purposes to avoid duplicate crawling. So seeing GPTBot in your logs but not OAI-SearchBot does not necessarily mean search is ignoring you.

OpenAI's crawlers are now a large share of AI crawling on the web. In Cloudflare's network data, GPTBot went from 5% of requests from AI-only crawlers in May 2024 to 30% in May 2025, the largest share of any AI crawler.

Step 1: Read your live robots.txt correctly

Open the robots.txt file on the exact hostname that serves the page. www.yourstore.com and yourstore.com can serve different files, and a staging domain's file tells you nothing about production. Read the whole file, not just the top, because a group further down can override what you expect.

How crawlers pick which rules to follow

Under the robots.txt standard (RFC 9309), a crawler follows the most specific group that names it. If your file has a User-agent: OAI-SearchBot group, OAI-SearchBot follows that group and ignores the User-agent: * group entirely. If there is no named group, the wildcard group applies. Within a group, the longest matching path rule wins, so Allow: /products/ beats Disallow: / for a product URL.

That last row is a common mistake. Because a named group replaces the wildcard group for that crawler, adding User-agent: OAI-SearchBot with Allow: / silently removes your cart, checkout and account disallows for it. If you add a named group, copy in the disallow rules you want to keep.

How common GPTBot blocks are

GPTBot is the AI crawler site owners block most. When Cloudflare checked robots.txt files on 3,816 of the top 10,000 domains in June 2025, 250 fully disallowed GPTBot and 62 partially disallowed it, more than any other bot. Fewer domains explicitly allowed it. If you inherited your robots.txt from an agency or a plugin, check whether a training block was written as a blanket AI block that also catches OAI-SearchBot.

After any change, recheck the live file and test the path again. OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.

Step 2: Check your CDN, firewall and bot settings

A clean robots.txt only tells compliant crawlers what you prefer. Your infrastructure decides what actually gets through. Plenty of sites allow OAI-SearchBot in robots.txt and then block it at the network edge without realizing it.

• "Block AI bots" toggles. CDNs and hosts increasingly offer one-click AI bot blocking. Cloudflare announced in July 2025 that it was changing its default to block AI crawlers unless they pay for content. Check whether a setting like this is on and which bots it covers.
• Bot challenge pages. JavaScript challenges and CAPTCHAs stop crawlers cold. A crawler that receives a challenge page instead of your product page has nothing to index.
• Rate limits. Aggressive limits can return 429 responses to bursts of crawler traffic.
• Country or ASN blocks. Blocking data center networks or specific countries can catch crawler traffic that comes from cloud infrastructure.
• Allowlists. OpenAI recommends allowing requests from its published IP ranges. Each agent has its own file: openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json.

Ask whoever manages your CDN or hosting to search the security event log for the OpenAI user agents. Blocked or challenged requests show up there even when they never reach your server logs.

Step 3: Test the page the way a crawler sees it

Your browser is a poor test. It runs JavaScript, carries cookies and may be on an allowlisted office network. A crawler gets none of that. Test with a raw request instead.

Fetch the page with the crawler's user agent

From a terminal, request the page with the OAI-SearchBot user agent string that OpenAI publishes (the version number may change):

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yourstore.com/products/your-product

Check that the status is 200, not 403, 429 or a 3xx redirect to a login or country page. Then drop the -I flag to fetch the full HTML and search it for your product name, price and a sentence from the description.

This test is useful but limited. It catches rules keyed to the user agent string. It does not reproduce OpenAI's IP addresses, so a firewall that allows or blocks by IP may treat the real crawler differently.

Make sure the content is in the HTML

Vercel analyzed AI crawler traffic across its network in December 2024 and reported that none of the major AI crawlers it studied, including OpenAI's, rendered JavaScript. ChatGPT's crawler fetched JavaScript files in 11.5% of requests but did not execute them. If your reviews, FAQs, size charts or prices are injected by a script after the page loads, a crawler may never see them.

The same study found ChatGPT's crawler spent a large share of its fetches on dead ends: 34.82% hit 404 pages and 14.36% followed redirects, compared with 8.22% and 1.49% for Googlebot. Clean redirects and an accurate XML sitemap help crawlers spend their visits on pages that exist.

A 200 status does not guarantee real content. Challenge pages, empty app shells and "are you human" screens can all return 200. Always look at the body.

Step 4: Verify real OpenAI requests in your logs

Every check so far shows whether a crawl could succeed. Logs show whether one did. If you have server, CDN or edge logs, filter requests by the user agent strings OAI-SearchBot, GPTBot and ChatGPT-User, and record the timestamp, path, source IP, status code and any security action.

OpenAI says its crawlers may use a variant user agent with a robots.txt marker when they fetch your robots.txt, which makes those requests easy to separate from page fetches. A recent robots.txt fetch with no page fetches can simply mean you haven't been crawled yet, or that a rule is stopping the crawler.

Be precise about what you report. "No OAI-SearchBot requests in the last 7 days of logs" is a fact. "ChatGPT can't crawl our site" is a conclusion that needs more evidence, because log retention, sampling and multiple infrastructure layers all affect what you see.

See whether crawling turns into visits

Crawling and referrals are different things. Cloudflare's 2025 Year in Review found OpenAI's ratio of crawling to referred visits reached as much as 3,700 to 1 in March 2025, while user-initiated crawling grew more than 21 times over the year. To see the referral side for your own store, check analytics for utm_source=chatgpt.com, which OpenAI says it adds to links from ChatGPT search.

How to check a Shopify store

Shopify stores have a few specifics worth knowing before you go hunting for a problem.

• The default robots.txt is a sensible starting point. Shopify's robots.txt documentation says every store has a default file built for SEO, which keeps crawlers out of cart, checkout and account paths. The default files we checked had no rules naming GPTBot or OAI-SearchBot.
• Custom rules live in robots.txt.liquid. If your theme has this template, someone has customized the file. Review it for AI crawler groups and blanket disallows. Shopify warns that incorrect edits can cost you all of your traffic, and that the customization is unsupported by Shopify Support.
• Password protection blocks everyone. A store or preview behind the storefront password shows crawlers a password page, not your products.
• Apps can hide content. Review widgets, FAQ accordions and bundles that load with JavaScript may be missing from the raw HTML. Run the curl test on a product page and confirm what's there.
• Agentic Storefronts are a separate channel. Shopify notes that blocking AI crawlers in robots.txt affects open-web discoverability only. It doesn't stop Shopify Catalog from sending product data to activated agentic storefronts such as ChatGPT.

For the wider picture of technical issues that hold Shopify stores back, see our Shopify SEO audit guide.

What your results mean and how to fix them

Use this grid to turn what you found into one clear next action.

If access checks out and you still don't show up, the work shifts to visibility. Our AEO checker flags on-page issues that make a page harder for AI answers to quote, and our guide on how to appear in ChatGPT covers the content and authority side.

Keep ChatGPT access working after you fix it

Crawler access breaks quietly. A theme update, a new security rule or an agency "cleaning up" robots.txt can undo it, and nothing in your dashboard will warn you. Put three checks on a monthly calendar: rerun the robots.txt tester on your top product and collection URLs, rerun the curl test, and scan a week of logs for verified OAI-SearchBot requests.

Keep three columns in your notes: what your rules allow, what the crawler actually fetched, and where ChatGPT cites you. Mixing them up is how teams end up changing the wrong setting. If you'd like a second pair of eyes on crawler access and AI visibility for your store, our ChatGPT SEO team can help, or you can get in touch.