A robots.txt tester checks whether a specific crawler is allowed to fetch a specific URL under your site's robots.txt rules. Fetch or paste your file into the QCK Robots.txt & AI Crawler Tester, enter a URL, and see which search and AI crawlers, including Googlebot, GPTBot, ClaudeBot and PerplexityBot, can reach it.
What robots.txt is
Robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt. It tells crawlers which paths they may and may not request. It follows the Robots Exclusion Protocol, which was formalized as an internet standard (RFC 9309) in 2022.
A file is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules:
User-agent: *Disallow: /cartDisallow: /searchUser-agent: GPTBotDisallow: /Sitemap: https://example.com/sitemap.xml
Three rules of matching are worth knowing:
- A crawler obeys only its most specific group. If a group names GPTBot, GPTBot follows that group and ignores the
*group entirely. - The most specific path rule wins. For Google, the longest matching rule applies, and when an Allow and Disallow rule are equally specific, Allow wins.
- Wildcards are supported by major crawlers.
*matches any sequence of characters and$marks the end of a URL.
Why robots.txt matters for SEO and AI search
Crawling is not indexing
Robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still appear in Google if other pages link to it, just without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or header instead. Google stopped supporting noindex rules inside robots.txt in 2019, and it ignores the crawl-delay directive.
AI crawlers have their own user agents
AI companies use separate crawlers for different purposes, and you can allow or block each one:
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search results |
| ChatGPT-User | OpenAI | Fetching pages when a user asks ChatGPT to |
| ClaudeBot | Anthropic | Collecting content for model training |
| PerplexityBot | Perplexity | Indexing sites for Perplexity search answers |
| Google-Extended | A control token for use of content in Gemini models; it does not affect Google Search | |
| CCBot | Common Crawl | Building an open web archive used by many AI projects |
Operators update their crawler lists, so check each company's documentation for current names. Note that Google AI Overviews are part of Google Search, so they are governed by Googlebot, not Google-Extended.
The strategic question for most brands is not "block or allow AI" but which purpose to allow. Many businesses that want to be cited in AI answers allow search and user-triggered agents even if they choose to block training crawlers. Blocking everything protects content but removes you from the answers your buyers read.
How to use the QCK robots.txt tester
- Open the Robots.txt & AI Crawler Tester.
- Enter your domain to fetch the live robots.txt, or paste a draft file to test changes before you deploy them.
- Enter the URL or path you want to test, such as a product page or blog post.
- Run the test and review the result for each crawler: allowed or blocked, plus the rule that decided it.
- Edit the draft if a crawler you want is blocked, or one you do not want is allowed, then test again.
- Deploy the final file to the root of every host you serve, including subdomains, since each host needs its own robots.txt.
How to read the results and fix common issues
- Important pages blocked for Googlebot: find the matching Disallow rule. Broad patterns like
Disallow: /*?can block every URL with a parameter, including paginated or filtered pages you may want crawled. - AI search crawlers blocked by accident: a blanket
User-agent: *withDisallow: /, often left over from a staging site, blocks everything. - A named group missing rules you expected: remember that a crawler with its own group ignores the
*group. Repeat any shared rules inside the named group. - CSS or JavaScript blocked: Google needs these files to render pages. Do not disallow asset folders.
- File returns an error: a 404 robots.txt means "crawl everything". A 5xx error can cause Google to pause crawling the site, so make sure the file is served reliably.
Robots.txt tips and mistakes to avoid
- Do not use robots.txt for security. The file is public and compliance is voluntary. Protect private content with authentication.
- Keep the file small and readable. Google processes only the first 500 KiB.
- List your sitemaps. A
Sitemap:line with an absolute URL helps crawlers find them. See the XML sitemap guide. - Test before every deploy. A single character in a Disallow line can change what is crawled sitewide.
- Consider llms.txt separately. It is a different, proposed file with a different purpose. Read what llms.txt is before adding one.
Deciding which AI crawlers to allow is a business decision as much as a technical one. Our AI search optimization team can help you set a policy that protects content while keeping you visible in AI answers, and our technical SEO team handles crawl configuration.