How the check works

For each AI crawler, we do two things:

  • We read your robots.txt exactly as that crawler does. robots.txt is the file that tells crawlers what they may read. If it says no, the real crawler is told no.
  • We request your page using the crawler’s name. This catches security plugins and firewalls that turn AI crawlers away even when robots.txt allows them. We skip this on Cloudflare sites, which recognise the real crawlers by where they come from and would turn away our test.

A normal browser request runs first, so a site that’s simply down isn’t reported as blocking AI.

Questions

Why does it matter if AI can read my website?

ChatGPT, Claude and Perplexity search the web before they recommend a business. Each uses its own search crawler, and if your robots.txt or firewall keeps that crawler out, your pages can’t be used in its answers. OpenAI, for example, says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT’s search answers.

Which crawlers do you check?

Googlebot, which Google Search and AI Overviews rely on; the AI search crawlers that decide what can appear in answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot); the ones that open a page when a person asks about it (ChatGPT-User, Claude-User, Perplexity-User); and two training crawlers (GPTBot, ClaudeBot), shown separately.

Should I block the training crawlers?

That’s your choice. Training crawlers collect content for future AI models; OpenAI and Anthropic each name a separate search crawler for what appears in search answers. Blocking GPTBot or ClaudeBot doesn’t block their search crawlers.

What about Google AI Overviews and Gemini?

AI Overviews are part of Google Search, which uses Googlebot, so we check that your robots.txt lets Googlebot in. We judge Google by robots.txt only: many security tools block requests that merely claim to be Googlebot, so testing by name would raise false alarms. Google-Extended is a separate robots.txt setting for Gemini; Google says it doesn’t affect your place in Google Search.

My site uses Cloudflare. Is the result reliable?

We read your robots.txt exactly as the crawlers do, so that part is reliable. Cloudflare’s AI Crawl Control can also block crawlers by name, including Googlebot, and that can’t be seen from outside. If your site is on Cloudflare, we tell you which switches to check.

For more detail, read our guides to ChatGPT, Claude and Perplexity.

Sources: OpenAI: Overview of OpenAI crawlers · Anthropic: Does Anthropic crawl data from the web? · Perplexity: Perplexity crawlers · Google: Google common crawlers