Blocking "AI bots" in robots.txt can remove a website from ChatGPT search answers. OpenAI does not run one crawler. It runs several, each with a different job. Its documentation states that sites which block OAI-SearchBot will not be shown in ChatGPT search answers. Many sites that blocked AI crawlers in 2023 and 2024 to keep their content out of model training also blocked the crawler that makes them eligible to be cited. Some sites that fixed their robots.txt are still invisible, because their CDN or firewall blocks the crawler anyway.
Why Robots.txt Now Controls AI Search Visibility
Robots.txt is a plain-text file at the root of a domain (for example, example.com/robots.txt). It tells automated crawlers which parts of a site they may access. Compliance is voluntary, but major AI companies, including OpenAI, document that their crawlers respect it.
For two decades, robots.txt mostly governed one relationship: search engine crawlers indexing pages for search results. AI search systems such as ChatGPT search work similarly. They rely on an index of web content that is retrieved, summarized, and cited when users ask questions.
A page that the indexing crawler cannot access cannot enter that index. A page outside the index cannot be retrieved or cited, however good it is. Robots.txt has therefore become a gate for AI-assisted discovery, not just for traditional search.
OpenAI's Crawlers and What Each One Does
OpenAI's crawler documentation lists separate user agents for separate purposes. Each can be allowed or disallowed independently.
| User agent | Purpose | Effect of blocking |
|---|---|---|
| GPTBot | Collects content that may be used to train OpenAI's foundation models | Signals that the site's content should not be used for training |
| OAI-SearchBot | Surfaces websites in ChatGPT's search features | Per OpenAI, the site will not be shown in ChatGPT search answers |
| ChatGPT-User | Fetches pages when a user's action in ChatGPT or a custom GPT requires it | OpenAI notes that, because these requests are user-initiated, robots.txt rules may not apply in the same way |
| OAI-AdsBot | Listed in OpenAI's documentation for advertising-related purposes | Scope is defined in OpenAI's documentation and may evolve |
The critical distinction is between GPTBot (training) and OAI-SearchBot (search visibility). These are different decisions:
- Training consent asks whether a site's content may shape a model's general knowledge.
- Retrieval eligibility asks whether a site may be found, quoted, and linked when someone asks a question.
A site that wants to opt out of training but remain eligible for ChatGPT search can express that directly:
User-agent: GPTBot
Disallow: /User-agent: OAI-SearchBot
Allow: /
One detail catches many sites. Crawlers generally follow the most specific group that matches their user agent. If a file contains User-agent: * with Disallow: / and has no group for OAI-SearchBot, the search crawler falls under the wildcard and is blocked. Adding an explicit OAI-SearchBot group is what overrides it.
OpenAI also notes that changes to robots.txt can take time, roughly a day, to be reflected in search. Its documentation adds that if a site allows both crawlers, OpenAI may use the results of one crawl for both purposes to avoid duplicate crawling.
The Training-Versus-Retrieval Split Is Industry-Wide
OpenAI is not unique. Several major AI and search providers now separate training access from retrieval or search access. A single blanket "block AI" rule is therefore increasingly likely to have unintended effects.
| Provider | Training-related control | Search or retrieval-related crawlers |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Google-Extended (a control token governing use in Gemini models, not a separate crawler) | Googlebot (Search, including AI features in Search) | |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Perplexity | Perplexity states it does not use crawled content for foundation model training | PerplexityBot, Perplexity-User |
Naming conventions and exact scopes differ by provider and change over time, so each provider's current documentation is the authoritative source. The pattern is consistent, though: "AI crawler" is no longer a single category. A policy written when it was a single category now produces outcomes its authors did not intend.
That is why the 2023 advice to "block the AI bots" aged badly. When GPTBot was announced in August 2023, it was OpenAI's main documented crawler, and blocking it was a coherent way to opt out of training. OAI-SearchBot arrived with OpenAI's search product in 2024. Many robots.txt files were never revisited. Some sites took a broader approach, blocking every known AI user agent from published lists, which swept search crawlers in alongside training crawlers.
Why a Correct Robots.txt Is Not Enough
Robots.txt states a site's intent. Other systems determine whether a crawler can actually reach the site:
- CDNs such as Cloudflare, Fastly, or Akamai can block or challenge requests based on bot classification. In 2025, Cloudflare began blocking known AI crawlers by default for new domains and offers one-click AI bot blocking.
- Web application firewalls (WAFs) may reject unfamiliar user agents or high-frequency requests.
- Bot management tools may serve JavaScript challenges or CAPTCHAs that automated crawlers cannot complete.
- Rate limits and geo-blocking can intermittently or entirely deny crawler requests.
When these layers block OAI-SearchBot, the robots.txt file still reads "Allow." The site looks open on paper and is closed in practice. Nothing visible breaks for human visitors, so the problem can persist indefinitely.
OpenAI publishes the IP ranges its crawlers use. Infrastructure teams can use them to verify legitimate OpenAI traffic and allowlist it, rather than relying on user-agent strings, which can be spoofed.
How to Audit AI Crawler Access
A basic audit covers both the declared policy and the enforced reality:
- Read the live robots.txt. Check for explicit groups for OAI-SearchBot, GPTBot, and other providers' crawlers. Look for a wildcard Disallow: / that would catch crawlers without a specific group.
- Decide each permission separately. Make training and retrieval two explicit choices, documented with the reasoning behind each.
- Check CDN and firewall settings. Review AI bot toggles, bot-fight or challenge modes, and custom rules that match AI user agents.
- Verify in server or CDN logs. Confirm that requests from OAI-SearchBot are arriving and receiving 200 responses, not 403s, challenges, or timeouts. Cross-check source IPs against OpenAI's published ranges.
- Assign ownership. Make one person or team responsible for reconciling robots.txt intent with infrastructure enforcement, and review both whenever providers announce new crawlers.
What Most People Miss
The usual framing treats this as a robots.txt typo: find the wrong line, fix it, move on. The deeper issue is that crawler access is decided in at least two places, and different people own them. Robots.txt is typically managed by marketing or SEO teams. CDNs, firewalls, and bot management are typically managed by security or infrastructure teams, whose job is to reduce automated traffic. These teams can be optimizing for opposite outcomes without knowing it.
The failure is also silent. A blocked search crawler produces no error page, no ranking alert, and no obvious traffic decline. For most sites, AI search referrals are still a small and noisy channel, and a site that has never appeared in ChatGPT answers has no baseline to lose. Teams therefore tend to diagnose absence as a content or authority problem and respond by publishing more, when the real constraint is that the retrieval system never received the pages.
The practical implication is a change of sequence. Access determines whether a site is in the candidate pool. Content quality determines whether it is chosen from that pool. AI visibility work that starts with content strategy, before confirming access across both layers, can mean investing in pages that no retrieval system can reach.
Frequently Asked Questions
Does blocking GPTBot remove a site from ChatGPT search?
Not according to OpenAI's documentation. GPTBot governs training. ChatGPT search visibility is governed by OAI-SearchBot, which is controlled separately.
Does allowing OAI-SearchBot guarantee a site will be cited in ChatGPT?
No. Access makes a site eligible for retrieval. Whether it is cited depends on relevance, content quality, and how the system selects sources for a given query.
Can a site opt out of training but stay in AI search?
Yes, for providers that separate the two. With OpenAI, that means disallowing GPTBot and allowing OAI-SearchBot.
Is robots.txt legally binding?
No. It is a voluntary standard. Major AI providers state that their crawlers respect it, but it is not an access control mechanism. Enforcement happens at the infrastructure level.
Key Takeaways
- OpenAI separates model training (GPTBot) from search indexing (OAI-SearchBot), and each is controlled independently in robots.txt.
- OpenAI states that sites blocking OAI-SearchBot will not appear in ChatGPT search answers, so a blanket AI block costs visibility, not just training exposure.
- A wildcard Disallow: / rule blocks OAI-SearchBot unless a more specific OAI-SearchBot group allows it.
- CDNs, firewalls, and bot management tools can block crawlers that robots.txt permits. Verify access in logs, not just in the file.
- Google, Anthropic, and Perplexity have made similar splits between training and retrieval, which makes crawler policy a recurring governance task rather than a one-time setting.
The Bigger Shift
The web's original crawler bargain was simple: let the search engine in, and it sends you traffic. AI has unbundled that bargain into separate transactions: training, indexing for answers, fetching on behalf of a user, and advertising. Each has its own value exchange, and site owners may reasonably accept some and decline others. That makes crawler policy a strategic decision rather than a technical default. Organizations that treat it as a set of deliberate, separately reviewed permissions, coordinated across marketing, legal, and infrastructure, will be able to protect what they want to protect without disappearing from the systems people increasingly use to find things.