A robots.txt file tells web crawlers, including AI crawlers, which parts of your site they may fetch, using named user-agent rules. AI crawlers do two different jobs: training crawlers that feed models (GPTBot, ClaudeBot, Google-Extended) and search crawlers that fetch pages to cite in answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and you can allow or block each independently. It is voluntary, not a security control: compliant bots obey it, but some ignore it, so pair it with server-level blocks when access must truly stop.
Your robots.txt file is probably years old and was written with Googlebot in mind. AI crawlers now read it too, and most site owners have never checked what it tells them.
There are now more than a dozen AI crawlers, and they split into two groups with very different jobs. Block the wrong one and you can disappear from AI answers. Block none and you may be feeding model training you never agreed to. The good news is that a few lines of robots.txt give you precise control.
This guide explains the two kinds of AI crawler, the exact user-agent names that matter, and the rules to allow or block each. It is written for marketers and site owners; you do not need to be a developer to follow it.
Two kinds of AI crawler: training versus search
Training crawlers collect pages to train or improve AI models. The main ones are GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google's Gemini). Blocking these keeps your content out of the training data behind those models.
Search crawlers do something else: they fetch pages in real time to answer a live question and cite their sources. The main ones are OAI-SearchBot (used by ChatGPT search), Claude-SearchBot, and PerplexityBot. These are the crawlers that can carry your brand into an AI answer and send visibility your way.
The key point is that you can allow search while blocking training. Per OpenAI's documentation, blocking GPTBot does not remove you from ChatGPT's search citations, and it has no effect on Google Search at all. Decide the two jobs separately.
The user-agents that matter
These are the crawlers worth naming in your robots.txt. On first use, note that a "user-agent" is simply the name a crawler identifies itself with. Each row's owner publishes official documentation for its bots.
| Crawler | Owner | Job | Obeys robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search / cite | Yes |
| ClaudeBot | Anthropic | Training | Yes |
| Claude-SearchBot | Anthropic | Search / cite | Yes |
| PerplexityBot | Perplexity | Search / cite | Yes |
| Google-Extended | Training (Gemini) | Yes | |
| Bytespider | ByteDance | Training | Often ignored |
The compliance column comes from each owner's own docs: OpenAI, Anthropic, Perplexity, and Google. Bytespider is the exception: it has been widely reported fetching disallowed pages, so treat it as non-compliant.
How to write the rules
robots.txt rules are grouped by user-agent. Here are the three configurations most sites need.
Scenario A: allow AI search, block AI training. This is the common choice for a brand that wants to appear in AI answers but keep its content out of model training.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Scenario B: block all AI crawlers. You can list several user-agents against one rule.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: OAI-SearchBot
Disallow: /
Scenario C: block one crawler from one section only. Use a path instead of the whole site.
User-agent: GPTBot
Disallow: /members/
One rule to remember: each crawler is a separate user-agent. Allowing ClaudeBot does not allow Claude-SearchBot, and blocking GPTBot does not block OAI-SearchBot. If a bot is not named, most default to being allowed.
Test it, and know the limits
To test, open yourdomain.com/robots.txt in a browser and confirm the rules are live, then check your server logs a day or two later to see which user-agents actually visited. A robots.txt testing tool can confirm your syntax.
Then respect the limits. robots.txt is public and voluntary. Compliant bots obey it, but it is a request, not a wall. Bytespider has been reported ignoring it, and Perplexity's user-initiated fetch, Perplexity-User, is documented as not always subject to robots.txt because it acts on behalf of a specific user. When access must truly be stopped, block at the server or CDN level with firewall rules, not just robots.txt.
Finally, remember the positive counterpart. robots.txt only gates access. An llms.txt file invites AI systems toward your best pages. Most brands want both: block what should not be trained on, allow the search crawlers that cite you, and point them at the right content.
Frequently asked questions
Does blocking GPTBot hurt my Google Search ranking?
Blocking GPTBot has no effect on Google Search. GPTBot is OpenAI's crawler and Googlebot is entirely separate, so disallowing GPTBot in robots.txt does not change how Google indexes or ranks your pages.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler, which collects pages to improve its models. OAI-SearchBot fetches pages in real time to cite them in ChatGPT's search answers. They are separate user-agents, so you can block one and allow the other.
Do AI crawlers respect robots.txt?
Most major AI crawlers, including those from OpenAI, Anthropic, Perplexity, and Google, document that they obey robots.txt. Some crawlers, such as ByteDance's Bytespider, have been reported ignoring it, so robots.txt is a request rather than a guarantee.
If I allow ClaudeBot, does that also allow Claude-SearchBot?
No. ClaudeBot and Claude-SearchBot are separate user-agents in robots.txt. A rule for one does not apply to the other, so you must name each crawler you want to allow or block.
Can I block AI training but still appear in AI search answers?
Yes. Block the training crawlers (GPTBot, ClaudeBot, Google-Extended) and allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). This keeps your content out of model training while letting AI answers cite your live pages.
Where to go from here
Your robots.txt is the first thing an AI crawler reads, and a few lines decide whether you feed training, appear in cited answers, or vanish from both. Review it today, make the training-versus-search call deliberately, and confirm you are not accidentally blocking the crawlers that carry your brand into AI answers. Crawler access is one layer of a wider AI-visibility picture, which is exactly what a Brand Visibility Audit maps end to end.