robots.txt for AI crawlers: search bots, training bots and fetchers
Most AI companies now run more than one bot, and they do different jobs. Blocking all of them is the quickest way to disappear from AI answers. Allowing all of them means your pages may also be used to train future models. robots.txt lets you choose bot by bot.
Three kinds of bot
Search crawlers build the index an assistant searches when it answers a question. Block one and that assistant stops citing you. Training crawlers collect pages that may be used to train future models. Blocking them does not affect search. Fetchers visit a single page because a person asked the assistant about it.
| Company | Search | Training | When a person asks |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | None listed | Perplexity-User |
| Googlebot | Google-Extended (a token, not a crawler) | Not listed | |
| Apple | Applebot | Applebot-Extended (a token, not a crawler) | Not listed |
Each company documents its own bots: OpenAI, Anthropic, Perplexity, Google and Apple. A few details from those pages matter:
- Perplexity says PerplexityBot is not used to crawl content for AI foundation models.
- Google-Extended has no user agent of its own. It is a name you use in robots.txt to say whether Google may use your pages to train Gemini models. Google says it does not affect inclusion or ranking in Google Search.
- Applebot-Extended does not crawl either. It controls whether pages Applebot crawled may train Apple's models, and Apple says it is not considered in search ranking.
- Fetchers may not follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them because a person asked for the page. Anthropic says its bots honour robots.txt.
- Changes are not instant. OpenAI and Perplexity both say a change can take about 24 hours to take effect.
An example: welcome search, refuse training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /The robots.txt standard says a crawler follows the group that names it, and uses the * group only when no group names it (RFC 9309). So the search crawlers and fetchers above use the open * group, and the named training bots are refused. CCBot is Common Crawl's crawler, whose public archive is widely used as training data.
This site allows every one of them, because being cited by AI assistants is the point of our GEO work. A publisher whose content is the product might reasonably choose the opposite for training bots, or charge them. See charging AI agents.
Two warnings
robots.txt is a request, not a lock. The standard itself says its rules are not a form of access authorization. Anything truly private needs a login.
Your firewall can override your robots.txt. On 1 July 2025 Cloudflare changed its default to block AI crawlers unless they pay (Cloudflare). A robots.txt that welcomes ClaudeBot does nothing if a firewall rule turns it away first. The free GEO audit checks which AI crawlers your robots.txt allows.
Sources
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
- Perplexity crawlers, Perplexity
- Google's common crawlers (Google-Extended), Google Search Central
- About Applebot, Apple
- RFC 9309: Robots Exclusion Protocol, IETF
- Content Independence Day: no AI crawl without compensation, Cloudflare, 1 July 2025