# AI-bot rules

Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.

**[Access](/docs/access)** · checked once per host (item type `access`) · 2 checks that can fail it

Naming a specific AI crawler only works if the token is spelled the way that crawler sends it, and a misspelled token reads as a rule for nobody. The token list is fetched at runtime from a maintained corpus, so this check is an inference, not a spec requirement.

## Adoption

**Platform default** (L3)

A major platform ships it by default or in one click, so its footprint is measurable at web scale. Direction of travel: ▲ rising, as of 2026-07-28.

Cloudflare blocks AI crawlers by default for new zones (Jul 1, 2025), and mid-2026 snapshots put 35 to 45% of prominent sites blocking at least one AI bot.

| Adopter | Depth | Evidence |
|---|---|---|
| Cloudflare (default for new domains) | default-on | [blog.cloudflare.com](https://blog.cloudflare.com/content-independence-day-ai-options/) |
| News publishers (79% block at least one AI training bot) | production | [buzzstream.com](https://www.buzzstream.com/blog/publishers-block-ai-study/) |

## What we check

Absence is never a finding: every check below runs only once `perBotRulePresent` is true, so a site that has not
adopted this is not assessed on it and appears in no report.

| Check | Kind | Raises | On whose authority |
|---|---|---|---|
| `perBotRuleContradictoryAllowDisallow` | high severity | usability | specification (RFC 9309) |
| `perBotRuleMisspelledToken` | medium severity | usability | inference |
| `perBotRuleEmptyGroup` | informational | nothing | specification (RFC 9309) |
| `perBotRuleDuplicateGroup` | informational | nothing | specification (RFC 9309) |

## Examples

These are the fixtures the test suite runs through the real collector, not snippets written for a document.

- Correct: `test/e2e/test-pages/access/correct.robots.txt`
- Present but wrong: `test/e2e/test-pages/access/issues.robots.txt`

## Specifications

| Document | Revision | Kind |
|---|---|---|
| [Google crawlers and user agents](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) | living document | guidance |
| [ai-robots-txt/ai.robots.txt — the token corpus we read at runtime](https://github.com/ai-robots-txt/ai.robots.txt) | living document | guidance |
| [vedangvatsa/ai-discovery-standards — a second bot user-agent corpus](https://github.com/vedangvatsa/ai-discovery-standards) | living document | guidance |

Last re-read against the published documents: 2026-08-12.