AI-bot rules — agent readiness

Why an agent cares

Naming a specific AI crawler only works if the token is spelled the way that crawler sends it, and a misspelled token reads as a rule for nobody. The token list is fetched at runtime from a maintained corpus, so this check is an inference, not a spec requirement.

Adoption

Platform default. A major platform ships it by default or in one click, so its footprint is measurable at web scale.

Direction of travel: ▲ rising, as at 2026-07-28.

Cloudflare blocks AI crawlers by default for new zones (Jul 1, 2025), and mid-2026 snapshots put 35 to 45% of prominent sites blocking at least one AI bot.

Adopter	Depth	Evidence
Cloudflare (default for new domains)	`default-on`	blog.cloudflare.com
News publishers (79% block at least one AI training bot)	`production`	buzzstream.com

What we check

Absence is never a finding. Every check below runs only once perBotRulePresent is true, so a site that has not adopted this is not assessed on it and appears in no report. 2 of these 4 can fail it. In a Lumar crawl these land on one row per site, in the crawl-level table where itemType is access.

Check	Severity	Raises	On whose authority
`perBotRuleContradictoryAllowDisallow`	high	usability	specification (RFC 9309)
`perBotRuleMisspelledToken`	medium	usability	inference
`perBotRuleEmptyGroup`	informational	nothing	specification (RFC 9309)
`perBotRuleDuplicateGroup`	informational	nothing	specification (RFC 9309)

Examples

Both of these are fixtures the test suite runs through the real collector, not snippets written for a document: the first is a shape we score as valid, the second is one we flag.

correcttexttest/e2e/test-pages/access/correct.robots.txt

# Correct robots.txt — every signal present and implemented correctly.
# robotsTxtValid=true, perBotRuleValid=true, contentSignalsValid=true.

User-agent: *
# The optional `use` extension takes its own vocabulary (immediate/reference/full) — a compliant
# use=reference must never be flagged (Cloudflare policy, re-verified 2026-07-28).
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtexttest/e2e/test-pages/access/issues.robots.txt

# Broken robots.txt — every tech present but implemented wrong.
# robotsTxtValid=false, perBotRuleValid=false, contentSignalsValid=false.

# robotsTxtDirectiveBeforeGroup: a Disallow before any User-agent group.
Disallow: /private/

# robotsTxtEmptyUserAgent: User-agent with no value.
User-agent:
Disallow: /nowhere/

User-agent: *
# contentSignalUnknownValue (ai-train=maybe) + contentSignalEmptyKeyValue (=no).
Content-Signal: search=yes, ai-train=maybe, =no
# robotsTxtMissingColon: no colon separator.
Allow /
# robotsTxtUnknownDirective: a misspelling of a real directive, so the rule is silently ignored.
# (A record simply outside the protocol, e.g. Example-Extension:, is RFC 9309 §2.2.4-legal and NOT flagged.)
Disalow: /nope/

# perBotRuleMisspelledToken (GPT-Bot vs GPTBot) + perBotRuleContradictoryAllowDisallow (/).
User-agent: GPT-Bot
Allow: /
Disallow: /

# perBotRuleDuplicateGroup: CCBot defined in two separate groups.
User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

# perBotRuleEmptyGroup: Bytespider group with no rules.
User-agent: Bytespider

Specifications

Document	Revision	Kind
Google crawlers and user agents	`living`	guidance
ai-robots-txt/ai.robots.txt — the token corpus we read at runtime	`living`	guidance
vedangvatsa/ai-discovery-standards — a second bot user-agent corpus	`living`	guidance

Last re-read against the published documents: 2026-08-12.