AI-bot rules — agent readiness

Why an agent cares

Naming a specific AI crawler only works if the token is spelled the way that crawler sends it, and a misspelled token reads as a rule for nobody. The token list is fetched at runtime from a maintained corpus, so this check is an inference, not a spec requirement.

Adoption

Platform default. A major platform ships it by default or in one click, so its footprint is measurable at web scale.

Direction of travel: ▲ rising, as at 2026-07-28.

Cloudflare blocks AI crawlers by default for new zones (Jul 1, 2025), and mid-2026 snapshots put 35 to 45% of prominent sites blocking at least one AI bot.

Adopter	Depth	Evidence
Cloudflare (default for new domains)	`default-on`	blog.cloudflare.com
News publishers (79% block at least one AI training bot)	`production`	buzzstream.com

What we check

Absence is never a finding. Every check below runs only when this technology is detected, so a site that has not adopted this is not assessed on it and appears in no report. 2 of these 4 can fail it. In Lumar these appear once per site, in the site-wide results of each crawl.

Check	Severity	Raises	On whose authority
Per-Bot Contradictory Allow/Disallow	high	usability	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Client guidance in Robots Exclusion Protocol. Bad: a bot's merged rules both allow and disallow the same path (RFC 9309 combines every group that targets the bot, so a pair split across groups counts), and the RFC resolves such a pair in favour of Allow, so the Disallow you wrote never takes effect. Fix: delete whichever rule you didn't mean.
Per-Bot Misspelled Token	medium	usability	our inference, based on Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Our inference from what the page serves. Bad: a group targets a mistyped AI bot token (e.g. "GPT-Bot" instead of "GPTBot"); matching is exact, so the rule never applies and the bot follows the default group instead. Fix: use the bot's exact published token, verified against the ai-robots-txt registry.
Per-Bot Empty Group	informational	nothing	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Robots Exclusion Protocol permits this, so it never fails the site. Informational (does not fail perBotRuleValid): no group targeting this bot declares any rule (checked after merging every matching group, as RFC 9309 does), which the spec permits, so nothing is restricted for it. Often an unfinished edit worth reviewing.
Per-Bot Duplicate Group	informational	nothing	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Robots Exclusion Protocol permits this, so it never fails the site. Informational (does not fail perBotRuleValid): the same bot token starts multiple groups. RFC 9309 requires crawlers to merge matching groups, so every rule still applies; merging them yourself just makes the policy readable.

Examples

Both run through the same checks as a live scan: the first passes, the second is flagged.

correcttext

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtext

Disallow: /private/

User-agent:
Disallow: /nowhere/

User-agent: *
Content-Signal: search=yes, ai-train=maybe, =no
Allow /
Disalow: /nope/

User-agent: GPT-Bot
Allow: /
Disallow: /

User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

User-agent: Bytespider

Specifications

Document	Revision	Kind
Google crawlers and user agents	Living document	guidance
ai-robots-txt/ai.robots.txt — the token corpus we read at runtime	Living document	guidance
RFC 9309 — Robots Exclusion Protocol	Published RFC	specification

Last re-read against the published documents: 2026-10-05.