agentic‑readiness docs
/
Scan at scale Blog Build one › Run a scan ›
Menu

Scan at scaleSites with broken AI bot rules

Sites with broken AI bot rules

A site-wide report: one row per host, over the access rows the crawl emitted.

What this report returns

These sites went to the trouble of writing robots.txt groups for specific AI bots (GPTBot, ClaudeBot, ...), but a rule they wrote never applies: a misspelled bot token matches no crawler, so the bot silently falls through to the default group; and where one group both allows and disallows the same path, RFC 9309 resolves the tie in favour of Allow, so the Disallow is dead.

A row appears here when any of these is wrong, worst first:

  • Contradictory Allow/DisallowA bot's merged rules both allow and disallow the same path (RFC 9309 combines every group that targets the bot, so a pair split across groups counts), and the RFC resolves such a pair in favour of Allow, so the Disallow you wrote never takes effect.Fix Delete whichever rule you didn't mean.
  • Misspelled TokenA group targets a mistyped AI bot token (e.g. "GPT-Bot" instead of "GPTBot"); matching is exact, so the rule never applies and the bot follows the default group instead.Fix Use the bot's exact published token, verified against the ai-robots-txt registry.

The columns it shows

Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.

  • Hosthost
  • Per-Bot Rules ValidperBotRuleValid
  • Per-Bot FindingsperBotRuleFindings
  • Per-Bot AI Bots NamedperBotRuleAgentTokens
  • Per-Bot Misspelled Token SamplesperBotRuleMisspelledTokenSamples
  • Per-Bot Contradictory Allow/DisallowperBotRuleContradictoryAllowDisallow
  • Per-Bot Misspelled TokenperBotRuleMisspelledToken
  • Per-Bot Empty GroupperBotRuleEmptyGroup
  • Per-Bot Duplicate GroupperBotRuleDuplicateGroup
  • Per-Bot User-Triggered Fetcher DisallowedperBotRuleUserTriggeredFetcher

About AI-bot rules

Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.

Read how AI-bot rules is checked, including the specification it is validated against and a worked correct and broken example.

Running it

This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.

Get Lumar pricing