# Sites with broken AI bot rules

A site-wide report: one row per host, over the access rows the crawl emitted.

## What this report returns

These sites went to the trouble of writing robots.txt groups for specific AI bots (GPTBot, ClaudeBot, ...), but a rule they wrote never applies: a misspelled bot token matches no crawler, so the bot silently falls through to the default group; and where one group both allows and disallows the same path, RFC 9309 resolves the tie in favour of Allow, so the Disallow is dead.

A row appears here when any of these is wrong, worst first:

-   **Contradictory Allow/Disallow**A bot's merged rules both allow and disallow the same path (RFC 9309 combines every group that targets the bot, so a pair split across groups counts), and the RFC resolves such a pair in favour of Allow, so the Disallow you wrote never takes effect.**Fix** Delete whichever rule you didn't mean.
-   **Misspelled Token**A group targets a mistyped AI bot token (e.g. `"GPT-Bot"` instead of `"GPTBot"`); matching is exact, so the rule never applies and the bot follows the default group instead.**Fix** Use the bot's exact published token, verified against the ai-robots-txt registry.

## The columns it shows

Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.

-   Host`host`
-   Per-Bot Rules Valid`perBotRuleValid`
-   Per-Bot Findings`perBotRuleFindings`
-   Per-Bot AI Bots Named`perBotRuleAgentTokens`
-   Per-Bot Misspelled Token Samples`perBotRuleMisspelledTokenSamples`
-   Per-Bot Contradictory Allow/Disallow`perBotRuleContradictoryAllowDisallow`
-   Per-Bot Misspelled Token`perBotRuleMisspelledToken`
-   Per-Bot Empty Group`perBotRuleEmptyGroup`
-   Per-Bot Duplicate Group`perBotRuleDuplicateGroup`
-   Per-Bot User-Triggered Fetcher Disallowed`perBotRuleUserTriggeredFetcher`

## About AI-bot rules

Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.

[Read how AI-bot rules is checked](https://agentic-readiness.lumar.io/docs/access/per-bot-rule), including the specification it is validated against and a worked correct and broken example.

## Running it

This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. [See the whole set](https://agentic-readiness.lumar.io/scan-at-scale), or check a single URL with the [scanner on this site](https://agentic-readiness.lumar.io/), which runs the same code with no account.

[Get Lumar pricing →](https://www.lumar.io/pricing/)