Scan at scaleSites with broken AI bot rules
Sites with broken AI bot rules
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
These sites went to the trouble of writing robots.txt groups for specific AI bots (GPTBot, ClaudeBot, ...), but a rule they wrote never applies: a misspelled bot token matches no crawler, so the bot silently falls through to the default group; and where one group both allows and disallows the same path, RFC 9309 resolves the tie in favour of Allow, so the Disallow is dead.
How to fix it. Use each bot's exact published token, and delete whichever half of a contradictory pair you didn't mean. (Empty and duplicate groups are shown for review but are valid: requires crawlers to merge matching groups, so those rules do still apply.)
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - Per-Bot Rules Valid
perBotRuleValid - Per-Bot Findings
perBotRuleFindings - Per-Bot Agent Tokens
perBotRuleAgentTokens - Per-Bot Misspelled Token Samples
perBotRuleMisspelledTokenSamples - Per-Bot Contradictory Allow/Disallow
perBotRuleContradictoryAllowDisallow - Per-Bot Misspelled Token
perBotRuleMisspelledToken - Per-Bot Empty Group
perBotRuleEmptyGroup - Per-Bot Duplicate Group
perBotRuleDuplicateGroup
About AI-bot rules
Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.
Read how AI-bot rules is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report ships in the Lumar agent-readiness container and appears on any project it is linked to, on every crawl. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.