Scan at scaleSites with a misspelled AI bot token
Sites with a misspelled AI bot token
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
robots.txt rules only apply to a bot when the User-agent token matches exactly; these sites wrote an AI bot token no bot identifies as (e.g. 'GPT-Bot' instead of 'GPTBot'), so the policy they wrote silently never applies and the bot follows the default group instead.
A row appears here when this is wrong:
- Per-Bot Misspelled TokenA group targets a mistyped AI bot token (e.g.
"GPT-Bot"instead of"GPTBot"); matching is exact, so the rule never applies and the bot follows the default group instead.Fix Use the bot's exact published token, verified against the ai-robots-txt registry.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - Per-Bot Misspelled Token
perBotRuleMisspelledToken - Per-Bot Misspelled Token Samples
perBotRuleMisspelledTokenSamples - Per-Bot Findings
perBotRuleFindings
About AI-bot rules
Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.
Read how AI-bot rules is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.