Scan at scaleSites with a misspelled AI bot token
Sites with a misspelled AI bot token
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
robots.txt rules only apply to a bot when the User-agent token matches exactly; these sites wrote an AI bot token no bot identifies as (e.g. 'GPT-Bot' instead of 'GPTBot'), so the policy they wrote silently never applies and the bot follows the default group instead.
How to fix it. Replace the token with the exact product token, e.g. User-agent: GPTBot; the authoritative list is the ai-robots-txt registry this check validates against (matching per RFC 9309).
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - Per-Bot Misspelled Token
perBotRuleMisspelledToken - Per-Bot Misspelled Token Samples
perBotRuleMisspelledTokenSamples - Per-Bot Findings
perBotRuleFindings
About AI-bot rules
Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.
Read how AI-bot rules is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report ships in the Lumar agent-readiness container and appears on any project it is linked to, on every crawl. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.