Scan at scaleSites with a malformed robots.txt
Sites with a malformed robots.txt
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
robots.txt is where a site tells crawlers and AI bots what they may fetch; a syntax error doesn't fail loudly, the rule just silently never applies.
These sites' file has structural problems: a rule before any User-agent group, a missing colon, a misspelled directive (e.g. "Disalow" for "Disallow"), an empty User-agent, or a file so large crawlers truncate it (~500 KB).
How to fix it. Correct the flagged lines: every rule inside a User-agent group, written as key: value with correctly spelled directive names, per the Robots Exclusion Protocol (RFC 9309). A record that isn't part of the protocol at all is allowed by RFC 9309 and is not flagged here.
The Conformance Issue Detected column is true for the syntax rules the protocol places on the file, and false when the only problem is size: RFC 9309's 500-kibibyte limit is addressed to the parser, not to the publisher, so an oversized file breaks crawlers without breaking the specification.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - robots.txt Valid
robotsTxtValid - robots.txt Conformance Issue Detected
robotsTxtConformanceIssueDetected - robots.txt Findings
robotsTxtFindings - robots.txt Bytes
robotsTxtBytes - robots.txt Unknown Directive Samples
robotsTxtUnknownDirectiveSamples - robots.txt Directive Before Group
robotsTxtDirectiveBeforeGroup - robots.txt Missing Colon
robotsTxtMissingColon - robots.txt Unknown Directive
robotsTxtUnknownDirective - robots.txt Empty User-agent
robotsTxtEmptyUserAgent - robots.txt Oversized
robotsTxtOversized
About robots.txt
The crawl-policy file every agent reads first.
Read how robots.txt is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report ships in the Lumar agent-readiness container and appears on any project it is linked to, on every crawl. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.