Scan at scaleSites with a malformed robots.txt
Sites with a malformed robots.txt
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
robots.txt is where a site tells crawlers and AI bots what they may fetch; a syntax error doesn't fail loudly, the rule just silently never applies.
A row appears here when any of these is wrong, worst first:
- Directive Before GroupAn Allow or Disallow rule appears before any
User-agentline, so it belongs to no group and every crawler ignores it.Fix Start the file with aUser-agentline and keep every rule inside a group. - Missing ColonA line is missing its
"key: value"colon, so parsers skip it and the intended rule silently never applies.Fix Write each directive as key: value, e.g. Disallow:/private/. - Empty User-agentA
"User-agent:"line with no value targets no crawler, so its whole group never applies.Fix Name a bot token (e.g. GPTBot) or use * for all crawlers. - OversizedThe file exceeds ~500 KB; Google and others stop reading there, silently dropping every later rule.Fix Trim the file (consolidate path patterns) so all rules fit under the limit.
The Conformance Issue Detected column is true for the syntax rules the protocol places on the file, and false when the only problem is size: RFC 9309's 500-kibibyte limit is addressed to the parser, not to the publisher, so an oversized file breaks crawlers without breaking the specification.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - robots.txt Valid
robotsTxtValid - robots.txt Conformance Issue Detected
robotsTxtConformanceIssueDetected - robots.txt Findings
robotsTxtFindings - robots.txt Size (Bytes)
robotsTxtBytes - robots.txt Unknown Directive Samples
robotsTxtUnknownDirectiveSamples - robots.txt Directive Before Group
robotsTxtDirectiveBeforeGroup - robots.txt Missing Colon
robotsTxtMissingColon - robots.txt Empty User-agent
robotsTxtEmptyUserAgent - robots.txt Oversized
robotsTxtOversized - robots.txt Unknown Directive
robotsTxtUnknownDirective
About robots.txt
The file a site uses to tell crawlers which pages they may fetch.
Read how robots.txt is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.