Scan at scaleSites with an invalid ai.txt
Sites with an invalid ai.txt
A site-wide report: one row per host, over the access rows the crawl emitted.
What this report returns
ai.txt (the Geodocs AI Agent Content Spec) states a site's AI attribution and citation preferences at /ai.txt.
A row appears here when any of these is wrong, worst first:
- Invalid Source URL
Source-urlisn't an absolute http(s) URL, so agents can't resolve the canonical source to attribute.Fix Use a full URL, e.g.Source-url: https://www.example.com/. - Unknown Attribution Value
Attribution-requiredis something other than"yes"or"no", so the attribution preference can't be interpreted and is dropped.Fix Set it to exactly yes or no.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - ai.txt Valid
aiTxtValid - ai.txt Findings
aiTxtFindings - ai.txt Directive Keys
aiTxtDirectiveKeys - ai.txt Source URL Value
aiTxtSourceUrlValue - ai.txt Attribution Value
aiTxtAttributionValue - ai.txt Invalid Source URL
aiTxtInvalidSourceUrl - ai.txt Unknown Attribution Value
aiTxtUnknownAttributionValue
About ai.txt
Attribution preferences for AI systems that use the content.
Read how ai.txt is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.