Scan at scaleSites with a broken llms.txt
Sites with a broken llms.txt
A site-wide report: one row per host, over the llms_txt rows the crawl emitted.
What this report returns
llms.txt is a hand-written Markdown index at /llms.txt that points an agent at the pages worth reading instead of making it crawl the whole site.
A row appears here when any of these is wrong, worst first:
- Missing H1The file has no H1, which is the one element the format requires, so it never names the site and a client assembling context has to guess from the URL.Fix Make the first line an H1 naming the site, e.g.
"# Acme Corp". - Content Before H1Prose, a blockquote or a list appears above the H1, but the format puts the H1 first (after an optional byte-order mark), so a reader taking the first block as the title takes the wrong thing.Fix Move the H1 to the top and put the summary underneath it as a
"> "blockquote. - Heading In DetailsA heading below H2 sits in the free-text block between the summary and the first file list, where the format allows sections "of any type except headings", so a parser looking for the first H2 can read it as the start of a file list.Fix Turn it into bold text or a list item, or promote it to an H2 with its own list of links.
- Insecure LinkThe file links pages over plain http://, so an agent reading them on its recommendation gets content that can be altered on the way. A Lumar bar: the proposal allows any URL.Fix Link the https URLs.
Spec: the llms.txt proposal at llmstxt.org (v2). Sites that publish no llms.txt don't appear here, because absence is not a failure.
Security checks are provisional; precision has not yet been measured. A clean row is not a security assessment.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - llms.txt Valid
llmsTxtValid - llms.txt Security Risk Detected
llmsTxtSecurityRiskDetected - llms.txt Findings
llmsTxtFindings - llms.txt Title
llmsTxtTitle - llms.txt Sections
llmsTxtSections - llms.txt Link Count
llmsTxtLinkCount - llms.txt Missing H1
llmsTxtMissingH1 - llms.txt Content Before H1
llmsTxtContentBeforeH1 - llms.txt Heading In Details
llmsTxtHeadingInDetails - llms.txt Insecure Link
llmsTxtInsecureLink - llms.txt Multiple H1
llmsTxtMultipleH1 - Ruleset Version
rulesetVersion - Security Check Status
securityCheckStatus - Related Security Frameworks
securityFrameworkReferences
About llms.txt
A Markdown index of a site's most useful pages, written for language models.
Read how llms.txt is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.