Scan at scaleSites with broken Markdown negotiation
Sites with broken Markdown negotiation
A site-wide report: one row per host, over the markdown_negotiation rows the crawl emitted.
What this report returns
Markdown negotiation is a site answering a request that asks for Markdown (Accept: text/markdown) with a Markdown version of the page, so an agent gets clean text from the same address.
These sites answer that request in a way an agent or a cache can't rely on: a full HTML web page labelled as Markdown, so the agent pays for markup it was told was text; or a Markdown response whose Vary header leaves out Accept although the same address serves HTML to browsers, so a shared cache can hand the Markdown to human visitors.
How to fix it. Serve the Markdown version for a Markdown request, with Content-Type: text/markdown; charset=utf-8 and Vary: Accept, for example "Vary: Accept, Accept-Encoding". Markdown Negotiation HTML Missing Vary is informational and never fails the site: it notes the HTML version lacks Vary: Accept too.
Spec: RFC 9110 (HTTP Semantics), for Content-Type and Vary. Sites that don't negotiate Markdown don't appear here, because absence is not a failure.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - Markdown Negotiation Valid
markdownNegotiationValid - Markdown Negotiation Findings
markdownNegotiationFindings - Markdown Negotiation URL
markdownNegotiationUrl - Markdown Negotiation Content Type
markdownNegotiationContentType - Markdown Negotiation Vary
markdownNegotiationVary - Markdown Negotiation HTML Body
markdownNegotiationHtmlBody - Markdown Negotiation Missing Vary
markdownNegotiationMissingVary - Markdown Negotiation HTML Missing Vary
markdownNegotiationHtmlMissingVary - Markdown Negotiation Partially Assessed
markdownNegotiationPartiallyAssessed
About Markdown negotiation
Markdown served to agents that ask for it with Accept: text/markdown.
Read how Markdown negotiation is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.