Scan at scaleSites with a broken Web Bot Auth directory
Sites with a broken Web Bot Auth directory
A site-wide report: one row per host, over the web_bot_auth rows the crawl emitted.
What this report returns
Web Bot Auth lets a site prove its outbound bots' identity: verifiers fetch the site's public keys from a JWKS directory at /.well-known/http-message-signatures-directory and check request signatures against them.
These sites serve a directory verifiers can't use: not a JWKS, an empty key list, no usable verification key, exposed private key material, or the wrong media type.
How to fix it. Publish {"keys": [...]} containing public Ed25519/EC/RSA verification keys only, served as application/http-message-signatures-directory+json (JWKS per RFC 7517; keys per the RFC 9421 algorithm registry). Two columns tell you how to triage: Security Risk Detected marks the one fault that hands an attacker your bots' identity, published private key material, and Conformance Issue Detected is false when the only problem is an empty key list, which is a directory nobody can verify against rather than a malformed JWK Set.
The columns it shows
Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.
- Host
host - Web Bot Auth Valid
webBotAuthValid - Web Bot Auth Conformance Issue Detected
webBotAuthConformanceIssueDetected - Web Bot Auth Security Risk Detected
webBotAuthSecurityRiskDetected - Web Bot Auth Findings
webBotAuthFindings - Web Bot Auth Key Types
webBotAuthKeyTypes - Web Bot Auth Served Media Type
webBotAuthServedMediaType - Web Bot Auth Malformed Directory
webBotAuthMalformedDirectory - Web Bot Auth Missing Keys
webBotAuthMissingKeys - Web Bot Auth Invalid Key
webBotAuthInvalidKey - Web Bot Auth Private Key Exposed
webBotAuthPrivateKeyExposed - Web Bot Auth Wrong Media Type
webBotAuthWrongMediaType - Web Bot Auth Unregistered Alg Name
webBotAuthUnregisteredAlgName
About Web Bot Auth directory
Public keys letting verified agents prove who they are.
Read how Web Bot Auth directory is checked, including the specification it is validated against and a worked correct and broken example.
Running it
This report ships in the Lumar agent-readiness container and appears on any project it is linked to, on every crawl. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.