agentic‑readiness docs
/
Scan at scale Blog Build one › Run a scan ›

Scan at scaleSites with a broken RSL licence

Sites with a broken RSL licence

A site-wide report: one row per host, over the rsl rows the crawl emitted.

What this report returns

RSL (Really Simple Licensing) puts a License line in robots.txt that points at an XML licence, telling AI crawlers on what terms they may use the site's content.

A row appears here when any of these is wrong, worst first:

  • License Not AbsoluteA License line in robots.txt gives a relative address such as /license.xml, but it must be a full URL, so crawlers can't reliably find the licence.Fix Write License: https://example.com/license.xml.
  • License UnreachableThe licence a License line points at answered that it isn't there (for example 404), so crawlers find no licence and treat the content as unlicensed.Fix Publish the licence at that address, or point the line at where it is.
  • Malformed XMLThe licence file isn't well-formed XML (an element that never closes), so a crawler can't read its terms.Fix Validate the file and close every element.
  • Wrong NamespaceThe licence's root isn't <rsl> with the RSL 1.0 namespace as its default namespace (an older draft namespace or a prefix instead), so crawlers that identify licences by namespace don't recognise it.Fix Start the file with <rsl xmlns="https://rslstandard.org/rsl">.
  • Missing ContentThe licence has no <content> element, so it licenses nothing.Fix Add <content url="/"> with a <license> inside it.
  • Content Missing URLA <content> element has no url attribute, so a crawler can't tell which content the licence covers.Fix Add one, e.g. <content url="/"> for the whole site.
  • Content Missing LicenseA <content> element has no <license> inside it (terms on their own are not a licence), so the content it names stays unlicensed.Fix Add a <license> with what you permit, e.g. <license><permits type="usage">ai-train</permits></license>.

Spec: RSL 1.0, sections 2.2, 3.2, 3.3 and 4.4.1. Sites with no License line don't appear here, because absence is not a failure.

The columns it shows

Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.

  • Hosthost
  • RSL ValidrslValid
  • RSL FindingsrslFindings
  • RSL License URLsrslLicenseUrls
  • RSL License Not AbsoluterslLicenseNotAbsolute
  • RSL License UnreachablerslLicenseUnreachable
  • RSL Malformed XMLrslMalformedXml
  • RSL Wrong NamespacerslWrongNamespace
  • RSL Missing ContentrslMissingContent
  • RSL Content Missing URLrslContentMissingUrl
  • RSL Content Missing LicenserslContentMissingLicense
  • RSL Partially AssessedrslPartiallyAssessed

About RSL (Really Simple Licensing)

A License line in robots.txt pointing at an XML licence that says on what terms AI systems may use the content.

Read how RSL (Really Simple Licensing) is checked, including the specification it is validated against and a worked correct and broken example.

Running it

This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.

Get Lumar pricing