agentic‑readiness docs
/
Scan at scale Blog Build one › Run a scan ›
Menu

Scan at scaleSites with invalid Content Signals

Sites with invalid Content Signals

A site-wide report: one row per host, over the access rows the crawl emitted.

What this report returns

Content Signals are Cloudflare's robots.txt extension for stating how AI may use your content: search, ai-input, and ai-train, each set to yes or no.

A row appears here when any of these is wrong, worst first:

  • ConflictingThe same key is set to two different values in one group (e.g. search=yes and search=no), which is contradictory and may be ignored entirely.Fix Keep one value per key per group.
  • Outside GroupA Content-Signal line sits before any User-agent group. Cloudflare only documents the signal inside a group, so a crawler that reads it per group may never see it.Fix Move it inside the group it is meant for, e.g. under User-agent: *.
  • Unknown ValueA recognized key carries a value outside its vocabulary (search/ai-input/ai-train take yes|no; use takes immediate|reference|full), so the preference is dropped.Fix Use the documented value, e.g. Content-Signal: ai-train=no.
  • Empty Key/ValueAn entry is missing its key or value (e.g. "=no", "search="), so nothing registers.Fix Write each entry as key=value, e.g. search=yes, ai-train=no.

The columns it shows

Every defect this technology can carry is a column in the grid, worst first, so nothing is hidden behind a filter you have to know to apply.

  • Hosthost
  • Content Signals ValidcontentSignalsValid
  • Content Signals FindingscontentSignalsFindings
  • Content Signal Declared ValuescontentSignalDeclaredValues
  • Content Signal Unknown Value SamplescontentSignalUnknownValueSamples
  • Content Signal ConflictingcontentSignalConflicting
  • Content Signal Outside GroupcontentSignalOutsideGroup
  • Content Signal Unknown ValuecontentSignalUnknownValue
  • Content Signal Empty Key/ValuecontentSignalEmptyKeyValue

About Content Signals

Cloudflare's robots.txt vocabulary for search, AI input and AI training.

Read how Content Signals is checked, including the specification it is validated against and a worked correct and broken example.

Running it

This report comes with Agentic Readiness in Lumar and appears on every crawl of any project that has it turned on. See the whole set, or check a single URL with the scanner on this site, which runs the same code with no account.

Get Lumar pricing