Content Signals — agent readiness

Why an agent cares

It separates three uses a blanket Disallow cannot: appearing in search, being used as AI input, and being used for training. Publication is enormous and consumption is nil, so treat it today as an expression of preference rather than a contract.

Adoption

Platform default. A major platform ships it by default or in one click, so its footprint is measurable at web scale. For this one in particular, published on millions of sites, and no crawler has committed to honouring it yet.

Direction of travel: ▲ rising, as at 2026-07-28.

Default on 3.8M+ Cloudflare-managed robots.txt domains since Sep 24, 2025, but no crawler or LLM operator has committed to honouring it.

Adopter	Depth	Evidence
Cloudflare (managed robots.txt)	`default-on`	blog.cloudflare.com
Liferay (CMS feature)	`production`	liferay.dev

What we check

Absence is never a finding. Every check below runs only once contentSignalPresent is true, so a site that has not adopted this is not assessed on it and appears in no report. 4 of these 4 can fail it. In a Lumar crawl these land on one row per site, in the crawl-level table where itemType is access.

Check	Severity	Raises	On whose authority
`contentSignalOutsideGroup`	high	usability	Cloudflare profile
`contentSignalConflicting`	high	usability	Cloudflare profile
`contentSignalUnknownValue`	medium	usability	Cloudflare profile
`contentSignalEmptyKeyValue`	medium	usability	Cloudflare profile

Examples

Both of these are fixtures the test suite runs through the real collector, not snippets written for a document: the first is a shape we score as valid, the second is one we flag.

correcttexttest/e2e/test-pages/access/correct.robots.txt

# Correct robots.txt — every signal present and implemented correctly.
# robotsTxtValid=true, perBotRuleValid=true, contentSignalsValid=true.

User-agent: *
# The optional `use` extension takes its own vocabulary (immediate/reference/full) — a compliant
# use=reference must never be flagged (Cloudflare policy, re-verified 2026-07-28).
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtexttest/e2e/test-pages/access/issues.robots.txt

# Broken robots.txt — every tech present but implemented wrong.
# robotsTxtValid=false, perBotRuleValid=false, contentSignalsValid=false.

# robotsTxtDirectiveBeforeGroup: a Disallow before any User-agent group.
Disallow: /private/

# robotsTxtEmptyUserAgent: User-agent with no value.
User-agent:
Disallow: /nowhere/

User-agent: *
# contentSignalUnknownValue (ai-train=maybe) + contentSignalEmptyKeyValue (=no).
Content-Signal: search=yes, ai-train=maybe, =no
# robotsTxtMissingColon: no colon separator.
Allow /
# robotsTxtUnknownDirective: a misspelling of a real directive, so the rule is silently ignored.
# (A record simply outside the protocol, e.g. Example-Extension:, is RFC 9309 §2.2.4-legal and NOT flagged.)
Disalow: /nope/

# perBotRuleMisspelledToken (GPT-Bot vs GPTBot) + perBotRuleContradictoryAllowDisallow (/).
User-agent: GPT-Bot
Allow: /
Disallow: /

# perBotRuleDuplicateGroup: CCBot defined in two separate groups.
User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

# perBotRuleEmptyGroup: Bytespider group with no rules.
User-agent: Bytespider

Specifications

Document	Revision	Kind
Cloudflare Content Signals Policy (managed robots.txt)	`living`	vendor profile
contentsignals.org — the policy text	`living`	guidance

Last re-read against the published documents: 2026-08-12.