Content Signals — agent readiness

Why an agent cares

It separates three uses a blanket Disallow cannot: appearing in search, being used as AI input, and being used for training. Publication is enormous and consumption is nil, so treat it today as an expression of preference rather than a contract.

Adoption

Platform default. A major platform ships it by default or in one click, so its footprint is measurable at web scale. For this one in particular, published on millions of sites, and no crawler has committed to honouring it yet.

Direction of travel: ▲ rising, as at 2026-07-28.

Default on 3.8M+ Cloudflare-managed robots.txt domains since Sep 24, 2025, but no crawler or LLM operator has committed to honouring it.

Adopter	Depth	Evidence
Cloudflare (managed robots.txt)	`default-on`	blog.cloudflare.com
Liferay (CMS feature)	`production`	liferay.dev

What we check

Absence is never a finding. Every check below runs only when this technology is detected, so a site that has not adopted this is not assessed on it and appears in no report. 4 of these 4 can fail it. In Lumar these appear once per site, in the site-wide results of each crawl.

Check	Severity	Raises	On whose authority
Content Signal Outside Group	high	usability	vendor profile: Cloudflare Content Signals Policy (managed robots.txt)
Supporting documents: Cloudflare Content Signals Policy (managed robots.txt) Related guidance in Cloudflare Content Signals Policy (managed robots.txt). Bad: a Content-Signal line sits before any User-agent group, so it applies to no crawler. Fix: move it inside the intended User-agent group.
Content Signal Conflicting	high	usability	vendor profile: Cloudflare Content Signals Policy (managed robots.txt)
Supporting documents: Cloudflare Content Signals Policy (managed robots.txt) Related guidance in Cloudflare Content Signals Policy (managed robots.txt). Bad: the same key is set to two different values in one group (e.g. search=yes and search=no), which is contradictory and may be ignored entirely. Fix: keep one value per key per group.
Content Signal Unknown Value	medium	usability	vendor profile: Cloudflare Content Signals Policy (managed robots.txt)
Supporting documents: Cloudflare Content Signals Policy (managed robots.txt) Related guidance in Cloudflare Content Signals Policy (managed robots.txt). Bad: a recognized key carries a value outside its vocabulary (search/ai-input/ai-train take yes\|no; use takes immediate\|reference\|full), so the preference is dropped. Fix: use the documented value, e.g. Content-Signal: ai-train=no.
Content Signal Empty Key/Value	medium	usability	vendor profile: Cloudflare Content Signals Policy (managed robots.txt)
Supporting documents: Cloudflare Content Signals Policy (managed robots.txt) Related guidance in Cloudflare Content Signals Policy (managed robots.txt). Bad: an entry is missing its key or value (e.g. "=no", "search="), so nothing registers. Fix: write each entry as key=value, e.g. search=yes, ai-train=no.

Examples

Both run through the same checks as a live scan: the first passes, the second is flagged.

correcttext

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtext

Disallow: /private/

User-agent:
Disallow: /nowhere/

User-agent: *
Content-Signal: search=yes, ai-train=maybe, =no
Allow /
Disalow: /nope/

User-agent: GPT-Bot
Allow: /
Disallow: /

User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

User-agent: Bytespider

Specifications

Document	Revision	Kind
Cloudflare Content Signals Policy (managed robots.txt)	Living document	vendor profile
contentsignals.org — the policy text	Living document	guidance

Last re-read against the published documents: 2026-10-07.