robots.txt — agent readiness

Why an agent cares

It is the only access declaration essentially all crawlers honour, so a structural mistake in it applies to every agent at once. A stray directive before the first User-agent group, or a rule in the wrong group, silently applies to nobody.

Adoption

Web baseline. Standard equipment on the web. Its absence is the exception rather than its presence.

Direction of travel: ► flat, as at 2026-07-28.

RFC 9309 standardised what had been universal practice since 1994.

What we check

Absence is never a finding. Every check below runs only once robotsTxtPresent is true, so a site that has not adopted this is not assessed on it and appears in no report. 4 of these 5 can fail it. In a Lumar crawl these land on one row per site, in the crawl-level table where itemType is access.

Check	Severity	Raises	On whose authority
`robotsTxtDirectiveBeforeGroup`	high	conformance, usability	specification (RFC 9309)
`robotsTxtMissingColon`	medium	conformance, usability	specification (RFC 9309)
`robotsTxtEmptyUserAgent`	medium	conformance, usability	specification (RFC 9309)
`robotsTxtOversized`	medium	usability	specification (RFC 9309)
`robotsTxtUnknownDirective`	informational	nothing	specification (RFC 9309)

Examples

Both of these are fixtures the test suite runs through the real collector, not snippets written for a document: the first is a shape we score as valid, the second is one we flag.

correcttexttest/e2e/test-pages/access/correct.robots.txt

# Correct robots.txt — every signal present and implemented correctly.
# robotsTxtValid=true, perBotRuleValid=true, contentSignalsValid=true.

User-agent: *
# The optional `use` extension takes its own vocabulary (immediate/reference/full) — a compliant
# use=reference must never be flagged (Cloudflare policy, re-verified 2026-07-28).
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtexttest/e2e/test-pages/access/issues.robots.txt

# Broken robots.txt — every tech present but implemented wrong.
# robotsTxtValid=false, perBotRuleValid=false, contentSignalsValid=false.

# robotsTxtDirectiveBeforeGroup: a Disallow before any User-agent group.
Disallow: /private/

# robotsTxtEmptyUserAgent: User-agent with no value.
User-agent:
Disallow: /nowhere/

User-agent: *
# contentSignalUnknownValue (ai-train=maybe) + contentSignalEmptyKeyValue (=no).
Content-Signal: search=yes, ai-train=maybe, =no
# robotsTxtMissingColon: no colon separator.
Allow /
# robotsTxtUnknownDirective: a misspelling of a real directive, so the rule is silently ignored.
# (A record simply outside the protocol, e.g. Example-Extension:, is RFC 9309 §2.2.4-legal and NOT flagged.)
Disalow: /nope/

# perBotRuleMisspelledToken (GPT-Bot vs GPTBot) + perBotRuleContradictoryAllowDisallow (/).
User-agent: GPT-Bot
Allow: /
Disallow: /

# perBotRuleDuplicateGroup: CCBot defined in two separate groups.
User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

# perBotRuleEmptyGroup: Bytespider group with no rules.
User-agent: Bytespider

Specifications

Document	Revision	Kind
RFC 9309 — Robots Exclusion Protocol	`RFC 9309`	specification
Google Search: robots.txt specification	`living`	vendor profile

Last re-read against the published documents: 2026-08-12.