robots.txt — agent readiness

Why an agent cares

It is the only access declaration essentially all crawlers honour, so a structural mistake in it applies to every agent at once. A stray directive before the first User-agent group, or a rule in the wrong group, silently applies to nobody.

Adoption

Web baseline. Standard equipment on the web. Its absence is the exception rather than its presence.

Direction of travel: ► flat, as at 2026-07-28.

RFC 9309 standardised what had been universal practice since 1994.

What we check

Absence is never a finding. Every check below runs only when this technology is detected, so a site that has not adopted this is not assessed on it and appears in no report. 4 of these 5 can fail it. In Lumar these appear once per site, in the site-wide results of each crawl.

Check	Severity	Raises	On whose authority
robots.txt Directive Before Group	high	conformance, usability	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Required by Robots Exclusion Protocol. Bad: an Allow/Disallow (or similar) rule appears before any User-agent line, so it belongs to no group and every crawler ignores it. Fix: start the file with a User-agent line and keep every rule inside a group.
robots.txt Missing Colon	medium	conformance, usability	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Required by Robots Exclusion Protocol. Bad: a line is missing its "key: value" colon, so parsers skip it and the intended rule silently never applies. Fix: write each directive as key: value, e.g. Disallow: /private/.
robots.txt Empty User-agent	medium	conformance, usability	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Required by Robots Exclusion Protocol. Bad: a "User-agent:" line with no value targets no crawler, so its whole group never applies. Fix: name a bot token (e.g. GPTBot) or use * for all crawlers.
robots.txt Oversized	medium	usability	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Client requirements in Robots Exclusion Protocol. Bad: the file exceeds ~500 KB; Google and others stop reading there, silently dropping every later rule. Fix: trim the file (consolidate path patterns) so all rules fit under the limit.
robots.txt Unknown Directive	informational	nothing	specification: Robots Exclusion Protocol
Supporting documents: Robots Exclusion Protocol Robots Exclusion Protocol permits this, so it never fails the site. Informational: a directive reads like a misspelling (e.g. "Disalow" for "Disallow"), so the rule it may have meant is being ignored. Consider correcting the spelling; RFC 9309 permits records outside the protocol, so this does not fail robotsTxtValid and a deliberate extension record is fine.

Examples

Both run through the same checks as a live scan: the first passes, the second is flagged.

correcttext

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

present but wrongtext

Disallow: /private/

User-agent:
Disallow: /nowhere/

User-agent: *
Content-Signal: search=yes, ai-train=maybe, =no
Allow /
Disalow: /nope/

User-agent: GPT-Bot
Allow: /
Disallow: /

User-agent: CCBot
Disallow: /tmp/

User-agent: CCBot
Disallow: /docs/

User-agent: Bytespider

Specifications

Document	Revision	Kind
RFC 9309 — Robots Exclusion Protocol	Published RFC	specification
Google Search: robots.txt specification	Living document	vendor profile

Last re-read against the published documents: 2026-10-05.