# robots.txt

The crawl-policy file every agent reads first.

**[Access](/docs/access)** · checked once per host (item type `access`) · 4 checks that can fail it

It is the only access declaration essentially all crawlers honour, so a structural mistake in it applies to every agent at once. A stray directive before the first `User-agent` group, or a rule in the wrong group, silently applies to nobody.

## Adoption

**Web baseline** (L4)

Standard equipment on the web. Its absence is the exception rather than its presence. Direction of travel: ► flat, as of 2026-07-28.

RFC 9309 standardised what had been universal practice since 1994.

## What we check

Absence is never a finding: every check below runs only once `robotsTxtPresent` is true, so a site that has not
adopted this is not assessed on it and appears in no report.

| Check | Kind | Raises | On whose authority |
|---|---|---|---|
| `robotsTxtDirectiveBeforeGroup` | high severity | conformance, usability | specification (RFC 9309) |
| `robotsTxtMissingColon` | medium severity | conformance, usability | specification (RFC 9309) |
| `robotsTxtEmptyUserAgent` | medium severity | conformance, usability | specification (RFC 9309) |
| `robotsTxtOversized` | medium severity | usability | specification (RFC 9309) |
| `robotsTxtUnknownDirective` | informational | nothing | specification (RFC 9309) |

## Examples

These are the fixtures the test suite runs through the real collector, not snippets written for a document.

- Correct: `test/e2e/test-pages/access/correct.robots.txt`
- Present but wrong: `test/e2e/test-pages/access/issues.robots.txt`

## Specifications

| Document | Revision | Kind |
|---|---|---|
| [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) | RFC 9309 | specification |
| [Google Search: robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) | living document | vendor profile |

Last re-read against the published documents: 2026-08-12.