# robots.txt for this page

Which search and AI crawlers robots.txt lets fetch this page.

**[Access](/docs/access)** · checked on every URL · 1 checks that can fail it

robots.txt is written for paths, and it is easy to block a crawler from a page by accident, or to block one from a page that carries a noindex Google then never reads. Which crawlers to admit is the site's choice, so that part is shown, never graded; only the hidden noindex is a fault, on Google's own rule. The free scanner checks it on the page it scans; a Lumar crawl does not, because it is not given robots.txt with each page.

## Adoption

**Web baseline** (L4)

Standard equipment on the web. Its absence is the exception rather than its presence. Direction of travel: ► flat, as of 2026-10-16.

robots.txt is a settled web baseline, and a noindex is read only by a crawler robots.txt lets in.

## What we check

Absence is never a finding: every check below runs only when this technology is detected, so a site that has not
adopted this is not assessed on it and appears in no report.

| Check | Kind | Raises | On whose authority |
|---|---|---|---|
| Noindex Hidden By robots.txt | low severity | usability | vendor profile: [Google Search Central: Block search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing), [Google Search Central: Block search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) |

## Specifications

| Document | Revision | Kind |
|---|---|---|
| [Google Search Central: Block search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) | Living document | vendor profile |
| [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) | Published RFC | specification |

Last re-read against the published documents: 2026-10-16.