robots.txt for this page

Which search and AI crawlers robots.txt lets fetch this page.

adoption Web baselinetrend ► flatchecked on every pagechecks 1verified 2026-10-16

Set up robots.txt for this page on your site How to see which crawlers robots.txt lets fetch a page, and how to stop it hiding the page's own noindex.

Free Let crawlers read your noindex

Why an agent cares

robots.txt is written for paths, and it is easy to block a crawler from a page by accident, or to block one from a page that carries a noindex Google then never reads. Which crawlers to admit is the site's choice, so that part is shown, never graded; only the hidden noindex is a fault, on Google's own rule. The free scanner checks it on the page it scans; a Lumar crawl does not, because it is not given robots.txt with each page.

Adoption

Web baseline. Standard equipment on the web. Its absence is the exception rather than its presence.

Direction of travel: ► flat, as at 2026-10-16.

robots.txt is a settled web baseline, and a noindex is read only by a crawler robots.txt lets in.

What we check

Absence is never a finding. Every check below runs only when this technology is detected, so a site that has not adopted this is not assessed on it and appears in no report. 1 of these 1 can fail it. In Lumar these appear on each page's own row.

Check	Severity	Raises	On whose authority
Noindex Hidden By robots.txt	low	usability	vendor profile: Google Search Central: Block search indexing with noindex, Google Search Central: Block search indexing with noindex
Supporting documents: Google Search Central: Block search indexing with noindex, Google Search Central: Block search indexing with noindex Related guidance in Google Search Central: Block search indexing with noindex and Google Search Central: Block search indexing with noindex. Bad: the page says noindex, but robots.txt blocks Googlebot from it, so Google never reads the noindex and can still list the address from links. Fix: allow Googlebot to fetch the page so it sees the noindex, or remove the noindex if blocking was the intent.

Check

Severity

Raises

On whose authority

Noindex Hidden By robots.txt

low

usability

vendor profile: Google Search Central: Block search indexing with noindex, Google Search Central: Block search indexing with noindex

Specifications

Document	Revision	Kind
Google Search Central: Block search indexing with noindex	Living document	vendor profile
RFC 9309 — Robots Exclusion Protocol	Published RFC	specification

Last re-read against the published documents: 2026-10-16.