# Release notes, 16 October 2026

Each scanned page now shows which crawlers robots.txt blocks from it, and whether it hides a noindex from Google.

![A release tag dated 16 October 2026 listing two new checks, a hidden noindex and user-triggered fetchers, beside the Lumar mark.](https://agentic-readiness.lumar.io/assets/hero-release-2026-10-16.webp)

## Summary

A page scan now reads robots.txt against the page itself: which crawlers may fetch it, and whether a noindex on it is one Google can never read. If your robots.txt disallows ChatGPT-User, Perplexity-User, Meta-ExternalFetcher or one of Google's user-triggered fetchers, your results also say that rule may not keep that assistant out. Scan a page again to see both. The command-line tool has the robots.txt note from version 0.3.6; the page check is in the free scanner, because a Lumar crawl is not given robots.txt with each page.

## New checks

-   Every scanned page now shows which search and AI crawlers robots.txt blocks from it. That list is never graded. One case is a fault: a page that says noindex while robots.txt blocks Googlebot from it. [Google says](https://developers.google.com/search/docs/crawling-indexing/block-indexing) such a page "must not be blocked by a robots.txt file", or the crawler "will never see the noindex rule".
-   robots.txt: a note when a Disallow names a fetcher that loads a page only because a person asked an assistant to, such as ChatGPT-User or Perplexity-User. Their operators say robots.txt may not apply to those fetches, so the rule may not keep them out. The note never fails a site.

## What the operators say

-   [OpenAI](https://developers.openai.com/api/docs/bots) on ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.”
-   [Perplexity](https://docs.perplexity.ai/guides/bots) on Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
-   [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) on Meta-ExternalFetcher: it “may bypass robots.txt rules” because its fetches are requested by a user.
-   [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) on its user-triggered fetchers: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”
-   Anthropic says its bots honour robots.txt, so a rule for Claude-User gets no note.

## Evidence

We read robots.txt from the 2,000 most visited sites on 16 October. 1,170 served one, and 185 of those disallow at least one of these fetchers. That is a choice those sites made on purpose, so it is a note and not a fault. To keep an assistant out of search answers, the operator's search crawler, such as OAI-SearchBot or PerplexityBot, is the one robots.txt controls. To keep a fetcher out completely, block it at the server.

Our own robots.txt named these fetchers too. It no longer does: the rule for every crawler already covers them.

For the hidden noindex we took paths that robots.txt disallows on the 400 most visited sites and read those pages. Of the 130 we could judge, 16 carried a noindex that Google is blocked from reading, mostly search, login and error pages. We checked each one by hand against the live robots.txt and page. It is graded low: the usual cost is an address listed without its content.

[Run a scan](https://agentic-readiness.lumar.io/)[Build an agent-ready file](https://agentic-readiness.lumar.io/build)[Read the reference](https://agentic-readiness.lumar.io/docs)