DocsAccess
Access
What has the site said agents may do with its content?
Permission, expressed in files agents actually fetch. Adoption here spans the whole range, from robots.txt on effectively every site to opt-out vocabularies with a handful of publishers, and a structural mistake in any of them means a rule that applies to nobody.
Scored
Detected and validated. Each page lists the checks that can fail it and on whose authority.
robots.txt web baseline ► flat
The crawl-policy file every agent reads first.
AI-bot rules platform default ▲ rising
Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy.
Content Signals platform default ▲ rising
Cloudflare's robots.txt vocabulary for search, AI input and AI training.
ai.txt specification only ► flat
Attribution preferences for AI systems that use the content.
TDMRep early production ▲ rising
The W3C text and data mining reservation, used mainly by publishers.
Watched
Followed but not scored: each page says what would change that.