# Access

What has the site said agents may do with its content?

Permission, expressed in files agents actually fetch. Adoption here spans the whole range, from robots.txt on effectively every site to opt-out vocabularies with a handful of publishers, and a structural mistake in any of them means a rule that applies to nobody.

| Technology | What it is | Adoption | Status |
|---|---|---|---|
| [robots.txt](/docs/access/robots-txt) | The crawl-policy file every agent reads first. | Web baseline, ► flat | scored |
| [AI-bot rules](/docs/access/per-bot-rule) | Named rules for GPTBot, ClaudeBot and friends, rather than one blanket policy. | Platform default, ▲ rising | scored |
| [Content Signals](/docs/access/content-signal) | Cloudflare's robots.txt vocabulary for search, AI input and AI training. | Platform default, ▲ rising | scored |
| [ai.txt](/docs/access/ai-txt) | Attribution preferences for AI systems that use the content. | Specification only, ► flat | scored |
| [TDMRep](/docs/access/tdm-rep) | The W3C text and data mining reservation, used mainly by publishers. | Early production, ▲ rising | scored |
| [AIPREF Content-Usage](/docs/access/ai-pref) | The IETF's standards-track vocabulary for stating how content may be used by AI systems. | Specification only, ► flat | watched |