State of the agentic web
We fetched the agent-facing files and endpoints of 150,074 hosts drawn from the Tranco list. This is what the sites that published them got wrong.
- 34,160sites have written a rule about AI crawlers
- 3,007have published an interface an agent can call
- 1 in 6sitemaps has a fault no submission fixes