# State of the agentic web We scanned a sample of 500,000 of the [million websites Chrome loads most](https://github.com/zakird/crux-top-lists): what they publish for AI agents, and how often they get it wrong. ![A dark field of hundreds of faint documents, a few labelled robots.txt, llms.txt and sitemap.xml, with threads drawing out of the crowd and converging into four lit cards: openapi, ucp, agent.json and an agent card.](https://agentic-readiness.lumar.io/assets/hero-agentic-web.webp) ## What we found Every figure here is an estimate for the whole top million, made from our sample of 500,000. Hover over a chart row to see its margin of error. - [14%of the websites name an AI crawler in their robots.txt file.](#scene-glance-step-2) - [7.1%of the websites let an AI agent do something on them, such as buy a product.](#scene-glance-step-3) - [82%of the websites an AI agent can do something on are Shopify stores.](#scene-glance-step-4) - [81%of the websites that answer an AI agent's request for plain text send a web page instead: 99.7% of Shopify stores, against 1.7% of other websites.](#scene-markdown-step-3) ## Few of the top million websites let an AI agent do anything One dot stands for 1,000 websites. Which dots are lit is drawn from our sample's estimates, not from checking each site. 1 of 5 ### The top million websites 959,000 websites, by our estimate, once names that are not websites are taken out of Chrome's list. 2 of 5 ### Rules for AI crawlers 14% of these websites name at least one AI crawler in robots.txt, the file that tells crawlers what they may read. 3 of 5 ### Something an agent can do 7.1% of these websites let an AI agent do something on them: buy a product, call an API, or talk to the site's own agent. 4 of 5 ### Mostly Shopify stores 82% of the websites an agent can do something on are Shopify stores, each with a shopping profile for agents. 5 of 5 ### More than a shop 0.9% of these websites offer an agent more than a shopping profile: an API description, an MCP server or an agent of their own. That is fewer than 1 in 100. Shopify stores are also a large share of the websites that publish some of the files agents read. ### Shopify stores among the websites that publish each file 1. UCP profile92% 2. Markdown answer80% 3. llms.txt≥45% Of the websites that publish each file, the estimated share that are Shopify stores, for every file where they are at least a quarter. We checked every website with a UCP profile or a Markdown answer for Shopify, so for any other file the share is a floor, marked ≥: a Shopify store with neither was not identified. ## Most rules for AI crawlers are about training, not about answering people Of the websites with a rule for an AI crawler, **69%** name GPTBot and **63%** name ClaudeBot, which collect pages to train AI models. Far fewer name the crawlers that build AI search or fetch a page when someone asks an assistant: **19%** name OAI-SearchBot and **8.6%** name Claude-SearchBot. A rule can let a crawler in or keep it out. We counted the rules, not which way they point. ### AI crawlers named in robots.txt 1. Named byWhat it is for 2. gptbot69%trains models 3. claudebot63%trains models 4. ccbot58%open archive 5. google-extended55%usage rule 6. bytespider53%trains models 7. amazonbot53%trains models 8. applebot-extended48%usage rule 9. anthropic-ai47%undocumented 10. meta-externalagent43%trains models 11. claude-web42%undocumented 12. cohere-ai40%for a person 13. omgili36%data for sale 14. chatgpt-user31%for a person 15. perplexitybot29%search index 16. oai-searchbot19%search index 17. claude-searchbot8.6%search index 18. claude-user7.5%for a person 19. perplexity-user6.3%for a person 20. duckassistbot4.9%for a person 21. mistralai-user1.5%for a person Share of the sites that name at least one AI crawler, an estimated 140,000 of the top million. What each crawler is for is its operator's own description, or the ai.robots.txt project's where the operator publishes none. ## Shopify stores send a web page when an AI agent asks for plain text An AI agent can ask a website for a page as plain text, in a format called Markdown. Plain text is shorter than a web page, so it is quicker and cheaper for the agent to read. **7.1%** of the websites answer that request. One dot stands for 100 websites that answer a request for plain text. Which dots are lit is drawn from our sample's estimates, not from checking each site. 1 of 4 ### They answer 68,300 websites answer when an AI agent asks for a page as plain text. 2 of 4 ### Mostly Shopify stores 80% of these websites are Shopify stores. 3 of 4 ### A web page instead 81% of these websites send their normal web page, labelled as plain text: nearly the same dots. **99.7%** of the Shopify stores do it, against **1.7%** of the other websites. 4 of 4 ### Plain text 13,500 websites send an answer that is not a web page. ### Websites that send a web page instead, Shopify against the rest 1. Shopify stores99.7% 2. Other websites1.7% Of the websites that answer a request for plain text, the estimated share that send their web page instead. Every website that answers was checked for Shopify. ### Check your own site Ask for your home page the way an agent does, and look at the first line of what comes back: curl -s -H "Accept: text/markdown" https://example.com/ | head -n 1 If that line is ``, your site sent a web page. If you build agents, check what a reply contains, not only what it says it is. How we identified Shopify stores On 26 October 2026 we asked each of the 31,482 sampled websites that answer a request for plain text or publish a shopping profile for its home page once. We counted a site as a Shopify store when the answer carried Shopify's own headers or its storefront code. 31,313 answered; the rest are left out of both groups. We also opened 30 of the stores by hand: in every one, the plain-text answer was the home page, with its menus and scripts. ## After the Markdown answer, the OpenAPI description is the file most often wrong: 46% have a mistake The Markdown answer is wrong most often, and almost only on Shopify: **99.7%** of Shopify stores send a web page instead, against **1.7%** of other websites. After that comes the OpenAPI description: of the websites that publish one, **46%** have at least one mistake in it, most often that the file leaves an operation with no description. ### Files with at least one mistake 1. 1. Sends its web page where Markdown was asked for80% 2. Lets a shared cache hand the Markdown to browsers0.2% 2. 1. Leaves an operation with no description20% 2. Gives two operations the same name17% 3. Leaves an operation with no name to call it by14% 4. Sends agents to an API server on plain http8.5% 5. Describes a value without saying what type it is7.8% 6. Declares a version that is not a real OpenAPI version6.5% 3. 1. Gives an entry an identifier in the wrong form23% 2. Leaves an entry with no name to show people4.0% 4. 1. Asks an agent to sign in without saying where12% 5. 1. Is served as the wrong type of file9.5% 6. 1. Lists pages on another domain5.1% 2. Missing the XML namespace2.2% 3. Is not a sitemap at all1.7% 4. Uses relative URLs1.1% 5. Is declared in robots.txt as a path, not a full address0.6% 6. Is over the size limit0.3% 7. Is an empty file0.2% 8. Will not parse0.2% 7. MCP server card6.5% 8. 1. Starts with text before its title4.8% 2. Has no title at all0.7% 3. Hides a heading inside a collapsed block0.4% 4. Links the site's pages over plain http0.3% 9. 1. Leaves out a registry the profile requires1.3% 2. Is served with caching rules the standard does not allow0.8% 3. Names no way to reach it0.3% 10. 1. Has a line missing its colon0.5% 2. Puts a rule before any User-agent line0.3% 11. 1. Allows and disallows the same path0.4% 2. Misspells a crawler's name0.1% 12. Content Signals0.5% Of the websites that publish each file, the estimated share with at least one mistake in it. Open a row for its mistakes seen on at least 50 sampled sites; hover for the margin of error. Files too new for the sample A2A agent card, Web Bot Auth directory, NLWeb endpoint, NLWeb endpoint, ai.txt and Personal Agent Protocol are too rare in the top million for a sample to measure. So we also scanned 16,486 websites listed in public directories of them. They chose to be listed, so these shares describe them, not the top million. ### Listed websites with at least one mistake 1. NLWeb endpoint41% · 60 of 145 2. NLWeb endpoint (REST)34% · 44 of 128 3. Web Bot Auth directory24% · 56 of 232 4. A2A agent card23% · 372 of 1,640 5. Personal Agent Protocol5.5% · 3 of 55 6. ai.txt1.3% · 1 of 77 Websites listed in public directories of these files, scanned separately. Each share is of the listed websites that publish the file, not of the top million. Shown where at least 50 could be read. ## Our checks found an exposed key or credential on 1 sampled website Among the websites listed in public directories, **2** published a private key and **9** a credential. We did not try them, and we name none. A few sampled websites publish a file that skips a protection its own standard asks for. ### Files that skip a security protection 1. llms.txt over plain http157 sites 2. API server over plain http115 sites 3. Sign-in server mismatch30 sites 4. Sign-in token in the web address16 sites 5. Sign-in details for another server15 sites 6. Retired sign-in method12 sites 7. API server on a private network11 sites 8. Schema from the wrong domain5 sites 9. API catalog link over plain http5 sites Numbers of sampled websites, not estimates for the top million. 13 more findings are on checks seen on fewer than five websites each, counted together so that no row points at one site. We name no website. A finding says the file breaks a rule. It does not say anyone was harmed. ## All the numbers Every technology we looked for, and every industry large enough to measure. Every technology ### Share of the top million websites that publish each one 1. Rules: what an agent may do 1. robots.txt70% 2. Rules for AI crawlers14% 3. Content Signals3.3% 4. ai.txt384 sites in sample 5. TDMRep266 sites in sample 6. RSL (Really Simple Licensing)38 sites in sample 2. Reading: help to understand the site 1. XML sitemap48% 2. llms.txt13% 3. Markdown answer7.1% 3. Acting: something an agent can use 1. UCP profile6.3% 2. OpenAPI description0.3% 3. MCP server card0.2% 4. MCP authorization0.2% 5. API catalog0.2% 6. Agentic Resource Discovery (ARD)0.3% 7. A2A agent card166 sites in sample 8. NLWeb endpoint10 sites in sample 9. Personal Agent Protocol11 sites in sample Together, **7.1%** of websites offer at least one of these. Leave out shop profiles and it is **0.9%**. Estimated share of the top million websites that publish each one, on one scale from nothing to every site. A row with no bar was too rare in our sample to estimate, so it gives how many of the sites we scanned publish it. By industry ### Share of each industry's websites that publish each one Swipe sideways for every column. Estimated share of each industry's websites, in the same order as the figure above. Tap or hover a cell for what the technology is and the margin on the figure. Numbers appear above 8%; shading is square-root scaled so the small shares stay visible. ## How we measured **The websites.** We started from [the Chrome UX Report's top million](https://github.com/zakird/crux-top-lists) for September 2026: the websites with the most page loads in Chrome. It counts only Chrome users who share usage statistics, on computers and Android phones, so Safari and Chrome on iPhone are not in it. We took out names that are not websites, such as name servers and parked domains, which leaves an estimated **959,000** websites. **The sample.** We scanned **500,000 websites** between 22 October 2026 and 26 October 2026: all of the 100,000 busiest, and up to 201,866 chosen at random from each band below them. We then weighted each website by how many it stands for, so every figure is an estimate for the top million, with a 95% margin of error on hover. We give a share only when at least **400** sites in the sample publish it; below that we give the count. **Websites we could not read.** A website that turned us away or never answered is left out of every figure, rather than counted as publishing nothing. We told every website who we were. In a trial on 36 websites, 8 turned away a crawler posing as Google and 2 turned away ours. ### Websites we could and could not read 1. Answered normally83% 2. Answered, with an error2.3% 3. Turned us away9.8% 4. Never answered4.9% Estimated share of the top million websites in each state when we asked for their home page. **Rules for AI crawlers.** A website counts when its robots.txt names at least one of the 192 AI crawlers on the ai.robots.txt list, as it stood on 23 October 2026. About **2.1 times** as many websites name some crawler; the others name only crawlers not on that list, such as search engines' and SEO tools'. **Shopify stores.** We checked for Shopify only on the sampled websites that answer a request for plain text or publish a shopping profile, and counted a site only on Shopify's own headers or storefront code. So the Shopify figures cover those two things, and Shopify stores that publish neither are not counted as Shopify. **The checks.** These are the checks Lumar runs on a customer's own site. We fetch the files; no page is rendered and no JavaScript is run. We check for 198 kinds of mistake. 90 break a rule in a published standard; the other 108 are allowed by the standard but leave an agent unable to use the file. A language model put each website in an industry from its home page; we have not yet checked those labels by hand. **Not comparable with our first edition.** That edition, in September 2026, used a different list (Tranco, which ranks names by how often they are looked up, not by page loads). It also counted a robots.txt rule for any named crawler as a rule for AI, which overstated that figure. A difference between the two editions is a different measurement, not a trend. **Using these figures.** They are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Quote them and build on them, and credit Lumar with a link. [The Chrome UX Report](https://developer.chrome.com/docs/crux) is Google's public dataset of page loads in Chrome. ## What to do 1. **Check what your site sends an AI agent.** Ask for your home page as plain text, as shown above. If a web page comes back, the plain-text answer is not working. 2. **Name the crawlers you mean.** A rule for GPTBot or ClaudeBot does not cover OAI-SearchBot, Claude-SearchBot or Claude-User. Each is a separate crawler with its own name. 3. **Check your OpenAPI description.** 46% have a mistake, most often that it leaves an operation with no description. 4. **If you build agents,** check what a reply contains, not only what it says it is. Our scanner runs every check in this study on your own site, free. [Run a scan](https://agentic-readiness.lumar.io/)[Build an agent-ready file](https://agentic-readiness.lumar.io/build)[Read the reference](https://agentic-readiness.lumar.io/docs)