agentic‑readiness docs
/
Scan at scale Blog Build one › Run a scan ›
Menu

BlogState of the agentic web

State of the agentic web

We scanned a sample of 500,000 of the million websites Chrome loads most: what they publish for AI agents, and how often they get it wrong.

A dark field of hundreds of faint documents, a few labelled robots.txt, llms.txt and sitemap.xml, with threads drawing out of the crowd and converging into four lit cards: openapi, ucp, agent.json and an agent card.

What we found

Every figure here is an estimate for the whole top million, made from our sample of 500,000. Hover over a chart row to see its margin of error.

Few of the top million websites let an AI agent do anything

One dot stands for 1,000 websites. Which dots are lit is drawn from our sample's estimates, not from checking each site.

The top million websites

959,000

websites, by our estimate, once names that are not websites are taken out of Chrome's list.

Rules for AI crawlers

14%

of these websites name at least one AI crawler in robots.txt, the file that tells crawlers what they may read.

Something an agent can do

7.1%

of these websites let an AI agent do something on them: buy a product, call an API, or talk to the site's own agent.

Mostly Shopify stores

82%

of the websites an agent can do something on are Shopify stores, each with a shopping profile for agents.

More than a shop

0.9%

of these websites offer an agent more than a shopping profile: an API description, an MCP server or an agent of their own. That is fewer than 1 in 100.

Shopify stores are also a large share of the websites that publish some of the files agents read.

Shopify stores among the websites that publish each file

  1. UCP profile92%
  2. Markdown answer80%
  3. llms.txt≥45%
Of the websites that publish each file, the estimated share that are Shopify stores, for every file where they are at least a quarter. We checked every website with a UCP profile or a Markdown answer for Shopify, so for any other file the share is a floor, marked ≥: a Shopify store with neither was not identified.

Most rules for AI crawlers are about training, not about answering people

Of the websites with a rule for an AI crawler, 69% name GPTBot and 63% name ClaudeBot, which collect pages to train AI models. Far fewer name the crawlers that build AI search or fetch a page when someone asks an assistant: 19% name OAI-SearchBot and 8.6% name Claude-SearchBot.

A rule can let a crawler in or keep it out. We counted the rules, not which way they point.

AI crawlers named in robots.txt

  1. Named byWhat it is for
  2. gptbot69%trains models
  3. claudebot63%trains models
  4. ccbot58%open archive
  5. google-extended55%usage rule
  6. bytespider53%trains models
  7. amazonbot53%trains models
  8. applebot-extended48%usage rule
  9. anthropic-ai47%undocumented
  10. meta-externalagent43%trains models
  11. claude-web42%undocumented
  12. cohere-ai40%for a person
  13. omgili36%data for sale
  14. chatgpt-user31%for a person
  15. perplexitybot29%search index
  16. oai-searchbot19%search index
  17. claude-searchbot8.6%search index
  18. claude-user7.5%for a person
  19. perplexity-user6.3%for a person
  20. duckassistbot4.9%for a person
  21. mistralai-user1.5%for a person
Share of the sites that name at least one AI crawler, an estimated 140,000 of the top million. What each crawler is for is its operator's own description, or the ai.robots.txt project's where the operator publishes none.

Shopify stores send a web page when an AI agent asks for plain text

An AI agent can ask a website for a page as plain text, in a format called Markdown. Plain text is shorter than a web page, so it is quicker and cheaper for the agent to read. 7.1% of the websites answer that request.

One dot stands for 100 websites that answer a request for plain text. Which dots are lit is drawn from our sample's estimates, not from checking each site.

They answer

68,300

websites answer when an AI agent asks for a page as plain text.

Mostly Shopify stores

80%

of these websites are Shopify stores.

A web page instead

81%

of these websites send their normal web page, labelled as plain text: nearly the same dots. 99.7% of the Shopify stores do it, against 1.7% of the other websites.

Plain text

13,500

websites send an answer that is not a web page.

Websites that send a web page instead, Shopify against the rest

  1. Shopify stores99.7%
  2. Other websites1.7%
Of the websites that answer a request for plain text, the estimated share that send their web page instead. Every website that answers was checked for Shopify.

Check your own site

Ask for your home page the way an agent does, and look at the first line of what comes back:

curl -s -H "Accept: text/markdown" https://example.com/ | head -n 1

If that line is <!doctype html>, your site sent a web page. If you build agents, check what a reply contains, not only what it says it is.

How we identified Shopify stores

On 26 October 2026 we asked each of the 31,482 sampled websites that answer a request for plain text or publish a shopping profile for its home page once. We counted a site as a Shopify store when the answer carried Shopify's own headers or its storefront code. 31,313 answered; the rest are left out of both groups. We also opened 30 of the stores by hand: in every one, the plain-text answer was the home page, with its menus and scripts.

After the Markdown answer, the OpenAPI description is the file most often wrong: 46% have a mistake

The Markdown answer is wrong most often, and almost only on Shopify: 99.7% of Shopify stores send a web page instead, against 1.7% of other websites. After that comes the OpenAPI description: of the websites that publish one, 46% have at least one mistake in it, most often that the file leaves an operation with no description.

Files with at least one mistake

  7. MCP server card6.5%
  12. Content Signals0.5%
Of the websites that publish each file, the estimated share with at least one mistake in it. Open a row for its mistakes seen on at least 50 sampled sites; hover for the margin of error.
Files too new for the sample

A2A agent card, Web Bot Auth directory, NLWeb endpoint, NLWeb endpoint, ai.txt and Personal Agent Protocol are too rare in the top million for a sample to measure. So we also scanned 16,486 websites listed in public directories of them. They chose to be listed, so these shares describe them, not the top million.

Listed websites with at least one mistake

  1. NLWeb endpoint41% · 60 of 145
  2. NLWeb endpoint (REST)34% · 44 of 128
  3. Web Bot Auth directory24% · 56 of 232
  4. A2A agent card23% · 372 of 1,640
  5. Personal Agent Protocol5.5% · 3 of 55
  6. ai.txt1.3% · 1 of 77
Websites listed in public directories of these files, scanned separately. Each share is of the listed websites that publish the file, not of the top million. Shown where at least 50 could be read.

Our checks found an exposed key or credential on 1 sampled website

Among the websites listed in public directories, 2 published a private key and 9 a credential. We did not try them, and we name none.

A few sampled websites publish a file that skips a protection its own standard asks for.

Files that skip a security protection

  1. llms.txt over plain http157 sites
  2. API server over plain http115 sites
  3. Sign-in server mismatch30 sites
  4. Sign-in token in the web address16 sites
  5. Sign-in details for another server15 sites
  6. Retired sign-in method12 sites
  7. API server on a private network11 sites
  8. Schema from the wrong domain5 sites
  9. API catalog link over plain http5 sites
Numbers of sampled websites, not estimates for the top million. 13 more findings are on checks seen on fewer than five websites each, counted together so that no row points at one site.

We name no website. A finding says the file breaks a rule. It does not say anyone was harmed.

All the numbers

Every technology we looked for, and every industry large enough to measure.

Every technology

Share of the top million websites that publish each one

  1. Rules: what an agent may do

    1. robots.txt70%
    2. Rules for AI crawlers14%
    3. Content Signals3.3%
    4. ai.txt384 sites in sample
    5. TDMRep266 sites in sample
    6. RSL (Really Simple Licensing)38 sites in sample

  2. Reading: help to understand the site

    1. XML sitemap48%
    2. llms.txt13%
    3. Markdown answer7.1%

  3. Acting: something an agent can use

    1. UCP profile6.3%
    2. OpenAPI description0.3%
    3. MCP server card0.2%
    4. MCP authorization0.2%
    5. API catalog0.2%
    6. Agentic Resource Discovery (ARD)0.3%
    7. A2A agent card166 sites in sample
    8. NLWeb endpoint10 sites in sample
    9. Personal Agent Protocol11 sites in sample

    Together, 7.1% of websites offer at least one of these. Leave out shop profiles and it is 0.9%.

Estimated share of the top million websites that publish each one, on one scale from nothing to every site. A row with no bar was too rare in our sample to estimate, so it gives how many of the sites we scanned publish it.
By industry

Share of each industry's websites that publish each one

Swipe sideways for every column.

robots.txtsitemapAI crawler rulesllms.txtMarkdownContent-SignalsUCPOpenAPIMCP cardMCP authTDMRepai.txtagent cardNLWebNews or publisher91743212Community or social815529Marketplace or classifieds86622413Reference or directory84622410Gambling513116Online shop928015453635Entertainment or media785615Local or physical business81651116Nonprofit or association765211Company or brand76601014Software or SaaS602988Health7152813Finance62429Education5831Government or public body4824
Estimated share of each industry's websites, in the same order as the figure above. Tap or hover a cell for what the technology is and the margin on the figure. Numbers appear above 8%; shading is square-root scaled so the small shares stay visible.

How we measured

The websites. We started from the Chrome UX Report's top million for September 2026: the websites with the most page loads in Chrome. It counts only Chrome users who share usage statistics, on computers and Android phones, so Safari and Chrome on iPhone are not in it. We took out names that are not websites, such as name servers and parked domains, which leaves an estimated 959,000 websites.

The sample. We scanned 500,000 websites between 22 October 2026 and 26 October 2026: all of the 100,000 busiest, and up to 201,866 chosen at random from each band below them. We then weighted each website by how many it stands for, so every figure is an estimate for the top million, with a 95% margin of error on hover. We give a share only when at least 400 sites in the sample publish it; below that we give the count.

Websites we could not read. A website that turned us away or never answered is left out of every figure, rather than counted as publishing nothing. We told every website who we were. In a trial on 36 websites, 8 turned away a crawler posing as Google and 2 turned away ours.

Websites we could and could not read

  1. Answered normally83%
  2. Answered, with an error2.3%
  3. Turned us away9.8%
  4. Never answered4.9%
Estimated share of the top million websites in each state when we asked for their home page.

Rules for AI crawlers. A website counts when its robots.txt names at least one of the 192 AI crawlers on the ai.robots.txt list, as it stood on 23 October 2026. About 2.1 times as many websites name some crawler; the others name only crawlers not on that list, such as search engines' and SEO tools'.

Shopify stores. We checked for Shopify only on the sampled websites that answer a request for plain text or publish a shopping profile, and counted a site only on Shopify's own headers or storefront code. So the Shopify figures cover those two things, and Shopify stores that publish neither are not counted as Shopify.

The checks. These are the checks Lumar runs on a customer's own site. We fetch the files; no page is rendered and no JavaScript is run. We check for 198 kinds of mistake. 90 break a rule in a published standard; the other 108 are allowed by the standard but leave an agent unable to use the file. A language model put each website in an industry from its home page; we have not yet checked those labels by hand.

Not comparable with our first edition. That edition, in September 2026, used a different list (Tranco, which ranks names by how often they are looked up, not by page loads). It also counted a robots.txt rule for any named crawler as a rule for AI, which overstated that figure. A difference between the two editions is a different measurement, not a trend.

Using these figures. They are published under CC BY 4.0. Quote them and build on them, and credit Lumar with a link. The Chrome UX Report is Google's public dataset of page loads in Chrome.

What to do

  1. Check what your site sends an AI agent. Ask for your home page as plain text, as shown above. If a web page comes back, the plain-text answer is not working.
  2. Name the crawlers you mean. A rule for GPTBot or ClaudeBot does not cover OAI-SearchBot, Claude-SearchBot or Claude-User. Each is a separate crawler with its own name.
  3. Check your OpenAPI description. 46% have a mistake, most often that it leaves an operation with no description.
  4. If you build agents, check what a reply contains, not only what it says it is.

Our scanner runs every check in this study on your own site, free.

Run a scanBuild an agent-ready fileRead the reference