AI Agent Readiness: What We Learned Scoring Our Own Site

RedHub AI Editorial10 min read

A woman stands at a glowing vending machine full of identical unmarked white cartons, hand raised, choosing nothing.
Jump to a section10

AI agent readiness is whether an AI agent can find your site, read what it sells, and act on it without a person in the loop. It is decided by about seven files, most of which take an afternoon. We built a free scorecard to measure it, ran it against our own site, and spent four days fixing what it found.

The short version. Seven surfaces decide whether agents can use your site: an llms.txt, a Content Signal in your robots file, a valid sitemap, product data in JSON-LD, a machine-readable commerce surface, a .well-known directory, and a plain-text mirror of your content. Our own site started at 61 out of 100 and finished at 94. The first number the tool printed was 39, and that one was a bug in the tool, not the site.

What an agent does when it visits you

A person landing on a product page reads a headline, looks at a picture, and decides. An agent does none of that. It fetches a small number of files at predictable addresses, and if those files are missing it either guesses from your HTML or moves on to a competitor whose files are present.

That is the whole mechanism, and it is why agent readiness is a checklist, not a strategy. The seven surfaces below are what a scorecard looks at.

SurfaceWhat it answersGood looks like
/llms.txtWhat is this site and where is everything?A heading and real links, not a stub
robots.txt Content SignalWhat may be done with this content after crawling?An explicit Content-Signal: line
/sitemap.xmlWhich URLs exist?Parses, and includes product URLs
JSON-LD ProductWhat is for sale and what does it cost?An offers block with a price
Machine commerce surfaceCan this be read or bought programmatically?A product feed, or an agent checkout endpoint
/.well-known/Does this site expose anything callable?An MCP server card, if one exists
Markdown mirrorCan the content be read without parsing HTML?Content negotiation, or an llms-full.txt

Not all seven apply to every site. A blog has nothing to say about checkout, and scoring it against commerce checks produces a low number that means nothing. That distinction turned out to matter more than we expected.

The first number was wrong

The first run put redhub.ai at 39 out of 100, verdict DARK TO AGENTS. That was a bad enough result that we checked it instead of accepting it, which is how we found that the form only asked for the homepage.

The number was an artifact of our own form, not a reading of the site.

Most sites keep their product schema on product pages. Ours does. Scoring a storefront from its homepage alone marks it zero on the heaviest check in the model, and the form gave you nowhere to paste a product page. Adding that one field moved the same site from 39 to 61 without changing a byte of the site itself, which is the tell: a score that moves when only the questionnaire changed was never measuring the site.

So 61 is where this actually starts.

A scorecard whose entire pitch is that other scorecards produce misleading numbers cannot ship a form that produces a misleading number. That was the first thing we fixed.

The second defect was worse and took longer to see. The form hardcoded the .well-known input to empty and never offered a box for it, so that check read 0 out of 2 for every site the tool could ever score, including sites that publish those files. It went unnoticed for a simple reason: our own zero was correct. We had nothing at /.well-known/. Where your own answer happens to be right is exactly where a broken input is invisible.

The top grade was hiding holes

With the form fixed, we ran the scoring model across its whole input space to see how it behaved. Of the cases that earned the top verdict, 48 percent had at least one applicable check sitting at zero. A site could be told agents can find, read, and act on it while an entire surface was missing.

That is an averaging problem, and it is the failure every deterministic gate we build exists to prevent. A weighted score lets strength in six places paper over an absence in the seventh. So the scorecard now runs a second gate: a top-band score is held at the middle verdict while any applicable check reads zero, and the result names which surface is dark.

The fix demoted 64 cases, all of them scoring between 75 and 81, and 56 of those were missing a Content Signal entirely. That last number is what convinced us the gate was discriminating and not blunt. A site that has never declared what agents may do with its content is not a site that has earned a top grade.

Then we fixed the site

With a scorecard we trusted, 61 was a real number and it was ours to fix. Four changes, in the order we made them.

  1. A Content Signal in robots.txt. One line declaring that we want to be indexed and cited, and do not want to be training data. It is advisory, not enforceable, and it is still the only place to say it. This took the score from 61 to 72.
  2. A machine product feed. Every product with a stable ID, price, category, canonical URL, and image, at /products.json. This was the biggest single move, from 72 to 83.
  3. A security.txt. Four lines under /.well-known/, published because a company selling security-adjacent tooling with no route for reporting a vulnerability is a gap on its own terms. It took us to 89.
  4. Markdown mirrors. The same URL now returns markdown to a client that asks for it by name and HTML to everyone else, for the storefront home, every product page and every post. This is the one we had written off as a scheduling call; it turned out to be an afternoon, and it took us to 94.

The feed is worth one more sentence, because how you build it matters more than whether you have one. A feed exported on a schedule drifts from the site between exports, and a feed that contradicts its own product page is worse than no feed at all. Ours is generated at build time from the same two files the product pages render from, so the two cannot disagree. We verified that instead of assuming it, by sampling 20 products and comparing the feed price against each page's structured data.

What we did not fix

One check still sits at half marks, and it stays there deliberately.

The .well-known check goes to full marks if you publish an MCP server card. We do not run an MCP server. Publishing a card describing an endpoint that does not exist would have bought ten points and made the file a lie, which is the exact behavior our products are built to catch. It moves the day we build one.

94 out of 100 with one honest gap beats 100 with a fabricated capability. If your readiness score is perfect, check whether you earned it or declared it.

One more correction

Writing this post turned up a claim on our own product page that we had not verified. We had described Cloudflare's Agent Readiness Score as running commerce checks against every site, capping content sites at roughly half marks. Cloudflare's launch post says the opposite in plain words: it checks the agentic commerce standards "but these do not currently count towards the score." A content site is not penalized for missing them, so the mechanism we described does not exist.

The page is corrected. The real difference between the two tools survives it, and it is a difference in question, not in quality. Cloudflare scans your live site and asks which standards it supports. Ours asks how much of what applies to your kind of site is in place, which is why it makes you declare what you are first. Run both. They disagree in useful ways.

The same site scores 94 here and 29 there

So we ran redhub.ai through Cloudflare's scorer too. It returned 29 out of 100, Level 2, Bot-Aware. Same site, same week, a 65-point spread. The reason is entirely in what each tool counts.

Fourteen checks carry that score and each is worth the same. We passed four, and four out of fourteen is 29, which is the number on the dial. Weighting its categories equally instead would have produced 38, so the flat count is the arithmetic that fits.

CategoryChecksWe passed
Discoverability42
Content10
Bot access control22
API, auth, MCP and skill discovery70
Commercenot scoredn/a

Half the score sits in one category. Seven of the fourteen checks are API, authentication and agent-endpoint discovery: an API catalog, OAuth authorization-server metadata, OAuth protected-resource metadata, an agent-registration document, an MCP server card, an agent-skills index, and WebMCP tools in the browser. We run one public API and no authenticated endpoints. Six of those seven describe infrastructure that does not exist here, and publishing them anyway would have bought about 35 points and made six files lie.

We shipped the four we could stand behind: an API catalog naming the one API we do operate, a Link header pointing at it, markdown content negotiation, and WebMCP tools that actually run. That should read eight of fourteen. We have not re-run it to confirm, and the WebMCP check in particular depends on whether the browser doing the checking implements the API our code looks for before it registers anything.

Read the arithmetic before you act on a readiness score. Knowing the ceiling was eight of fourteen is what stopped six files from being written to close a gap that is not a gap.

Neither number is wrong. Cloudflare asks which agent standards a site supports and asks it the same way of every site. Ours asks how much of what applies to your kind of site is in place, which is why it makes you declare what you are before it scores anything. A blog has nothing to say about checkout. A storefront with no public API has nothing to say about OAuth. Whether that silence is a failure or a non-question is the whole disagreement, and it is worth knowing which one your tool assumes.

How to check your own site

You can do most of this with a browser and ten minutes.

  • Open yourdomain.com/llms.txt, /sitemap.xml, and /robots.txt. Note which ones 404.
  • View source on one product page and search for application/ld+json. If there is no offers block with a price, agents cannot read what you charge.
  • Check /.well-known/security.txt. Most sites have nothing there.
  • Ask whether anything on your site is machine-readable as a catalog, not as a page.

Then paste those files into the Agent-Readiness Scorecard. It is free, it runs in your browser, and nothing you paste is uploaded. Declare what your site is first, because a check that does not apply to you is excluded rather than failed.

Frequently Asked Questions

What is AI agent readiness?

AI agent readiness is whether an AI agent can find your site, read what is on it, and act on it without a person involved. It is decided by a small set of files at predictable addresses: an llms.txt, a Content Signal in robots.txt, a valid sitemap, JSON-LD product data, a machine-readable commerce surface, a .well-known directory, and a plain-text mirror of your content. Unlike traditional SEO it is almost entirely technical, which means it is fixable in days, not months.

How do I check if my website is ready for AI agents?

Open your own llms.txt, sitemap.xml, and robots.txt in a browser and note which ones return a 404, then view source on a product page and search for an application/ld+json block containing offers and a price. For a scored result, paste those files into a readiness scorecard. The RedHub Agent-Readiness Scorecard is free, runs client-side so nothing you paste is uploaded, and scopes its scoring to what you declare your site to be.

What is a Content Signal in robots.txt?

A Content Signal is a line in robots.txt declaring how your content may be used after it has been crawled, separately from whether crawling is allowed at all. It distinguishes search indexing, use as input to a generative answer, and use as training data. It is advisory and honored voluntarily, not enforced, but it is currently the only standard place to state the preference, and its absence is read as having no preference.

Do I need a product feed for AI shopping agents?

If you sell online, yes, because a shopping agent has to read your catalog before it can consider you. A feed lists every product with a stable identifier, price, canonical URL, and image in one machine-readable file, so the agent does not have to scrape individual pages. Generate it from the same source your product pages render from, not exported on a schedule, because a feed that contradicts its own product page is worse than having no feed at all.

Why do agent-readiness tools give the same site different scores?

Because they disagree about which checks apply. A scorer that runs the same fixed list against every site will mark a content site down for missing OAuth metadata and a storefront down for missing an agent-skills index, whether or not either would ever be appropriate. A scorer that scopes its checks to a declared site type excludes what does not apply rather than failing it, so the denominator is smaller and the number is higher. Neither is wrong, but they answer different questions, so check the arithmetic and the applicable set before acting on either.

Should I publish an mcp.json if I do not run an MCP server?

No. An MCP server card describes a callable endpoint, so publishing one when no such endpoint exists points agents at nothing and misrepresents what your site can do. Readiness checks that reward the file are rewarding a real capability, not the file itself. Leave the check unscored until you run a server, and treat any scorecard advice that suggests otherwise as a reason to distrust the scorecard.

Get found and cited by AI

Agent readiness gets an agent through the door. Being the answer it quotes is a separate job. The GEO / AI Visibility Playbook ($149) is the full method for earning citations in generative answers, and the AEO Citation Audit Kit ($79) measures where you stand today. If you sell online, the Agentic-Commerce Product-Feed Readiness Gate ($79) grades your feed item by item.

How it decides
Diagram of the Agentic Commerce Product Feed Readiness Gate: seven feed signals scored by the weakest and a feed-vs-page consistency gate forcing UNREADABLE.

The gate this post refers to, drawn from the tool’s own logic. See the tool.