Your Website Has a New Audience, and Most of Us Haven't Noticed Yet

A field report from running Cloudflare's Agent Readiness diagnostic against real B2B websites — what it measures, what it gets wrong, and what I'd actually prioritize.

TL;DR

In April 2026, Cloudflare launched a public 0–100 score that checks whether a website is technically legible to AI agents — separate from whether it's legible to humans or ranks well in search.

Cloudflare's own scan of the 200,000 most-visited domains found 78% have a robots.txt file, but only 4% have declared AI usage preferences, and just 3.9% serve a clean Markdown version of their content to agents. The infrastructure for talking to agents barely exists yet — that's the opportunity.

I've been running this diagnostic (and the implementation work behind it) against B2B client websites. The pattern is consistent: it's rarely one big "AI problem." It's a dozen small assumptions baked into the site years ago, when a browser was the only thing that mattered.

The composite score is a useful trigger, not a target. Independent testing has shown the same website can score 33/100 on one scan preset and 67/100 on another — a 34-point swing with zero changes to the site. Treat category-level detail, not the headline number, as the real signal.

A technically "agent-ready" score answers one question: can a machine access and parse your site. It says nothing about whether that machine, once it understands you, actually recommends you over a competitor. That's a separate discipline — Answer Engine Optimization (AEO) — and it's where marketing, not engineering, does the real work.

My recommendation: fix Discoverability and Content first (cheap, universal, high leverage), treat Capability and Commerce as roadmap items scoped to what your business actually does, and track AI citation/mention rate as the outcome metric — not the Cloudflare score itself.

For fifteen years, I've optimized websites for one visitor: the human being on the other end of the screen. Page speed, navigation, mobile responsiveness, forms, CTAs, conversion paths — all built around the assumption that a person is scrolling, clicking, deciding. That assumption is starting to crack. Not because humans are disappearing. Because increasingly, they're sending something ahead of them. A buyer now asks an AI assistant to shortlist vendors, compare solutions, summarize a product line, or check pricing before a human ever lands on your homepage. By the time your prospect actually visits, an agent may have already formed an opinion about you — based on how well your website answered its questions, not how impressive your hero section looked. That agent doesn't care about your animation. It doesn't follow the navigation your team spent three sprints designing. It's asking something much blunter: Can I discover this company? Can I understand what it offers? Can I retrieve what I need? Can I act on it? I used to think of that as an engineering question. I don't anymore. I think it's becoming one of the more important marketing questions of the next few years — and I want to walk through why, using both Cloudflare's new diagnostic and what I've actually found running it against client websites.

Cloudflare just built the Lighthouse score for agents

In April 2026, Cloudflare launched isitagentready.com, a free tool that scores any website on how well it serves AI agents rather than humans, described in Cloudflare's own announcement as a deliberate echo of Google Lighthouse — a public score that nudges an entire industry toward a new set of standards simply by making the gap visible. The scoring isn't asking whether your site has a chatbot bolted on. It's checking the plumbing: can an agent find your sitemap, request a lightweight version of your content, understand what it's allowed to use your content for, and — eventually — call your APIs or transact with you directly. Cloudflare groups this into four scored dimensions (Discoverability, Content, Bot Access Control, and Capabilities), plus an unscored, forward-looking Commerce layer covering emerging payment protocols like x402, ACP, and UCP. That last distinction matters more than it looks. Cloudflare is explicitly telling the market: these commerce protocols are real, but they're not table stakes yet. It's a maturity signal, not a checklist to panic over.

The web is not ready, and the data is specific about it

Cloudflare didn't just ship a scanner — it ran it against the 200,000 most-visited domains on the internet and published the results on Cloudflare Radar. The numbers are worth sitting with:

78% of sites have a robots.txt file, but almost all of them were written for search crawlers, not AI agents.
Only 4% have declared their AI usage preferences using the new Content Signals standard.
Markdown content negotiation — serving agents a clean, lightweight version of a page — is live on just 3.9% of sites.
Newer standards like MCP Server Cards and API Catalogs appear on fewer than 15 sites in the entire sample.

Read that again: the infrastructure to talk to search engines is nearly universal. The infrastructure to talk to agents barely exists. That gap is the opportunity, and it's a rare one — most of B2B marketing has been fighting over ranking positions on a mature channel for a decade. This one is still wide open.

What I found actually implementing this on client sites

Here's where the plan I've been running for clients earns its keep. The instinct, when you first look at the Cloudflare checklist, is to hand it to a developer and move on. That's the mistake. A sitemap isn't a dev task — it's a statement of what exists. Markdown negotiation isn't a formatting trick — it determines how efficiently an agent can actually consume what you've written. Content Signals aren't an HTTP header — they're your first real mechanism for telling AI systems what they can and can't do with your content, distinct from what you allow for search. An API catalog isn't documentation — it's a declaration of what your business can programmatically do. Every client site I've run through this diagnostic has the same pattern: not one giant "AI problem," but a dozen small assumptions baked in years ago, when a browser was the only consumer of the site that mattered. Legacy sitemaps that stopped updating after a CMS migration. robots.txt files that were copy-pasted from a template in 2019 and never touched again. Pricing and product information that lives entirely in a PDF a crawler can't parse cleanly. None of this shows up as a "problem" to a human visitor — a person just clicks around it. An agent has no such patience.

The score is a trigger, not a target — and the data backs that up

I want to be direct about a limitation here, because SignalShift doesn't do hype. The composite Cloudflare score can be misleading if you take it at face value. Independent testing by Search Engine Journal and nohacks.co found that the same website, scanned on the same day, produced a score of 33/100 on the default "All Checks" preset and 67/100 on the "Content Site" preset — a 34-point swing caused entirely by which checks were counted, not by anything that changed on the site. A pure content site will always look artificially weak on a preset that scores it against API catalogs and commerce protocols it has no business implementing. That's not a reason to dismiss the tool. It's a reason to read the category breakdown instead of the headline number, and to pick the scan configuration that actually matches your business model. A SaaS platform and a content-driven B2B publisher should not be graded on the same curve, and Cloudflare's own preset options acknowledge that. The lesson for marketers: don't hand the composite score to leadership as a KPI. Hand them the category-level gaps that are actually relevant to how your buyers use AI to find you.

A five-layer way to think about it

I've stopped treating this as a single implementation project and started treating it as a maturity model, because not every business needs to reach the top:

Discoverability — robots.txt, sitemap, Link headers. Can an agent find you at all.
Readability — Markdown negotiation, clean semantic content. Can it understand you efficiently.
Permission — Content Signals, AI crawler rules. Does it know what it's allowed to do.
Capability — API catalogs, MCP, Agent Skills, OAuth discovery. Can it actually do something with your business.
Transaction — x402, agentic commerce protocols. Can it act on a customer's behalf.

A content-heavy B2B site probably gets more value from nailing layers one through three before it ever touches MCP or agentic checkout. A SaaS platform with public APIs has a very different roadmap, and layer four is where the real payoff sits. The question isn't "how do we hit 100%" — it's "which of these would actually make our business easier for an agent to find and use."

The best proof of this isn't a hypothetical — it's Cloudflare's own docs

Cloudflare didn't just publish a standard; they rebuilt their own developer documentation against it, and shared the results. A few things stood out to me: They didn't just add an llms.txt file and call it done. They restructured the underlying information architecture — cutting roughly 450 low-value directory-listing pages that added no semantic value for an LLM, splitting one massive llms.txt into per-directory files so it would actually fit inside a model's context window, and tightening page titles and descriptions so an agent could tell which page to fetch without guessing. The result, in their own benchmark: an agent working against their documentation used 31% fewer tokens and reached correct answers 66% faster than against a typical unoptimized docs site. That's not a marginal SEO gain. That's the difference between an agent recommending you accurately and an agent giving up halfway through and guessing. Worth noting too: Cloudflare cites research from Checkly showing that as of February 2026, of seven agents tested, only three — Claude Code, OpenCode, and Cursor — actually requested Accept: text/markdown by default. That's exactly why Cloudflare built a URL-based fallback (serving Markdown at /index.md regardless of headers) rather than betting entirely on header negotiation. It's a good reminder that "supporting a standard" and "supporting how agents behave in practice" aren't always the same implementation task. Structure, in other words, isn't just an accessibility nicety for machines — it's what determines whether the agent's answer about you is actually correct.

Don't confuse the technical score with the marketing outcome

This is the distinction I'd push hardest on. Cloudflare's Agent Readiness score answers one question: can a machine technically access and use your site? It says nothing about whether that machine, once it understands you, actually recommends you over a competitor. That's a separate discipline Cloudflare itself frames as AEO — answer engine optimization — measured by things like citation rate and share of voice in AI-generated answers. So there are really two problems stacked on top of each other: Can the machine understand us? That's infrastructure — robots.txt, Markdown, Content Signals, API catalogs. After it understands us, does it choose us? That's marketing — positioning, evidence, third-party validation, how clearly your differentiation is stated. Getting a green diagnostic score solves the first problem. It doesn't touch the second. I've seen teams treat a high Agent Readiness score as the finish line, when it's really the entry ticket.

What I'd tell a CMO to do this quarter

Skip the instinct to implement every protocol on the list. Instead:

Run your domain through the scanner using the preset that matches your business type, and treat the failing checks as a prioritized backlog, not a mandate.
Own the content side yourself — sitemap coverage, page descriptions, Content Signals decisions — rather than handing the whole thing to engineering and hoping it comes back marketing-shaped.
If you're API-driven or platform-based, start scoping an API catalog and MCP exposure now. Being one of the first fifteen sites with a working MCP Server Card, per Cloudflare's own data, is still a real differentiator today.
Leave commerce protocols like x402 alone for now unless agentic checkout is already core to how your customers buy. Cloudflare itself doesn't score them yet.
Separately, start tracking whether AI assistants are actually citing or recommending you. That's the metric that tells you if the infrastructure work is paying off — not the composite score.

The real shift

The old model was: search → click → browse → evaluate → form → conversation → purchase. Agents compress most of that. A buyer may never see your homepage — the agent sees it for them, forms a judgment, and hands the human a shortlist. That means your website now has to perform when no one is looking at it. Engineering owns the protocols that make that possible. But marketing owns nearly everything those protocols expose — how the company describes itself, what evidence backs its claims, which page is the actual source of truth. I don't think the future is "websites built for AI." That framing is too narrow. The more accurate way to say it: your website is becoming a shared information layer between humans, search engines, and autonomous agents — and for the first time in a while, there's a public, measurable way to check how far behind you are.

FAQs

What is Cloudflare's Agent Readiness score, exactly?

It's a free public scanner at isitagentready.com that checks any website against a set of technical standards AI agents use to discover, read, and interact with sites — things like robots.txt, sitemaps, Markdown content negotiation, Content Signals, API catalogs, and MCP server cards. It produces a 0–100 score across four scored categories, plus an informational Commerce category, as Cloudflare details in its launch post.

Is this the same thing as SEO?

No, and treating it that way is the most common mistake I see. Traditional SEO optimizes for ranking in a search results page. Agent readiness optimizes for whether an autonomous system can technically access, parse, and act on your content at all — a prerequisite that comes before ranking or recommendation even enter the picture. They share some overlapping mechanics (sitemaps, clean markup) but different goals.

Do I need a developer to fix this?

For the foundational layers — robots.txt, sitemap, Content Signals — yes, but it's a small, well-scoped task, not a rebuild. Cloudflare's tool even generates a ready-to-use prompt for each failing check that you can hand directly to a coding agent. The harder work, in my experience, is deciding what your Content Signals policy should be and which pages actually deserve to be in your llms.txt — and that's a marketing and legal decision, not a coding one.

Should marketing or engineering own agent readiness?

Both, but for different pieces. Engineering owns implementing the protocols correctly. Marketing owns what those protocols expose: how the business describes itself, which pages are the canonical source of truth, what evidence backs up claims, and what the company is willing to let AI systems train on versus simply reference. If marketing isn't in the room, you'll end up with a technically compliant site that still says nothing useful to the agent reading it.

What's the difference between Agent Readiness and AEO?

Agent Readiness measures technical accessibility — can a machine reach and parse your site. AEO (Answer Engine Optimization) measures the outcome — does an AI assistant actually cite, mention, or recommend you once it understands you, tracked through metrics like citation rate and share of voice. You can be perfectly agent-ready and still lose the recommendation to a competitor with a stronger evidence base and clearer positioning.

Should I trust the composite score as a KPI?

Be careful with it. As independent testing has shown, the same website can score over 30 points differently depending on which scan preset is used, since the default preset scores every site against checks — like API catalogs or commerce protocols — that only apply to certain business models. Use the category-level breakdown that matches your site type, and track your own trend over time rather than comparing your composite number against unrelated sites.

What should I actually implement first?

For most B2B marketing sites: a clean, current sitemap and robots.txt with explicit AI crawler rules, followed by Content Signals to state your AI-usage preferences, followed by Markdown content negotiation. That covers the Discoverability, Bot Access Control, and Content layers, which apply almost universally. Save API catalogs, MCP, and commerce protocols for when your business model actually calls for agents to take action, not just read.