The web is quietly paying for the agent economy

There's a story the industry is telling itself about agentic traffic that's mostly about discovery. Who cites you, who doesn't, how to be readable to a model, what schema to add, which surface to optimise for next. The framing is: agents are a new audience, treat them like one.

That framing is missing the actual thing that's happening. Which is that agents are behaving on the open web the way a badly-written scraper behaves — fetching the same unchanged page dozens of times per session, holding stateful connections open when they don't need to, replaying context every turn because there's no session memory worth the name. Cloudflare has been publishing about this for months. Their language is unusually blunt for infrastructure marketing: "billions of requests" re-fetching pages that haven't changed. The MCP v2 announcement is essentially an admission that v1's connection model was expensive enough to be a problem worth rewriting the spec for.

Nobody in the SEO world is talking about this. Nobody in the marketing world is talking about this. It's being covered as a Cloudflare product story rather than what it actually is, which is the first public accounting of the transition cost of the agent economy onto the web's existing infrastructure.

I think that's the story worth writing about this week. Not because it changes what any single site should do tomorrow. But because the assumptions underneath the "agents as new audience" framing are going to break, and the way they break will matter for anyone whose business depends on the open web still functioning as a low-friction publishing surface.

The architecture wasn't built for this

The web we have was designed around a specific traffic pattern. A human types a URL or clicks a link. The browser fetches the page once. The user reads it, maybe clicks something, and the whole session lasts a few minutes with a handful of requests. Servers, CDNs, caching layers, and rate limits are all tuned to that shape.

The web is being used as a scratchpad by systems it wasn't built to serve.

Agentic traffic doesn't look like that at all. An agent working on a task might fetch the same product page four times in ten minutes because it's checking the price after each intermediate decision. A comparison agent might pull twenty similar pages, hold a working memory of all of them, and re-query several of them when the user asks a follow-up. A research agent might make forty requests to build one answer, and then another twenty when the human asks "actually, can you check this again."

None of this is malicious. None of it is even inefficient by the agent's own logic — it's optimising for correctness, not for the origin's bandwidth. But from the server's perspective, it looks like a single "user" hammering the site with redundant requests that a human never would have made.

Cloudflare's framing of this as billions of unnecessary fetches is, if anything, an undercount. That's just what one CDN sees. Multiply it across the whole delivery layer of the internet — every origin, every edge, every intermediate cache — and the aggregate is enormous.

The web is being used as a scratchpad by systems it wasn't built to serve.

MCP v2 is an admission, not a feature launch

The Model Context Protocol is the plumbing agents use to talk to external tools and data sources. Version one shipped fast, worked, and was expensive to run at scale. Version two is explicitly about fixing the connection overhead — sessions, replay, state persistence, all the things that make it viable to run a remote agent without holding a hot socket open for every tool call.

Read the v2 announcement as a product post and it looks like an incremental improvement. Read it as an infrastructure post and it's more revealing. The problems v2 addresses are the problems you only notice when your bill gets big enough to matter, or when the systems you're building can't scale on the current architecture. It's an admission that the way agents were talking to the internet in the first generation of MCP wasn't sustainable.

This matters for anyone who runs a site because MCP-style protocols are the layer that decides how agents interact with your infrastructure. If the protocol requires long-lived stateful connections, your server has to hold them. If the protocol allows lightweight stateless requests with cached context, it doesn't. The choices being made in these specs right now determine what the load pattern on your site looks like for the next five years, and almost nobody outside the infrastructure vendors is paying attention to those choices.

The SEO industry treats protocol design as somebody else's problem. It isn't. The protocol is upstream of everything else. If the standard settles on chatty stateful patterns, every site pays the cost forever. If it settles on efficient stateless patterns, the transition is manageable. Right now those choices are being made in Cloudflare and OpenAI and Anthropic offices with input from a small technical community, and the marketing world is treating it as ambient background noise.

Who actually pays for this

The economics here are worth stopping on. When a human visits a site and reads a page, the origin pays a small delivery cost and the visitor might see an ad, generate a lead, buy a product, or leave. The economic model, however imperfect, is a rough exchange — bandwidth for attention, delivery for chance-of-conversion.

Agentic traffic breaks the exchange. The origin pays the delivery cost, often several times over for redundant fetches, and receives nothing in return. No ad impression that converts. No pageview that counts toward analytics that inform business decisions. Frequently no attribution at all, because the agent's user never learns which sources it consulted, and even if they did, the click doesn't come back to the origin — the answer does.

The costs are real. Bandwidth isn't free. Server capacity isn't free. Cache invalidation isn't free. Rate limiting infrastructure isn't free. Log storage for traffic you're now getting ten times as much of isn't free. Every site above a certain size is quietly absorbing an operating cost increase to serve traffic that produces no direct revenue and often no downstream benefit.

Big sites can afford this. They have negotiated CDN deals, sophisticated caching, and enough scale that a 15% increase in raw request volume shows up as a line item rather than an existential problem. Small sites cannot. The independent publisher on a shared host with a $30-a-month plan has no visibility into which of their traffic is agent-driven and no leverage to price it differently. They just see the bill go up.

The measurement problem underneath the measurement problem

I've written before about the measurement problem in AI search — the fact that citations don't show up in analytics, attribution is broken, and log-file analysis is the only reliable way to see what's actually happening. That's the surface layer.

The measurement problem isn't just that you can't see AI referrals. It's that you can't see the cost of serving them either.

The deeper layer is that most sites can't distinguish agentic traffic from other kinds of automated traffic at all. Bot detection is optimised for known bad actors and known good crawlers. It's not designed to differentiate a helpful agent making legitimate on-behalf-of-user requests from a scraper making the same requests for a different purpose. The user agent strings are often generic. The behaviour patterns overlap. The IP ranges are increasingly the same cloud providers everyone else uses.

Which means that when Cloudflare says billions of requests are redundant agent fetches, they can say that because they have the aggregated telemetry to see the pattern. Individual site operators cannot. You look at your own logs and see traffic. You can't easily tell what proportion of it is an agent burning through your pages on behalf of a user who will never visit, versus a competitor scraping you, versus a bot that just wants to train on your content, versus a genuinely useful referral that might one day send someone your way.

The measurement problem isn't just that you can't see AI referrals. It's that you can't see the cost of serving them either.

Until that changes — and it will change, because CDN providers have every commercial reason to sell that visibility as a product — most site operators are running blind on both the revenue and the cost side of the agent economy.

Why the standard conversation misses this

The reason nobody in marketing writes about this is that it doesn't fit the shape of the industry's usual output. It's not a ranking factor. It's not a checklist. It's not something you can fix with schema or a plugin. It's an infrastructure story, and the marketing industry has always been bad at infrastructure stories except when the infrastructure vendor pays for the coverage.

But the second-order implications are entirely a marketing story. If the cost of serving the open web rises meaningfully because of agent traffic, sites will respond. Some will paywall. Some will block. Some will negotiate. Some will get acquired. Some will disappear. The composition of the open web that agents can read from a year from now is being shaped by economic pressure the industry isn't naming.

Cloudflare's aeo positioning — pitching themselves as the layer that makes your site legible to agents while managing the cost of serving them — is a preview of where this ends up. The CDN and edge providers become the arbiters of who agents can talk to and on what terms. Small sites get whatever default the vendor decides is reasonable. Large sites cut deals. The independent web pays.

The honest limits

I don't have hard numbers on how much of the average site's traffic is agentic right now. Nobody does, publicly. The estimates that exist are vendor estimates, which means they're structured to sell whatever the vendor sells. Cloudflare's numbers are probably directionally right, because their vantage point is genuinely comprehensive, but "billions of requests" across their whole network doesn't translate cleanly to what any individual site is experiencing.

I also don't want to overstate the crisis. The web has absorbed traffic shifts before. Mobile added enormous load. Video added enormous load. Social sharing added enormous load. In every case, the infrastructure eventually caught up, protocols improved, costs distributed, and the ecosystem kept working. This is probably the same shape of problem, and MCP v2 is exactly the kind of protocol-level adjustment that suggests the industry is starting to iterate its way to a workable steady state.

What I do think is defensible is that the transition cost is real, is being paid by parties who have no say in the design of the systems generating the load, and is almost entirely absent from the discourse that most marketers are consuming. That last part is what I keep coming back to. If your only exposure to the agent economy is "how do I get cited by ChatGPT," you are looking at one visible surface of a much larger thing.

What to actually do

Nothing panicked. But three concrete things worth doing this quarter:

Look at your logs. Not your analytics. Your server logs. If you don't have easy access to them, get it, or ask your host for it. Look at the ratio of requests to unique sessions, look at the user agents you don't recognise, look at the pages being fetched most often and ask whether the pattern makes sense for human traffic. You may be surprised.

Talk to your CDN or host about agent traffic specifically. Not blocking, not filtering — just visibility. Ask what they can tell you about the composition of your automated traffic and what tools they'll be rolling out to distinguish helpful from unhelpful requests. If they can't answer, that's a data point about which providers are actually thinking about this.

Pay attention to protocol conversations, even boringly. When MCP or something like it settles into a stable form, the design choices in that spec will affect your infrastructure costs for years. You don't need to participate in the working groups. You do need to know what the choices are, so you can ask your vendors whether they've implemented the efficient patterns or the wasteful ones.

None of this is glamorous work. It's not the stuff that makes for punchy LinkedIn posts. But it's the layer underneath everything else the industry is currently obsessing about, and the sites that get ahead of it will spend less and see more than the sites that don't.

The agent economy is not free. Someone is paying. Right now that someone is you, and you're not being told.