Web Bot Auth is the story. Robots.txt isn’t.

Google added Google-Agent to its list of fetchers on 20 March 2026. The coverage has focused on the wrong line in the documentation.

Everyone's leading with the robots.txt angle — Google-Agent ignores it, ChatGPT-User and Claude-User respect it, websites need server-side controls now. Fine. That's true. It's also the least interesting thing about the announcement.

The interesting thing is one sentence buried further down: Google-Agent is experimenting with the web-bot-auth protocol using the identity `https://agent.bot.goog`. Cryptographically signed bot traffic. An IETF draft becoming an actual deployed standard. Cloudflare, Akamai, Amazon already support it. Google brings the critical mass.

That's the announcement. Robots.txt is a distraction.

Why the robots.txt framing misses

I get why people are leading with it. It's concrete. You can write a "what to do now" section. You can pitch a robots.txt audit. It plays well as a 24-hour news story.

But robots.txt has been a voluntary protocol since the 1990s. Compliance is honour-system. Anyone serious about scraping has been ignoring it for two decades. The fact that one user-triggered fetcher will ignore it whilst two others respect it is a minor inconsistency in a system that was never load-bearing to begin with.

If you were relying on robots.txt as access control, you were already doing it wrong. Google-Agent doesn't change that — it just makes the existing problem slightly more visible.

The real shift is what's replacing it.

What Web Bot Auth actually does

Web Bot Auth is a digital passport for bots. Each agent holds a private key. Each request is cryptographically signed. The receiving server verifies the signature against a public directory and knows, with mathematical certainty, who's making the request.

Web Bot Auth is the first piece of infrastructure that makes agent traffic measurable as a category.

User agent strings can be spoofed by a 14-year-old with curl. Signed requests can't. That's a step-change in what's possible.

Three things follow from this that nobody seems to be talking about.

Agent identity becomes a discovery signal. If your server can cryptographically verify that a request comes from Google-Agent, ChatGPT, or Claude on behalf of a real user, you can serve them differently. Faster responses. Cleaner markup. Structured data prioritised. Whatever optimisations you want to apply specifically when an agent is on the page. This isn't cloaking — the user gets the same content via the agent — it's content negotiation by verified identity.

The "is this an agent or a scraper" question becomes answerable. Right now, traffic from `ChatGPT-User` in your logs could be ChatGPT, or could be anyone who set a user agent string to that. With Web Bot Auth, you know. That changes what you do with the data. Suddenly agent traffic becomes something you can actually measure, segment, and report on. The measurement problem I keep banging on about gets one piece of its foundation.

Bad actors lose the cover of legitimate traffic. Scrapers currently hide in the noise by pretending to be browsers or known bots. Cryptographically verified agent identity makes that harder. Not impossible — they'll just stop pretending and go back to looking like residential proxy traffic — but the legitimate agent channel becomes cleaner.

Web Bot Auth is the first piece of infrastructure that makes agent traffic measurable as a category.

The three-tier visitor model is real

Search Engine Journal framed Google-Agent as creating a three-tier model: humans, crawlers, agents. That framing is right, and it's the part of the piece worth taking seriously.

Humans browse for themselves. Crawlers index for later retrieval. Agents act on behalf of a specific person, in real time, with a specific task. Each tier has different access expectations, different rendering needs, different conversion implications.

I wrote last week about what agents do to attribution — the moment an agent fills out your contact form on behalf of a user, your single-bucket "Google traffic" model breaks. Web Bot Auth is what eventually makes the cleanup possible. You'll be able to identify agent sessions cryptographically and treat them as a separate channel in your analytics, your CRM, your lead scoring.

We're not there yet. The protocol is an IETF draft. Most websites don't verify signatures. Most analytics tools don't know what to do with the data even if you captured it. But the standard is being deployed by the four companies that operate most of the world's CDN and bot infrastructure. That's not "if." That's "when, and probably faster than you expect."

What this means for measurement

The schema conversation last week was about whether AI citation lift is measurable at all. Ahrefs ran a controlled test on 1,885 pages and found nothing statistically meaningful. The honest answer was: we can measure citations, but we can't yet measure what causes them.

Web Bot Auth is the other side of that problem. Citations are upstream — what AI systems say about you. Agent traffic is downstream — what AI systems do on your site after they've decided to send you a user. Today we can't measure either properly. The signed-agent standard fixes the downstream side first, because it's the more tractable engineering problem.

If you're a site owner, the practical implication isn't "audit your robots.txt." It's: get familiar with how Web Bot Auth verification works, because in 18 months you'll be asked whether your analytics can separate agent traffic from human traffic, and the answer needs to be yes.

If you're an agency selling GEO services, the same applies. Reporting on AI search performance currently relies on screenshot evidence and prompt-spam monitoring. Verified agent traffic becomes a real metric you can put on a dashboard. The vendors who build that integration first will own the category.

The honest limits

Web Bot Auth doesn't solve the citation measurement problem. It doesn't tell you why an AI system chose to send a user to your site. It doesn't help with rankings or visibility in AI Overviews.

It also won't be universal for a while. The protocol is in draft. Implementation is uneven. Most of the AI ecosystem outside the big four (Google, OpenAI, Anthropic, Perplexity) won't adopt it quickly, and the long tail of smaller agents will keep operating in the user-agent-string Wild West.

And there's a perfectly reasonable read where this just makes life easier for the big agent operators whilst doing nothing for anyone else. Cryptographic identity benefits the entities that already have infrastructure. Smaller AI products that depend on scraping the open web get squeezed. That's not necessarily a bad outcome, but it's a real one.

The bit worth remembering

Google announced two things on 20 March. One was that a new agent doesn't respect robots.txt. The other was that Google is starting to sign its bot traffic cryptographically.

The first is a 24-hour news cycle. The second is infrastructure that reshapes how agent traffic gets measured, segmented, and trusted for the next decade.

The industry is talking about the first one. That's the loop. And we built it.