The measurement layer of SEO is broken

There is a quiet crisis running underneath every SEO conversation happening right now, and the industry is doing an extraordinary job of not naming it. Impressions are climbing whilst clicks flatten. Ranking trackers show volatility on days Google denies making any changes. Prompt-tracking dashboards report on prompts nobody typed. Pages vanish from the index for reasons that don't show up in any report. Case studies get quoted in decks with no methodology attached. And the whole apparatus keeps producing numbers, month after month, that everyone reports on and almost nobody trusts.

We have arrived at the point where the measurement layer of SEO — the instruments we use to tell clients whether the work is working — has decoupled from the reality it's supposed to describe. Not entirely. Not everywhere. But enough that anyone doing this seriously has to acknowledge it. The dashboards look the same. The numbers keep flowing. The graphs still slope. But under the hood, the wiring has come loose from the world.

This is the piece I've been circling for weeks in the morning and afternoon posts. Every individual story I've written about — the AI visibility tool proliferation, the security interstitials silently deindexing pages, Google's unverifiable "billions of AI clicks" claim, the AI Overviews image generation move, the collapse of the ranking concept under Personal Intelligence — is a different symptom of the same disease. And the disease is that the instruments are lying, and the industry has decided collectively to pretend they aren't.

I want to lay out what's actually happening, why it matters more than any individual algorithm update, and what a person who actually cares about doing this work well should do about it. Because there is a version of this argument that ends in nihilism ("nothing can be measured, everything is guesswork") and there is a version that ends in vendor-capture ("just buy the enterprise dashboard"). Both are wrong. There is a third path, and it requires being honest about what we can and can't see.

---

The four instruments SEOs have relied on for a decade

Before I explain what's broken, it's worth being explicit about what SEO measurement has consisted of. For roughly the last decade, we have relied on four instruments:

Google Search Console — impressions, clicks, average position, and the index coverage report. The closest thing to a source of truth about how Google sees your site. Free. Owned by the platform we're trying to understand.

Third-party rank trackers — SEMrush, Ahrefs, AccuRanker, Sistrix, Mangools, Wincher, and a dozen others. Query a defined keyword set on a schedule and report where a domain sits in the SERP for each one.

Analytics (GA4, Plausible, server logs) — what happens after the click. Sessions, engagement, conversions.

Backlink and authority tools — Ahrefs, Majestic, Moz. Not measurement of performance exactly, but measurement of the signals presumed to drive performance.

Everything else built on top of that stack — the content briefs, the technical audits, the reporting decks, the KPIs, the retainers, the case studies — depended on the assumption that those four instruments produced roughly reliable readings of a roughly stable underlying system. When the SERP volatility trackers agreed something had happened, something had usually happened. When Search Console showed clicks dropping, humans were usually clicking less. When rank tracking said you'd moved from position 4 to position 7, that was usually where you sat when a searcher typed the query.

That model is falling apart. Not because any single instrument has failed catastrophically. Because the system they're measuring has quietly transformed into something the instruments were never designed to see.

Search Console is measuring machines, and calling them people

The most consequential instrument failure — the one nobody wants to talk about — is that Google Search Console is increasingly reporting machine behaviour as if it were human behaviour, and the industry is still reading those numbers as if they represented human demand.

Every impression graph you're currently reading is a mixture of human and machine behaviour, and the mix is shifting monthly.

Slobodan Manic's piece for Search Engine Journal this week finally put a name to what a lot of us have been quietly seeing in client data. AI systems now hit Google constantly to ground their answers. A single user prompt in ChatGPT or Perplexity fans out into multiple parallel queries, most of which read the results without any human ever laying eyes on them. Every one of those searches lands as an impression on the pages that rank for it. That's not new information — grounding via search has been documented for a while now — but the scale is now large enough to visibly distort what Search Console reports.

There was a live demonstration of this last year. Real users' ChatGPT prompts were turning up inside Google Search Console because of a bugged prompt box that was making ChatGPT search almost every time, with a ChatGPT URL leading the queries that Google then tokenised. Website owners were seeing strangers' private prompts appear in their dashboards. That was the visible version. The invisible version — the one nobody caught, because there was no leak to trace it back to — is happening at scale right now, and it's inflating impressions across a growing share of the index.

Every impression graph you're currently reading is a mixture of human and machine behaviour, and the mix is shifting monthly.

This is not a small thing. It rewires the meaning of the most-referenced chart in SEO. When impressions climb whilst clicks flatten — the "crocodile mouth" pattern Search Console watchers have been complaining about for two years — the conventional read is that visibility is up but click-through is down, and therefore the SERP is getting worse or the titles need work or the AI Overviews are cannibalising. Sometimes that's true. Increasingly, the alternative read is more accurate: impressions are up because machines are querying more, humans aren't clicking more because there aren't more humans, and the whole apparent "problem" is a measurement artefact of the instrument catching traffic it was never designed to distinguish.

Nobody is going to fix this on the Google side. Search Console has never distinguished bot from human, because for two decades the distinction didn't matter enough to build. Now it does. And there's no roadmap I've seen — public or private — that suggests it's being addressed.

Rank trackers are polling a SERP that no longer exists

The second instrument that's decoupling from reality is the third-party rank tracker. This weekend's unconfirmed Google update — the July 18th volatility that spiked across AWR, Semrush, Sistrix, Mozcast, AccuRanker and the rest — is a useful case study in what these tools now actually measure.

Google didn't confirm anything. The tools all showed movement. The chatter on WebmasterWorld ranged from "traffic up 400%, haven't touched the site" to "impressions going down whilst rankings recovered." The signals contradicted each other because the thing being measured — a stable, personalisation-light, roughly identical SERP served to a large population of users — no longer exists in the way the instruments assume it does.

Rank tracking works by simulating a query from a defined location on a defined device, capturing the SERP, and finding your position. That was a reasonable approximation of user experience when the SERP was roughly the same for most users. It's an increasingly bad approximation now, for reasons that compound:

Personalisation has deepened. Google's Personal Intelligence work — the Calendar-connected AI Mode I wrote about a few weeks ago — is the endpoint of a trend that's been building for years. The SERP a rank tracker sees at 3am from a datacentre in Iowa is not the SERP your customer sees at 9am from their phone on the way to work with their calendar and location and search history feeding into the ranking.

AI Overviews now inject a layer above the traditional results that rank trackers weren't built to parse. Some tools have retrofitted AIO detection; most report the presence-or-absence flag and stop there. Whether your domain was cited inside the Overview, in what position, at what confidence — that data doesn't sit in a rank tracker's model of the world.

The image generation move Google made last month closes another loop. AI Overviews now generate their own images, which means the last consistent traffic pathway rank trackers assumed — that a ranked result reliably received clicks proportional to its position — is decaying at another interface.

You can still use rank trackers. I do. But the numbers they produce are one signal amongst several, weighted lower than they used to be, and the volatility charts they publish are increasingly telling you about the instrument's confusion rather than the SERP's movement.

The AI visibility tool market is selling instruments that don't work

If the traditional instruments are decoupling from reality, the new instruments — the "AI visibility" dashboards that have been proliferating since late 2025 — are worse. I wrote about this three days ago in a shorter piece, but the argument needs restating in the flagship because it's the clearest case study in the industry's willingness to pay for the appearance of measurement rather than measurement itself.

The pitch is familiar: a tool types a set of prompts into ChatGPT, Perplexity, and Google's AI Overviews on a schedule, and reports how often your brand shows up. It looks like rank tracking. It sells like rank tracking. It fits neatly into the reporting deck slot where rank tracking used to go.

It's measuring something that isn't your customers' experience of AI search.

The prompts the tool tracks are prompts someone invented. Not prompts real customers type. Nobody knows what prompts real customers type — the platforms don't share that data, and grounding-inflated Search Console data can't tell you either. So the tool is measuring your brand's presence in a fictional universe of prompts a marketer imagined, and calling the results "AI visibility."

Jono Alderson said this on the No Hacks podcast recently, and he was right: "It's copy-paste the current modality of rank tracking into a new thing. It doesn't really fit, but it's better than nothing." Except I'd push harder than "better than nothing." When a measurement tool creates the impression of insight without the substance, it's not neutral. It actively substitutes for the harder work of understanding what's actually happening. Reports get built on it. Retainers get justified by it. Decisions get made off it. And the underlying reality — whether real humans are finding you, choosing you, buying from you — drifts further from the graph on the slide.

Mistake 1

Assuming your AI visibility dashboard is measuring what your customers experience. It's measuring what a scheduled bot experiences when it types prompts a marketer wrote. Those are not the same.

Mistake 2

Treating rank tracker volatility spikes as confirmation that "an update happened." Rank tracker volatility can mean an update happened. It can also mean personalisation deepened, AIO coverage shifted, the tool's sampling changed, or several minor tweaks compounded. The chart is not the algorithm.

Mistake 3

Reading rising Search Console impressions as rising demand. A growing share of impressions in most niches is now machine-driven grounding traffic. Rising impressions with flat clicks may not be a CTR problem at all — it may be the instrument catching traffic that was never going to click.

Mistake 4

Ignoring index-level failures because Search Console shows nothing broken. The security interstitial issue John Mueller flagged last week is a live example: pages get served bot-check screens, Google indexes those, treats them as duplicates of every other bot-check screen on the web, canonicalises them to somebody else's domain, and your traffic disappears with no error report anywhere. If you're only auditing what Search Console flags, you're missing the failures Search Console doesn't know how to flag.

The index itself is failing in ways nobody's auditing

That last point deserves its own section, because it's the sharpest example of instrument failure in the whole stack. Search Console is meant to be the ground truth for how Google sees your site. And Search Console is systematically undercounting a class of failure that's becoming more common as bot protection layers grow more aggressive.

The instrument only catches failures the instrument was designed to catch.

Mueller's Search Off the Record segment last week described the failure mode plainly. A CDN or bot-protection layer flags a request as suspicious. It serves an "are you a bot?" interstitial instead of the real page. Googlebot sometimes gets that interstitial. Google indexes it. Because every "are you a bot?" screen on the web looks similar, Google's duplicate detection kicks in, picks a canonical version from somewhere else, and marks your page a duplicate of a page you don't own. Your real content silently falls out of the index.

The instrument reports "duplicate, submitted URL not selected as canonical" in a report most site owners don't check. There's no error. Nothing looks broken. The page loads fine when you visit it in a browser. The failure only reveals itself if you run the URL Inspection tool and notice the canonical Google chose belongs to a different domain entirely.

The instrument only catches failures the instrument was designed to catch.

This category — CDN and bot-protection layers silently serving Googlebot the wrong content — is expanding as sites layer on more aggressive security. It expanded again this month with Google's Gemini Notebook rebrand, where a whole new class of user-triggered fetchers now needs to be handled explicitly in firewall rules and htaccess files if you want to block them, and where getting the rules wrong means either exposing content to scraping you didn't consent to, or accidentally blocking legitimate Googlebot traffic. Neither failure mode shows up cleanly in Search Console.

Marie Haynes wrote something related last week about "crawled — currently not indexed" pages, and her diagnosis matched what I see in client audits: in almost every case, the pages have quality issues Google is silently downweighting to the point of exclusion. That's another category of index-level failure the instruments don't surface well. Search Console will tell you the count. It will not tell you why. You have to reason about it from outside the tool.

Volatility is being manufactured by the observer effect

There's a subtler failure worth naming: a portion of what the industry now reads as "algorithm volatility" is being manufactured by the instruments themselves and the way SEOs respond to them.

The pattern goes like this. A tracker shows a spike. SEOs check their sites. Some see movement, some don't. The chatter starts on WebmasterWorld, Twitter, the private Slacks. Google doesn't confirm anything. Some percentage of sites make changes in response to what they think happened. Rankings shift because sites changed, not because Google did. The tools pick up the shift. The next spike gets attributed to a follow-up update. The loop tightens.

I'm not saying every unconfirmed update is imaginary. Something clearly did move this weekend — the tool consensus and the traffic reports are too consistent to be nothing. But the ratio of actual-algorithm-change to observer-effect-noise has shifted in a way that makes reading the charts genuinely harder than it was five years ago. When SEOs' response to volatility signals is itself a driver of volatility, the signal-to-noise ratio degrades, and the confidence you can place in any individual volatility reading degrades with it.

That's the loop. And we built it.

The steelman: the instruments still work, mostly

I want to take the other side of this seriously before I go further, because there's a version of my argument that's overstated and I don't want to be caught in it.

The steelman for the current measurement stack goes like this. Yes, impressions are noisy. Yes, rank tracking is imperfect. Yes, AI visibility dashboards are directional at best. But directionally, all of these instruments still track something real. If your Search Console impressions are trending down over six months, that's meaningful, even if any individual week is contaminated by machine traffic. If your rank tracker shows you falling off page one for your money queries across every tool at once, something is genuinely wrong, even if the exact position is fuzzy. If you're not being cited in any AI Overview across a broad prompt sample, you're probably not being cited by real users either. The instruments are noisy, not lying.

There's real weight in that. I don't disagree with any of it in isolation. My argument isn't that the instruments produce zero information. It's that the ratio has shifted — that the signal-to-noise has degraded to the point where you can no longer read the graphs the way you could five years ago, and that treating the numbers with the confidence they once warranted is the actual failure mode.

Where I'd concede more ground: for large sites with stable traffic patterns and long histories, the instruments still tell you a lot. If you're an established brand with millions of sessions a month and you can see six-quarter trend lines in Search Console, the machine-traffic contamination is a smaller share of the signal and the tools remain useful. The measurement crisis is worst for smaller sites, newer sites, and sites where AI Overviews cover a large share of the relevant queries. It's not uniform.

What actually works now: the triangulation stack

The response to broken instruments is not to abandon measurement. It's to stop trusting any single instrument and to triangulate across a wider set of signals — including some the industry has neglected because they're harder to automate.

Here's the framework I've been using with clients for the last six months. It's not novel. It's more work than reading one dashboard. But it's the only approach I've found that produces answers I'm willing to defend to a client.

Pillar 1: Server-side truth

Log files, server-side analytics, and CDN data are the ground truth for what actually happened at your server. They don't lie about whether a request came in. They tell you what user agent it was, what path was requested, what status was returned. Every client I take on now, the first thing I ask for is 30-90 days of log data. It reveals things Search Console can't: which bots are hitting you (including the AI grounding traffic that's inflating your impressions), where they're being blocked, which pages they're crawling that Google won't tell you it's crawling. If you're not looking at logs, you're reading a translated summary of what your server did, mediated by tools with their own agendas.

Pillar 2: Human-only conversion truth

The second anchor is the funnel from click to conversion, measured with the cleanest attribution you can build. Server-side conversion tracking, first-party analytics, actual revenue reconciliation. This is the thing AI search cannot inflate and machines cannot fake. When a real human buys something, that's real. If your traffic is up and your conversions are flat, either the traffic isn't human or it isn't qualified. Both diagnoses are useful. Neither shows up in a visibility dashboard.

Pillar 3: Direct citation observation

Instead of paying for a prompt-tracking dashboard that types invented queries, do the work of observing where you actually show up. Set up alerts for brand mentions across the surfaces that matter — Reddit, YouTube, industry press, LinkedIn, Substack. When someone at a client company asks ChatGPT a question they'd actually ask, watch what happens. When a prospect describes how they found you, capture that. Sales conversations are a better AI visibility signal than any dashboard I've seen.

Pillar 4: Index-level auditing you don't get for free

The security-interstitial failure I described above only reveals itself if you look for it. Run periodic canonical audits: pick a sample of your important pages, check the URL Inspection tool, confirm Google's chosen canonical is the URL you expect. Watch the "duplicate, submitted URL not selected as canonical" report. Test Googlebot rendering with a real user-agent switcher. This is 30 minutes a month and catches failures that would otherwise cost you a quarter of traffic before anyone noticed.

Pillar 5: Brand-search baseline

The one metric that remains a relatively clean signal of underlying demand is direct brand search volume. Machines don't type your brand name into Google unless a human asked them to. If your brand searches are growing, humans are hearing about you somewhere. If they're flat or declining, no amount of dashboard optimism should reassure you.

Pillar 6: The qualitative anchor

Talk to customers. Talk to prospects. Ask sales what people say in discovery calls. Read your support tickets for how people describe finding you. This is the hardest instrument to automate and the one that scales worst, which is why the industry has systematically underweighted it. It is also the least corruptible. A customer who tells you "I saw you cited in a Perplexity answer about X" gives you more useful information than any AI visibility tool has ever produced.

What this changes about how you should work

If the measurement layer has degraded, the practical response is not to give up on doing SEO. The response is to stop letting broken instruments dictate what work you do.

A lot of the SEO work I see agencies selling right now is work that only makes sense if you trust the dashboard. Optimising for AI visibility scores. Chasing prompt-tracking dashboard improvements. Rewriting titles because Search Console CTR looks bad on a query bucket that's 70% machine traffic. Building content around keyword volumes inflated by AI grounding. These are not neutral activities. They cost time and money, and they're directed by numbers that don't reflect what the numbers say they reflect.

The alternative isn't cynicism. It's returning to work that produces value whether or not you can perfectly measure it. Publishing genuinely useful content. Fixing genuinely broken technical foundations. Earning genuine mentions from places real humans read. Building a brand people search for by name. Making sure your site can be crawled, rendered, and indexed cleanly. Getting the schema right. Making the pages fast.

None of that is new. That's the point. The fundamentals didn't stop working — they got harder to see working, because the instruments that used to tell you they were working have decoupled from what they're supposed to describe. If you keep doing the fundamentals and stop trying to prove they're working via corrupted dashboards, you'll be fine. If you stop doing the fundamentals because the dashboard says they aren't producing enough of a spike, you'll be in trouble.

The uncomfortable close

There's a version of this argument that ends in an easy pitch. Buy the enterprise measurement suite. Hire the consultant who has proprietary access. Trust me, because I have the numbers.

I don't have the numbers. Nobody does. That's the argument. The people telling you they have the numbers are either selling you something or haven't looked closely enough at their own instruments to notice what's decoupled.

What I have is 18 years of watching the SEO industry cycle through this exact pattern — measurement instruments degrade, the industry pretends they haven't, someone eventually says so out loud, the market adjusts, new instruments emerge that are better for a while and then also decouple. We're mid-cycle. The instruments have decoupled. The market hasn't adjusted. The gap between what dashboards report and what's actually happening is the largest it's been in a decade, and it's still widening.

The clients who will do well over the next two years are the ones whose SEO decisions are anchored to server-side reality, human conversion truth, and qualitative evidence from actual customers — not the ones with the prettiest dashboards. The agencies who will do well are the ones honest enough to say "I don't fully know what's happening in AI search, here's what I can defensibly measure, here's what I can't, here's what we're going to do that's worth doing regardless."

Anyone claiming more certainty than that is reading a graph that's stopped measuring what it says it measures. And the confidence in the room is going to look like a very poor decision in eighteen months when the instruments finally catch up to where the ground already moved.