Personal Intelligence just broke the idea of a ranking

Google connected Calendar to Personal Intelligence in AI Mode yesterday. US only for now, more countries following. On the surface it's a small feature update — AI Mode can now add invites to your calendar and factor your existing schedule into its answers. Third data source after Gmail and Photos. Robby Stein announced it on X. That's the news.

The story is what it does to measurement.

Because once your Calendar, Gmail, and Photos are all shaping the answer, the same query no longer returns the same result to two different people. Not "roughly the same with some personalisation." Genuinely different answers, drawn from different sources, citing different sites. iPullRank's May test already showed this: connecting Gmail to Personal Intelligence changed which brands appeared in AI Mode responses to identical prompts. Calendar makes it worse, because time-of-day and existing commitments now factor in too.

The old idea of a search result — one page, one ranking, one set of sources, checkable — is over for anyone signed into a Google account and using AI Mode. What replaces it is a probability distribution of possible answers, weighted by whatever the user's connected apps happen to contain that morning.

The ranking check just stopped working

Here's the practical problem. For twenty years, "did we rank" was a question with an answer. You open incognito, type the query, look at position three. Done. Even after personalisation crept in around 2011, the variance was small enough to ignore for most commercial keywords.

That check no longer works in AI Mode. If Calendar shifts the answer based on what's on your schedule, and Gmail shifts it based on what you've booked or bought, then position three for you doesn't exist for me. There's no shared surface to check against.

Rank tracking tools have been quietly dealing with this for a while by running queries from clean environments — no login, no history, no personal context. That's fine for measuring the *baseline* AI Mode response, the one shown to a signed-out user or someone who hasn't connected any apps. But it tells you almost nothing about what's actually being shown to real people who use Google the way Google wants them to use it.

And Google wants everyone to connect everything. That's the entire product direction.

Two studies, published on the same day, that don't fit together

Two data points crossed my desk yesterday that on their own look useful, but placed side by side reveal how shaky the measurement floor has become.

The precision is entirely fictional.

SE Ranking published a study showing AI Mode ads now appear on 29.45% of commercial queries in the US. Nearly one in three. They analysed 50,032 keywords across 20 niches on 30 June. Solid methodology, clear finding.

Except they also flagged this: *"the real ad rate may be higher because AI Mode results are inconsistent across sessions."* Meaning even their clean-environment test found the same query returning different results run to run. Not because of personalisation — because AI Mode itself is non-deterministic. Same prompt, different output, no user context involved.

Then there's the arXiv paper published this morning on variance decomposition in LLM brand measurement. It found that when you ask an LLM the same question repeatedly across paraphrased prompts, different models, and different languages, only 1.5% of the variance in brand mentions comes from actual brand identity. The rest is noise: resampling variance (34.8%), brand-in-context interaction (29.6%), query language (26.5%).

A single AI answer, in other words, carries almost no signal about whether a brand is "genuinely" recommended. Reliability at a single-query level is around 0.01. To get to 0.36 reliability — still not great — you need to spread queries across multiple languages and multiple models.

Put those two findings together and the industry is currently doing this: measuring 29.45% of queries on a system that gives different answers to the same query run twice, then feeding that into dashboards that quote three-decimal-place "AI visibility scores." The precision is entirely fictional.

The measurement problem is the actual problem.

Why this is different from the personalisation everyone shrugged off

I know what the pushback is. "Search has been personalised for years, this is just more of the same, adapt or die."

It's not. And here's why the distinction matters.

Traditional personalisation shifted *rankings* on a *shared results page*. You and I might see position three and position five swap based on location or history, but we were looking at broadly the same ten blue links drawn from the same index. The set of eligible sources was stable. Personalisation was a re-shuffle on top.

Personal Intelligence with connected apps is different. It changes which sources are considered eligible in the first place. If your Gmail contains a receipt from a specific restaurant, AI Mode may draw on that as context, cite that restaurant's site, and skip a competitor it would have shown me. The candidate set itself has moved.

That's not a re-shuffle. That's a different game.

And it's the reason the "just track your AI visibility" pitch from every GEO tool vendor is starting to look shakier than it did six months ago. Not because the tools are lying — most are doing exactly what they claim — but because the thing they're measuring only exists in a clean-room state that fewer and fewer real users experience.

What this actually means for anyone doing this work

Three shifts, all uncomfortable.

First, single-query rank checks in AI Mode are close to useless as a measurement of what real users see. They're still fine as a baseline — is my site being cited by AI Mode when nothing else is influencing the answer? That's a valid question. But don't confuse that baseline with "my AI visibility."

Second, brand strength gets more important, not less. The arXiv paper is worth internalising: the more variance there is in individual answers, the more it matters that your brand shows up *across* many possible answers. You can't optimise a single query into being reliably cited. You can build enough breadth of citation, mention, and topical authority that whichever context Personal Intelligence pulls in, you're in the candidate set.

That is a brand-building argument, not a schema-and-markup argument. And it's why I keep coming back to the same position: brand is the moat in AI search. The more contextual the answer becomes, the more it favours brands the model has encountered often enough to reach for by default.

Third — and this is the one no vendor will say out loud — the honest answer to "how are we doing in AI Mode?" is currently *"we don't fully know, and neither does anyone else."* You can measure trends. You can measure baselines. You can spot obvious presence or obvious absence. But confident claims about specific citation share, movement, or ROI in AI Mode are running well ahead of what the measurement can support. I wrote about this in more detail in the AI visibility dashboards piece last month, and Calendar integration makes that argument stronger, not weaker.

The shape of the next twelve months

Google is going to keep connecting apps to Personal Intelligence. Nick Fox said as much in December. I/O signalled the direction. Calendar shipping this week is one more step in a roadmap that clearly extends to Drive, Maps, YouTube history, Chrome browsing, and whatever else Google can plausibly justify. Each connection adds another variance source to the arXiv paper's decomposition.

Which means the measurement problem doesn't stabilise — it compounds. Anyone selling you an "AI visibility platform" today is selling you a snapshot of a system that will look meaningfully different in six months and unrecognisable in eighteen.

The businesses that come out of this in a good position won't be the ones who bought the best dashboard. They'll be the ones who quietly kept building the underlying signals — real editorial authority, earned coverage, structural quality, brand recognition — that Personal Intelligence draws on regardless of which app it's pulling context from.

That's the loop. And we built it.