The Claude chats leak is a robots.txt lesson we keep failing

Over the weekend, thousands of shared Claude conversations turned up in Google search results. Medical details. Internal company documents. Files with the names and phone numbers of primary school-aged children. All findable through a basic `site:` query, which is how 404 Media and TechCrunch surfaced them.

Anthropic's response was that share links only get indexed if people post them somewhere crawlers can see. Which is technically true and completely beside the point.

The actual story is that Claude's share pages were configured with a control setup that has been documented as broken for years. Robots.txt disallowed the share path. An `X-Robots-Tag: none` header was set on the same URLs. Matt Southern's write-up at Search Engine Journal on Monday confirmed both were still in place on 27 July, after the pages were already gone from search.

This is the industry's most persistent technical misconception, sitting inside a live AI company at production scale. It is worth pulling apart, because I see the same mistake in client audits every single week.

Robots.txt is a crawl instruction, not an index instruction

The two things sound similar. They are not the same thing.

Two controls. Neither did the job. Both pointed at each other and cancelled out.

Robots.txt tells a crawler whether it's allowed to fetch a URL. That's it. It doesn't tell Google whether it's allowed to *list* a URL in search results. Google has been saying this for over a decade. John Mueller has said it. Martin Splitt has said it. The official documentation says it in bold.

If Googlebot can't fetch a page because robots.txt blocks it, but Google finds links to that page from anywhere else on the web, the URL can still appear in search results. Google just won't have a description for it — because it hasn't read the page. The URL itself is still fair game.

Noindex is different. Noindex is a directive Google reads *from inside the page or its HTTP headers*. It tells Google: you may crawl this, but you may not include it in search results. It's the only reliable way to keep something out of the index.

The two directives fight each other. If you block a page with robots.txt, Google can never fetch the noindex, because it never reads the page. So the noindex is invisible. The URL floats freely, indexable via any external link.

That is what happened at Claude. The share path was blocked in robots.txt. The `X-Robots-Tag: none` header — which Google treats as equivalent to noindex, nofollow — was served on the same URLs. Whether the header was in place before the weekend is unclear. Southern couldn't confirm. But even if it was, blocking the crawler meant Google couldn't read it, which meant it had no effect on whatever links had already been discovered.

Two controls. Neither did the job. Both pointed at each other and cancelled out.

This is not an AI company mistake — it's an industry-wide mistake

I want to be clear about something: this is not a story about Anthropic being uniquely careless. It's a story about a mistake that is baked into how most technical SEO people think about crawler controls, and one that shows up in the majority of technical audits I run.

The pattern I see most often: a client wants to hide a directory. Staging area. Old campaign pages. Internal search results. Admin URLs. They — or their previous agency — add a `Disallow` line to robots.txt, feel satisfied, move on. Six months later, the pages are in Google, indexed with no description, sometimes ranking for branded queries, occasionally leaking sensitive URL structures.

The client thinks they've hidden the pages. They've done the opposite. They've told Google not to look inside, but they've done nothing to stop Google from writing the URL on the wall.

The Claude situation is the same mistake at higher stakes. The chats were shared with links intended for private circulation. Anthropic added robots.txt controls and an `X-Robots-Tag`. The controls contradicted each other. The chats got indexed.

What actually works

The Google guidance on this hasn't changed in years, and it isn't complicated:

If you want a page out of search results, use noindex. Full stop. Don't block it in robots.txt, because if you do, Google can't read the noindex you've applied. Let Googlebot crawl the page. Let it see the noindex. Let it remove the URL from its index. Once the URL is properly deindexed, *then* you can consider adding a robots.txt block if you want to save crawl budget.

The order matters. Noindex first, robots block second, and only if you need it. Never both simultaneously on a page that isn't already deindexed.

For genuinely sensitive content, none of this is the right control anyway. Robots directives are hints. Well-behaved crawlers respect them; badly-behaved ones ignore them. If a URL leads to something that must not be public, the answer is authentication, not a meta tag. A password. A signed URL. Server-side access checks. The only reliable way to keep something out of a search index is to make sure the URL doesn't return content to anonymous requests in the first place.

Anthropic's spokeswoman told TechCrunch the share links "are not guessable or discoverable unless people choose to share them themselves." That's true. But it's also security theatre for anything sensitive. Once a URL is shared — pasted into a Slack channel, a support ticket, a tweet, a Zapier workflow — it can end up somewhere crawlable, and from that moment on, robots.txt is not going to save you.

Why this keeps happening

Two reasons, both cultural rather than technical.

The first is that robots.txt *feels* like it's doing something powerful. It's the file that talks to search engines. You edit it, and things change. The mental model — this file controls what Google sees — is intuitive and wrong. The reality is more subtle: this file controls what Google *fetches*, which is not the same as what Google *shows*.

The second is that noindex requires actually reading the page. Which means it requires the developer or SEO to think about *how* Google encounters the URL, not just whether it can. Most technical documentation for developers doesn't make this distinction cleanly. Frameworks make it easy to add a robots.txt entry and hard to reliably serve an `X-Robots-Tag`. So people reach for the tool that's closest to hand, even when it's the wrong one.

The AI industry has now imported this misconception at scale. Share features on LLM products, artifact pages, hosted mini-apps — all of these are being deployed by teams that treat "put it in robots.txt" as a privacy control. It isn't. It never was. And the moment one of those URLs ends up in a place a crawler can see, the pages are in the index whether you wanted them there or not.

What this actually means for you

If you run a site — any site — this is the audit item to check this week.

Look at your robots.txt. For every `Disallow` line, ask two questions. First: is there anything at that path you'd genuinely prefer not to appear in Google? Second: if there is, does that path also serve a noindex tag or `X-Robots-Tag: noindex` header when Googlebot fetches it *without* the block?

If the answer to the first question is yes and the second is no, you have Claude's problem in miniature. The URLs are one shared link away from surfacing in a `site:` search.

The fix isn't complicated. Remove the robots.txt block. Add noindex. Wait for Google to recrawl and deindex. Then, once the pages are out, decide whether you actually need the robots block back for crawl budget reasons. Usually you don't.

And if the content genuinely needs to be private — real client data, internal documents, anything with names or personal information — get it behind authentication. Meta tags are not a security layer. They never were.

The Claude leak will be forgotten by next week. The technical mistake behind it will keep showing up in audits for the next ten years. It's the most durable misconception in SEO, and it's now the most durable misconception in AI product infrastructure too. Same mistake, bigger stakes.