robots.txt vs noindex: the Claude chats lesson

Yesterday the story broke that shared Claude conversations — and in some cases Artifacts — were showing up in Google and Bing search results. TechCrunch and WIRED both reported it on July 27. If you use Claude's share button, that is worth knowing on its own. But the more useful part for anyone who ships to the web is the technical reason it happened, because it is the same mistake developers make constantly and it is almost never caught until something private ends up in a search index.

Here is the short version. When you clicked share on a Claude conversation, it produced a public URL under claude.ai/share/…. Anyone with the link could open it — that is the intended behavior of a share link. The problem is that some of those links leaked into places search engines crawl: social posts, forums, other indexed pages. Once a URL is discoverable, search engines will try to index it. And the shared Claude pages did not carry the one signal that tells a crawler "do not put this in your index." They were missing a noindex tag.

The part everyone gets wrong

The instinct, when you want to keep a URL out of Google, is to block it in robots.txt. That is exactly what Anthropic had done. According to a detailed writeup by Daniel Glover, the Internet Archive shows Disallow: /share/* appearing in claude.ai/robots.txt around August 2, 2025 — right after the near-identical ChatGPT shared-chat indexing story broke — and it has been there ever since. So the directory was disallowed. And pages got indexed anyway.

This surprises people every time, so it is worth stating plainly: robots.txt controls crawling, not indexing. Those are two different things.

• Crawling is a search engine fetching the content of a URL. Disallow in robots.txt asks well-behaved crawlers not to fetch that path.
• Indexing is a search engine listing a URL in its results. A page can be indexed without ever being crawled — if Google sees enough links pointing to a URL, it can index the URL itself (title from anchor text, no snippet) even though robots.txt told it not to fetch the body.

Google has documented this behavior for years. Their own guidance says, in effect: if you want a page kept out of search results, do not block it in robots.txt — use a noindex directive instead. And there is a trap inside the trap: noindex only works if the crawler is allowed to fetch the page and see the tag. If you both Disallow the path in robots.txt and add a noindex tag, the crawler never fetches the page, never sees the noindex, and can still index the bare URL from links. The two mechanisms fight each other.

Why noindex was the missing piece

WIRED reviewed a sample of the exposed Claude share pages and confirmed they did not include the noindex tag that both Google and Bing honor. That is the whole gap. The robots.txt Disallow was doing its documented job — discouraging crawlers from fetching /share/* — but it was never capable of preventing indexing of URLs that had leaked into the link graph. Reporting indicates Google began dropping the results around July 26 after Anthropic updated its configuration and added noindex handling.

It is easy to read this as an Anthropic-specific slip, but it is not. The identical thing happened to ChatGPT shared links a year earlier, which is presumably why Anthropic added the Disallow rule in the first place. Two of the most sophisticated AI companies on earth made the same web-fundamentals mistake, twelve months apart. That should tell you how counterintuitive this distinction is even for strong engineering teams.

What this means if you ship to the web

If you run any site with URLs that are technically public but not meant to be discoverable — share links, preview environments, password-reset landing pages, invoice URLs, staging domains, user-generated content pages — assume they can be indexed the moment a single link leaks. A link in a tweet, a Slack message that gets archived publicly, a referrer header that shows up in someone's analytics: any of these can introduce a URL to the crawl graph.

Concretely:

• Do not rely on robots.txt to hide anything. It is a crawling hint, not an access control, and it is itself a public file that advertises the paths you would rather people not look at.
• Use X-Robots-Tag: noindex as an HTTP response header for pages that should stay out of search. It works for non-HTML resources (PDFs, JSON) where you cannot add a meta tag, and it applies before any HTML parsing.
• Keep noindex pages crawlable. If you noindex a path, do not also Disallow it in robots.txt, or the crawler will never see the tag.
• For genuinely private content, require authentication. A "secret" URL is not a security boundary. If exposure would be harmful, the only correct answer is a login, a signed expiring token, or server-side access checks.

The AI-agent angle

There is a second-order lesson here for anyone building agentic systems. Share features, public artifact hosting, and "send someone this conversation" flows are becoming standard in AI products. Every one of those features mints a public URL that may contain whatever the user pasted into the chat — API keys, internal docs, customer data, half-finished code. The convenience of a shareable link is in direct tension with the sensitivity of what people put into these tools.

If you are building a product that generates public URLs from user content, treat search-engine exposure as a design requirement, not an afterthought. Default share pages to noindex via an HTTP header, make the noindex path crawlable so the tag is actually seen, and be explicit with users about what "share" means. The gap between "anyone with the link" and "anyone searching Google" is enormous, and users do not intuitively understand that a share link can become a search result.

The takeaway

This was not an exotic vulnerability. No exploit, no breach — just the crawl-versus-index distinction that has tripped up developers since robots.txt was invented in 1994, applied to a new generation of AI share links. The fix is one HTTP header and a mental model: robots.txt manages crawling, noindex manages indexing, and authentication manages privacy. Pick the right tool for what you actually need, and never assume an unlinked URL will stay unlinked.