Shared AI opt-out explained: one robots.txt line to refuse training

Every website that wanted to keep its words out of AI training has had to play whack-a-mole: one rule for OpenAI's crawler, another for Google's, another for Anthropic's, and a new one whenever a lab launches a bot. On 30 October 2026, OpenAI, Google, Microsoft and Anthropic each said they will honor a single shared signal instead. It is a genuine simplification, but it's narrower than "opt out of AI" sounds, and understanding exactly where it stops is the whole story.

The plain-English version

Imagine your website is a public library reading room with a sign-in desk. For years, visitors who wanted to photocopy everything and use it to teach a student were turning up, and the only way to refuse was to give the desk a list of names: "Don't let Alice copy. Don't let Bob copy." Every time a new student showed up under a new name, you had to update the list, and if you missed one, they copied away.

The new deal is a sign on the door that doesn't mention any names. It says: "Copying this to teach students is not allowed." Four of the biggest copiers have said they'll read that sign and follow it. They are still allowed to walk in and read, because the sign is about what they do with what they read, not about who comes through the door.

That's the core distinction. Websites already have a standard text file, called robots.txt, where they leave instructions for automated visitors. The new proposal adds a pretend visitor named __usage_ai-training. It's not a real bot. It stands for a purpose: training an AI model. Whatever you write for that pretend visitor ("don't train on anything, except the press kit") is a message to every company that has promised to listen.

A few honest limits belong in the plain version too. Nobody is checking that the companies comply; it rests on their word. It only concerns future training, not what has already been collected. It does not make anyone pay you. And the companies are not all starting at the same time: Anthropic says it has already begun, Microsoft starts on 1 February 2027, Google sometime in 2027, and OpenAI "in the coming months". Finally, both Google and Microsoft warn that opting out of AI uses could make your content show up less in their AI products.

If you can tell a friend "four AI companies promised to honor one simple 'don't train on this' sign, but it's a voluntary promise that mostly starts next year and doesn't fix the lawsuits," you've got it right.

How it actually works

The problem: purpose versus bot name

robots.txt was designed in the 1990s to manage access: which crawler may fetch which path. AI creates a second question that access can't answer cleanly: what may the fetched content be used for? The same page might be fetched to build a search index, to answer a user's question live, or to train a model. A site owner might be happy with the first two and not the third. Under the old system you could only approximate this by guessing which named bot does which job, and each company named its bots differently.

The new idea: name the purpose, not the bot

According to the IETF Internet-Draft by Kevin Kelley of Anthropic, the mechanism adds no new syntax at all. A site writes an ordinary robots.txt group, but the User-Agent line names the policy token:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That reads as "no AI training on anything except /press-kit/". Path matching follows the existing robots.txt rules from RFC 9309, so anyone who has written a robots.txt file already knows how to use it.

What a participating company does

1. It fetches your robots.txt, as crawlers already do.
2. It looks for a group that names __usage_ai-training (matched case-insensitively).
3. For each page it holds, it checks whether the path is disallowed for that token.
4. If so, it should not use that page for training. It may also choose not to fetch pages it will only use for training, but the draft makes that optional.

Three rules that trip people up

Where companies differ

The signal is shared; what each company builds around it is not. According to PPC Land's reading of Google's documentation, Google will treat the shared token as a Google-Extended disallow (we could not read Google's post in full to confirm how an explicit Google-Extended rule interacts). PPC Land concludes this also removes grounding in Gemini Apps and Vertex AI, since Google-Extended covers both, while Google Search, AI Overviews and AI Mode remain governed by Googlebot and a Search Console setting. That is an interpretation, not a statement Google's post makes directly. Microsoft goes further and adds a separate page-level grounding control.

The deep dive

What the draft actually specifies

The document is an individual Internet-Draft, draft-kelley-aipref-training-preference-signal-00, dated 30 October 2026 and expiring 3 May 2027. It carries no IETF endorsement or formal standing. It sits beside the work of the IETF's AIPREF working group, whose vocabulary draft (draft-ietf-aipref-vocab) defines categories such as train-ai, ai-use and search, and whose attachment draft defines a Content-Usage robots.txt directive and HTTP header (for example Content-Usage: train-ai=n). Those AIPREF drafts are working-group documents still in progress (the attachment draft linked here is an older revision, -04, with a newer -05 available).

Kelley's draft takes a different route to the same destination: instead of a new directive, it reuses the robots.txt group grammar and a reserved product-token style name. The leading double underscore makes accidental collision with a real crawler name unlikely, although the draft acknowledges that some deployed parsers match loosely. Several design decisions are worth reading closely:

• Wildcards are excluded. "Wildcard groups MUST NOT be interpreted as matching the policy token." This avoids a blanket User-agent: * rule, written years ago for other reasons, from silently becoming an AI-training opt-out or a (more worrying for publishers) being read as no preference. Absence of a named group means no preference expressed.
• Multiple tokens per group. A group may list several policy tokens on consecutive User-Agent lines, and may mix them with ordinary product tokens. This leaves room for future purposes without new grammar.
• Precedence is soft. Specific rules "SHOULD" take precedence over general ones, and consumers may treat proprietary-token rules (like Google-Extended) as more specific. The draft also says "private agreements MAY override preferences", a clause that matters to anyone who has licensed content to a lab.
• Time of retrieval. A preference applies according to the robots.txt in effect when the content was retrieved. The draft doesn't address data collected before a preference was posted.
• Definition of training. It borrows the AIPREF vocabulary: modifying a generative model's learned parameters. Ranking, classification and retrieval at answer time sit outside. (PPC Land notes that OpenAI's own wording about "foundational" training doesn't exactly match this definition.)

Why a pretend crawler instead of a new directive?

Deployment cost. Every web server and CMS already serves robots.txt, and every crawler already parses it. Adding a new directive means waiting for libraries and parsers to learn it; reusing the User-Agent grammar means a parser that has never heard of the token simply treats it as a bot nobody is running, which is harmless. The cost is a little conceptual awkwardness: robots.txt now carries two kinds of groups, ones about who and ones about why. The AIPREF Content-Usage directive in the attachment draft is a different syntax for the same job, and other proposals (such as Cloudflare's Content Signals) also exist, so this announcement is partly a bet on which one the largest signers will actually read.

Company-by-company status

Details below are as reported by the companies' posts and secondary coverage; we could not read the OpenAI and Microsoft posts directly, so those rows rely on PPC Land and AFP.

Meta's spokesperson Andy Stone backed the idea on X, calling a common signal "an important step", but Meta wasn't among the pledgers. According to PPC Land, Meta, Amazon and Apple made no commitments.

A subtle trap: Microsoft's migration

PPC Land reports that Microsoft will stop reading NoArchive and NoCache as training or grounding signals after 1 March 2027, and that grounding has no site-wide robots.txt control, only page-level ones (Allow, Disallow, or DisallowExtended, which limits grounding to Microsoft-operated experiences). A site that relied on NoCache to keep pages out of Copilot answers and adds only the shared token could therefore lose a restriction it thought it had. That is a nice example of a general problem: one shared signal for training sits next to many non-shared signals for everything else.

What the numbers can and can't say

The announcement carries no benchmarks, but one piece of context is widely quoted. An analysis by digidai summarizing a Cloudflare press release of 15 September 2026 cites three figures: 36.6% of verified crawler traffic on Cloudflare's network comes from crawlers gathering content for both search and AI training; 17% of Cloudflare-hosted site owners restrict AI training; and under 1% block search crawlers. As that analysis stresses, these have different denominators and measure one network's customers, not the whole web, so they shouldn't be read as one chart. They do suggest where the pressure sits: site owners are already acting, and the shared token is partly an attempt to make that behaviour cheap to express and cheap to honor.

Why it matters

For everyday users. Little changes today. If the signal is honored, fewer sites might feel forced to hide behind login walls or block every bot, which would keep more of the open web readable. If you publish anything yourself (a blog, a portfolio, a small shop), you gain one place to say what you're comfortable with.

For developers and builders. Crawler authors now have a target for compliance: read a named group, check path rules, record the retrieval time. The wildcard exclusion and the "time of retrieval" language effectively require you to store when you fetched each page and what robots.txt said then, which is a data-governance task for anyone training on web data. If you run a small lab or open-source project, you aren't covered by the pledge, but you will be judged against it.

For companies and publishers. The practical gain is administrative: a single group instead of a bot inventory. The trade-offs are strategic. Google and Microsoft both warn that opting out of AI uses can reduce visibility in their AI experiences, so each publisher has to decide whether training exposure, answer exposure and referral traffic are the same thing for them. They're not, and the digidai analysis argues they need separate metrics.

For the field. There's a precedent angle: four direct competitors publicly aligning on a machine-readable preference vocabulary makes it harder to claim later that no workable signal existed. That matters for courts and regulators even though the draft explicitly avoids copyright claims. It also pushes toward a world where training data is something with provenance and permissions, a theme we explored in OpenAI's offline-training world, where the web stops being the default input.

What to be skeptical of

• It is a promise, not a mechanism of enforcement. The draft has none, participation is voluntary, and TechBuzz compares it to Do Not Track, a header many ad companies ignored. There is also no independent way for a publisher to verify what a model was trained on.
• Timing is thin. Only Anthropic claims to have started; OpenAI gives no date; Google and Microsoft are in 2027. Until then, per-bot rules are what actually work.
• Not a legal settlement. It doesn't touch past scraping, and doesn't address lawsuits such as the New York Times case against OpenAI and Microsoft, where the companies rely on fair use. The "private agreements may override" clause also means licensing deals can supersede a posted preference.
• Motives are mixed. TechBuzz notes that leadership on a standard could ease regulatory scrutiny for Anthropic ahead of its IPO and for Google and Microsoft amid antitrust investigations. That's the outlet's interpretation, not something the companies stated, but it's a fair reason not to read the announcement as purely altruistic.
• It's one draft among several. Cloudflare's Content Signals, the AIPREF Content-Usage drafts and an ai.txt proposal all compete. The AIPREF drafts are themselves still works in progress.
• Source limits. We could read the IETF draft, but not Google's post in full, and OpenAI's and Microsoft's posts only through secondary coverage. Details about their exact wording come through PPC Land and should be checked against the originals before you change a production robots.txt.

What to watch next

1. Anthropic's rollout. Since it says it has started, a published crawler documentation update or a test showing the token being respected would be the first real evidence.
2. OpenAI's date. "In the coming months" has to become a date and a documented behaviour, including how it interacts with GPTBot and OAI-SearchBot.
3. 1 February 2027. Microsoft's honoring date, and 15 December 2026 when Bing Webmaster Tools' transition controls are due to open (per PPC Land).
4. Google's mapping. Whether Google clarifies that a shared-token opt-out also removes Gemini grounding, and how it handles path-level token rules.
5. The IETF process. Whether Kelley's individual draft is adopted by the AIPREF working group before it expires on 3 May 2027, and whether it converges with the Content-Usage approach.
6. Non-signers. Whether Meta, Amazon, Apple, DeepSeek or open-source crawlers adopt it.

For related context on how AI companies are changing where their training data comes from, see OpenAI's offline world and our coverage of OpenAI's textGrain watermark, which tackles the other end of the provenance problem: marking what AI writes.

How this was made: researched and written by an AI model (Claude) from the primary sources listed above, then checked claim-by-claim against those sources in a separate AI fact-check pass. Spotted an error? Email [email protected] and we correct it publicly. Our process.