OpenAI, Google, Microsoft back one AI training opt-out

Two lines in one file may soon do what a growing list of crawler names does today.

An Internet-Draft by Anthropic's Kevin Kelley, dated Oct. 30, defines a robots.txt token, __usage_ai-training, that tells AI companies whether you want your content used for training. OpenAI, Google, Microsoft and Anthropic say they will support it, SiliconANGLE reports.

One token instead of a crawler list

Today, a site that wants out of AI training has to name each company's web crawler in robots.txt, and sometimes several per company. The draft replaces that with one group:

User-Agent: __usage_ai-training
Disallow: /

Per the draft, a disallowed path states a preference against AI training, an allowed path states a preference for it, and a site that never names the token states nothing. A User-Agent: * group does not count. The draft is explicit that the token is not a crawling directive: it says what may be done with content, not whether a crawler may fetch it.

The status matters. This is an individual draft, version 00. The IETF has not endorsed it, and it expires May 3, 2027 unless it moves forward.

Microsoft has the most detail. Its Bing post says it will honor the new preferences as it crawls starting Feb. 1, 2027. SiliconANGLE reports that OpenAI and Microsoft plan support in the coming months, Google sometime next year, and that Meta backs the idea without committing to it.

Training is only half the question

Here's the thing. Most publishers are not losing sleep over training anymore. They are losing traffic to AI answers. Reddit said this week its search referrals are falling.

This token does not touch that. Microsoft is the only company here with a separate control for grounding, its word for a model reading your page to answer someone's question. Its page-level setting has three values: Allow, DisallowExtended, which keeps your pages in Microsoft's own AI experiences but out of third-party grounding services, and Disallow.

Microsoft is also plain about the limits: "Neither choice guarantees inclusion, traffic, or payment."

So the verdict: a real simplification for one choice, with the bigger choice still company by company.

What to do with your robots.txt

The steps if you run a site:

• Decide on training data and on AI answers separately. They are now different switches.
• Add the two-line group above if you want out of training. It costs nothing to add early, though no company has said it is honoring it yet.
• If you use Bing's NoArchive or NoCache tags, Microsoft says it will stop reading them as training or grounding preferences after March 1, 2027. Set the new grounding preference before then.

A preference takes effect only after a crawler visits the page again, Microsoft notes.

Watch Google's date

Google's support is the one that moves the most traffic, and it has no firm date in what we could verify. Watch for it, and for Meta to commit.

Add the lines. Then decide what you want AI answers to do with your pages.