OpenAI, Google, Microsoft, and Anthropic to Support a Shared robots.txt Token for Opting Out of AI Training - Start Dates Differ by Company
A token called __usage_ai-training, which lets a site tell several AI companies through one robots.txt group that it does not want its content used for AI training, was submitted to the IETF as a draft, and four companies said they will support it. Existing User-Agent: * rules are not treated as a preference. Microsoft will honor it from February 1, 2027, and is adding a separate grounding setting alongside training.
On October 30, 2026, a draft was submitted to the IETF (Internet Engineering Task Force) describing a way for website operators to tell multiple AI companies, with one entry in robots.txt, that they do not want their content used for AI training1. The draft defines a “policy token” named __usage_ai-training, and Google and Microsoft each published plans to support it that same day34.
According to The Verge, OpenAI, Google, Microsoft, and Anthropic all say their AI crawlers will hold back from sites that carry this token6. Until now, a site that wanted to refuse training had to look up each AI company’s crawler or control name and add rules for them one by one. The companies are not starting at the same time, though, and the draft has not been agreed on as an IETF standard.
How to write it in robots.txt
No new syntax is involved. You create an ordinary robots.txt group and put __usage_ai-training on the User-Agent line where a crawler name would normally go1. Microsoft’s guidance gives this example for refusing training across a whole site4:
User-Agent: __usage_ai-training
Disallow: /
Specifying a path, as in Disallow: /example-path/, limits the preference to that part of the site. The draft also gives an example in which a site objects to training everywhere except under /press-kit/, where it allows it. Matching uses the same path rules and longest-match precedence as RFC 9309, the robots.txt specification2.
The draft implies three practical points1.
User-Agent: *does not match. The draft says wildcard groups must not be treated as matching this token. A robots.txt file with noUser-Agent: __usage_ai-trainingline has stated no preference at all, and the author of the proposal notes that every robots.txt file published today is in that position2- It is separate from crawl permission. The token expresses a preference about how content is used; it does not tell crawlers what to fetch. Rules for individual crawler names are read exactly as before, so a site that wants to stop crawling itself still needs groups for those crawler names
- It covers generative AI training only. The token borrows its meaning from how the vocabulary draft of the AI Preferences (AIPREF) working group at the IETF defines “AI Training”: using content to adjust the parameters a generative AI model has learned. Training models used only for non-generative tasks is excluded
The draft says more specific statements should win over general ones, and lets each company document that it treats rules for its own proprietary tokens as more specific than __usage_ai-training1. If a site writes both, which one applies has to be checked against each company’s documentation.
When each company plans to start
| Company | What it said | Timing |
|---|---|---|
| Anthropic | The author of the draft posted that Anthropic has begun implementing support, that it adds to rather than replaces existing crawler controls, and that covering all of its services will take time2 | Implementation underway |
| OpenAI | Expects to add support within the next few months, according to The Verge6 | Coming months |
| Plans to treat a site that opts out of training with this token, and does not name Google-Extended, as if it had disallowed Google-Extended3 | Aims to support it in 2027 | |
| Microsoft | Applies robots.txt settings to a site or path, and page-level settings to individual pages4 | Applies them during crawling from February 1, 2027 |
The Verge describes Kevin Kelley, who proposed the draft, as Anthropic’s technical lead on standards work6. Google says Google-Extended, its AI training control since 2023, will keep working, and that sites which explicitly allow Google-Extended can appear in Gemini app responses and receive clicks through grounding3. Anthropic’s help page already explains that restricting ClaudeBot signals that a site’s future content should be left out of its training data5. The proposal’s author also says Anthropic’s support does not replace its existing crawler controls. Microsoft, by contrast, is replacing its existing directives with the new preferences, as described below.
Microsoft separates training from grounding
On top of the shared training preference, Microsoft has added its own “AI Grounding” preference4. Grounding means using content as input when an AI generates a response, and it does not include indexing or ranking in traditional search results. The idea is close to RAG. A site can, for instance, refuse training while still allowing its pages to be used when responses are generated.
- AI Training: Set per site or path in robots.txt, and Microsoft also extends it to individual pages. The page-level form is
<meta name="Bingbot" content="ai-training:disallow">orX-Robots-Tag: Bingbot: ai-training:disallow, and a page-level setting overrides robots.txt - AI Grounding: Page-level only; there is no way to set it for a whole site in robots.txt. Besides Allow (the default) and Disallow, there is DisallowExtended. It permits use in Microsoft’s own experiences (AI Mode, AI Overview, Copilot, Microsoft 365) while excluding the grounding services Microsoft sells to other companies, for example Web IQ or API-based grounding in Foundry
- If nothing is set, both are treated as Allow. The settings do not change whether a site can appear in traditional Bing search results, and they do not create any licensing or payment arrangement. Microsoft cautions that choosing DisallowExtended or Disallow will sharply reduce a site’s visibility in AI-generated experiences
For training and grounding, the new preferences replace Microsoft’s existing NoArchive and NoCache4. Sites that have used NoArchive or NoCache to limit AI use face this transition schedule:
- December 15, 2026: A temporary grounding setting opens in Bing Webmaster Tools for sites that are using NoArchive or NoCache on that date
- February 1, 2027: Microsoft begins applying the new preferences when it crawls pages
- March 1, 2027: After this date, NoArchive and NoCache no longer count as training or grounding preferences. To keep grounding restricted, a site must add the new preference to its pages and record a temporary setting in Bing Webmaster Tools by this date. There is no temporary setting for training, which needs robots.txt or page-level markup
- September 30, 2027: Temporary settings expire (for any page where a recrawl finds the new preference earlier, it is replaced at that point)
Microsoft also notes that publishing a setting does not activate it; it takes effect only after Microsoft crawls the page and finds it4.
Still a draft, with open questions
The document is an -00 version submitted under an individual’s name. It carries no IETF endorsement, and it expires on May 3, 20271. Its author says it is offered as input to the working group’s ongoing work on how preferences are attached to content2. Microsoft likewise writes that the working group is still moving toward a final standard4. Google says the proposal fits with the EU AI Act’s Code of Practice for general-purpose AI model providers3, which, by Google’s account, treats robots.txt as a main way for publishers to state their preferences in machine-readable form.
The draft also allows private agreements to take precedence over preferences stated this way1.
This site has covered several cases where each AI company required its own instructions. In Meta’s revised commercial terms, Meta split its crawlers by purpose and told site owners to add a robots.txt rule for each crawler they wanted to refuse. In litigation, Newsday says in its complaint that its robots.txt barred both OpenAI and Common Crawl from crawling. If the shared token comes to be widely honored, chasing each company individually to refuse training should become less necessary. How crawling and the use of content in search and AI answers are handled, however, will still depend on each company’s own mechanisms.
What site operators can check now
If you manage robots.txt for a company site or blog, these are the points to look at:
- To refuse training through this mechanism, you have to add an explicit
User-Agent: __usage_ai-traininggroup. A Disallow underUser-Agent: *has no effect on this token - Because companies are starting at different times, it seems sensible for now to keep existing company-specific entries such as Google-Extended and ClaudeBot as well. If the shared token and a company-specific entry say different things, check each company’s documentation for which takes precedence
- Sites that have used NoArchive or NoCache on Bing to limit AI use need to move to the new preferences by March 1, 2027, deciding separately on training and grounding
- With Microsoft, refusing grounding lowers visibility in AI-generated experiences such as Copilot. Decide separately whether you want to refuse only training or also use in generating responses
Sources
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) - IETF Internet-Draft (October 30, 2026)
- New draft: draft-kelley-aipref-training-preference-signal-00 - IETF AI Preferences WG mailing list, post by the draft’s author (October 30, 2026)
- Supporting a shared AI training control for websites - Google Search Central Blog (October 30, 2026)
- Your content, your choice: clearer preferences for training and grounding - Microsoft Bing Blog (October 30, 2026)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? - Claude Help Center (checked October 31, 2026)
- The biggest names in AI are backing a new way to opt out of scraping - The Verge (October 30, 2026)
Was this article helpful?
Thank you!
Received. Thank you!