Google, Microsoft, OpenAI and Anthropic said on October 30, 2026, that they will honor a shared robots.txt signal telling them whether a site allows its pages to be used to train their generative AI models. Instead of writing one rule per company, a site owner will be able to state that preference once, under a common name: __usage_ai-training. The mechanism comes from an Internet-Draft that Anthropic’s Kevin Kelley posted the same day to the IETF, the body that sets internet protocol standards.
The timelines differ. Anthropic says it has started implementing the signal, OpenAI is aiming for “the coming months,” Microsoft will read it starting February 1, 2027, and Google plans support in “early 2027.” Microsoft is also using the announcement to separate training from the use of pages in Bing’s generated answers, on a schedule that starts December 15, 2026.
One more group in robots.txt
The draft adds no new directive and keeps the existing robots.txt syntax and groups. In a group’s User-Agent line, it puts a “policy token,” a name for a use of content. The draft gives this example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That group states a preference against training for the whole site, except the /press-kit/ folder, where the preference is in favor. Path rules and longest-match precedence stay as defined in RFC 9309, the robots.txt standard published in 2022.
Two rules in the draft matter for existing files. A User-Agent: * group never counts as a preference: under the draft, a site that blocks every crawler has said nothing about training. A file with no __usage_ai-training group expresses no preference at all; as Kelley wrote on October 30, that described every file published up to then.
The token doesn’t control crawler access. Rules written for Googlebot or Bingbot still apply, and according to Kelley the draft only asks crawlers not to fetch what they can’t use.
The definition of training comes from the vocabulary of the IETF’s AIPREF working group: modifying the learned parameters of a generative AI model. The training of models used only to classify, rank or score content isn’t covered. Our October 10 story on AI crawlers and robots.txt lists the crawler names published by the main AI companies.
Google maps the token to Google-Extended
On the Search Central blog, Google Search Advocate John Mueller laid out how the company will read the signal. If a site opts out of training with the shared token and doesn’t mention Google-Extended, Google plans to treat that as if the site had disallowed Google-Extended, the control it has offered since 2023. Each company’s own controls keep working alongside the shared signal. The same day, Google added a note to its crawler documentation saying support is planned for “early 2027.”
The effect of that mapping depends on what Google-Extended covers. According to that page, it lets sites decide whether content Google crawls is used to train future Gemini models and to ground answers in the Gemini apps and in Vertex AI’s Grounding with Google Search service. It isn’t a ranking signal and has no effect on whether a site appears in Google Search, including in AI Overviews, where Google is testing source descriptions it writes itself. The post adds that sites explicitly allowing Google-Extended remain eligible to appear in Gemini’s answers and to get clicks from them.
Google describes the approach as consistent with the European Union’s Code of Practice for general-purpose AI model providers under the AI Act. According to Google, that code recognizes robots.txt as a primary way for publishers to express a machine-readable preference.
Bing splits training from generated answers
At Microsoft, the announcement came in a post by Krishna Madhavan, a principal product manager at Bing. The training preference goes in robots.txt with the shared token, and Microsoft also lets sites set it on individual pages, with a meta tag or an HTTP header such as X-Robots-Tag: Bingbot: ai-training:disallow. A page-level preference overrides robots.txt.
A second preference, Microsoft’s own, covers grounding: using a page to inform a generated answer. It’s set page by page, with three values: Allow (the default), Disallow or DisallowExtended. DisallowExtended limits use to products Microsoft operates, such as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes the grounding services Microsoft provides to third parties, such as Web IQ or the grounding API in Foundry.
Bing warns that opting out of grounding will significantly reduce a site’s visibility in its AI answers, without affecting its eligibility for traditional search results. Allowing grounding or training, it says, doesn’t guarantee inclusion, traffic or payment.
These settings will replace the NoArchive and NoCache tags, which Microsoft has used since 2023 as training and grounding controls. Starting December 15, 2026, sites that use those tags will be able to record a temporary grounding preference in Bing Webmaster Tools. From February 1, 2027, Bing will apply the new preferences as it crawls pages; after March 1, 2027, it will stop reading NoArchive and NoCache as an opt-out. Temporary preferences expire when Bing recrawls a page and finds its new grounding preference, or on September 30, 2027, at the latest.
OpenAI and Anthropic set no firm date
OpenAI says it will honor the shared signal, which it ties to “foundational generative AI training,” and calls the move a “voluntary effort.” The company aims to implement it “in the coming months.” The signal won’t replace its current robots.txt controls, which will still apply.
On October 30, OpenAI also published a post titled “Fair use benefits everyone.” In it, the company defends training on copyrighted works as fair use, an exception in U.S. copyright law, as it fights a lawsuit from The New York Times.
At Anthropic, Kelley wrote on the working group’s mailing list on October 30 that the company has “already started implementing support” for the draft, on top of its existing crawler controls. He warned that adding support across Anthropic’s services will take time and that the way the company interprets these signals may change as standards evolve.
Meta, which delayed the terms that open business websites to its crawlers until November 20, was not part of the October 30 announcements.
An individual draft the IETF hasn’t endorsed
Kelley’s text is an individual Internet-Draft. Its IETF page notes that it isn’t endorsed by the IETF and has no formal standing in the standards process. It expires May 3, 2027. The AIPREF working group, chaired by Mark Nottingham (Cloudflare) and Suresh Krishnan (Cisco), is still working through the rest of its vocabulary, whose version 08 is dated September 13, 2026.
Nate Hake of Travel Lemming, who takes part in the group, made that point on X on October 30, in posts quoted by Search Engine Roundtable. The work is “not actually done at the ietf level,” he wrote, and “some significant areas of dispute remain.” He objects that the current draft exempts anything a user does from these preferences, which he says hurts creators whose work gets copied by mass-produced AI content. He added that “generally the process has been productive.”
The signal is only a preference. Under the draft, the preferences that apply to content are the ones in robots.txt when it was fetched, so an opt-out added today doesn’t cover pages already collected. Bing notes that its settings create no licensing or payment arrangement, and the draft acknowledges that private agreements can override the stated preference.
What it means
Until now, opting out of training meant knowing each company’s crawler names and keeping up with their changes. On October 30, four of the leading model developers committed to one shared name. For publishers and online stores that depend on Google, opting out of training without dropping out of search gets simpler, at least on paper: neither Google-Extended nor Bing’s training preference removes a site from traditional results.
The commitments are uneven. Anthropic says it has started, Microsoft gives dates, Google an interpretation rule and a target, and OpenAI only an intention.
Google is the least clear case. Because the company treats the shared opt-out as a Google-Extended opt-out, and Google-Extended also covers grounding in Gemini, a site that refuses training through the token could drop out of Gemini’s answers too, unless it allows Google-Extended separately. The post doesn’t spell that out, but its line about sites that explicitly allow Google-Extended, along with Google’s documentation, points that way.
Generated answers in search fall outside the signal. Among the October 30 announcements, only Bing adds a separate grounding setting, and appearing in Google’s AI Overviews doesn’t depend on this token.
OpenAI’s date, whether other companies such as Meta or Apple will join, and the IETF’s final text are all still unknown. The next markers are Bing’s December 15 window, its milestones on February 1 and March 1, 2027, and the update OpenAI has promised.
Sources
- Supporting a shared AI training control for websites, Google Search Central, October 30, 2026
- Google’s common crawlers (Google-Extended), Google for Developers, updated October 30, 2026
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), IETF Datatracker, October 30, 2026
- New draft: draft-kelley-aipref-training-preference-signal-00, Kevin Kelley on the IETF ai-control list, October 30, 2026
- Making it easier for websites to express AI training preferences, OpenAI, October 30, 2026
- Your content, your choice: clearer preferences for training and grounding, Microsoft Bing Blog, October 30, 2026
- A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), IETF Datatracker, September 13, 2026
- Google Adopts New AI Training Control, Search Engine Roundtable, October 30, 2026
Header image: MACCUS diagram. Data: Google Search Central, Microsoft Bing, OpenAI, Kevin Kelley (Anthropic) on the IETF ai-control list, IETF Datatracker, October 30, 2026
