Google, Microsoft, OpenAI and Anthropic to honour one shared AI training opt-out

One rule for four AI crawlers

Google, Microsoft, OpenAI and Anthropic have committed to honour a single opt-out from generative AI training, written into a website's robots.txt file. Until now, a publisher wanting to refuse had to write a separate rule for each company's crawler. The proposal is an IETF draft by Kevin Kelley, a technical lead for standards at Anthropic, published on Friday, 30 October 2026. Google, OpenAI and Microsoft announced their support the same day, and Anthropic has already started building the signal in. Krishna Madhavan, principal product manager for Microsoft Bing, wrote:

“Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”

How the signal works

The shared signal lives in robots.txt, the plain-text file that already tells crawlers what they may fetch. A site adds a group for a new policy token, __usage_ai-training, and then lists the paths it wants kept out of training. The draft's example closes an entire site while leaving a press kit open. Sites can therefore keep specific sections available while refusing the rest. The token covers AI training only, and according to Microsoft, scraping for training remains allowed by default wherever no preference is posted. Google will read the token as a block on Google-Extended, its own AI training control, unless a site names Google-Extended directly.

Who starts when

Anthropic has already started work on the signal and says it will complement its existing crawler controls rather than replace them. Kevin Kelley wrote that adding support across the company's services would take time:

“It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover.”

OpenAI aims to implement the signal in the coming months, and its existing crawler controls, including GPTBot, will still apply. Google aims to support the signal in 2027, and its crawler documentation says early 2027. Microsoft will honour the new preferences from 1 February 2027, replacing the NoArchive and NoCache tags it has used since 2023. It is also adding a page-level setting for grounding, which applies when an AI answer draws on a live web page: one option allows Microsoft's own products such as Copilot, while another blocks the grounding services it sells to other companies.

What the opt-out leaves untouched

The signal cannot undo past scraping, because the companies already hold any content they crawled before. It does not stop crawling altogether, and Microsoft and Google say it will not remove sites from their search results. Microsoft says websites will still appear in Bing search results regardless of their preferences. Google Search Advocate John Mueller wrote that sites allowing Google-Extended can appear in Gemini app answers and receive clicks. Microsoft has warned that refusing the grounding option will reduce a site's visibility in answers from assistants such as Copilot. Meta, the parent company of Facebook, called the move an important step through its communications chief, Andy Stone, but did not say whether it would respect the signal.

The copyright fight behind the move

The announcement comes as publishers face a steady flow of bots collecting their articles to feed AI models, and some have already gone to court. The New York Times has accused OpenAI and Microsoft since December 2023 of using millions of its articles without permission or payment to train ChatGPT. Both companies invoke a copyright exception. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and the Really Simple Licensing standard lets publishers charge the robots that visit their pages. Microsoft says the new arrangement creates no licensing or payment deals.