The AI Training Opt-Out Is Coming to robots.txt. It Won't Take You Out of AI Answers

“If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences.”

That sentence is from Microsoft’s own announcement on October 30, 2026. It’s a blunt warning, and it’s the most useful line in a busy week for AI training opt-outs.

Two things happened on the same day. Google, Microsoft and OpenAI announced they will honor one shared robots.txt signal for opting out of AI training, and Search Engine Land reported that Anthropic is joining them. Separately, Bing split its AI controls in two: one for training and one for grounding, which means using your page to write an answer right now.

For a local business or a marketer, the main point is simple. An AI training opt-out decides whether your content teaches future models. It doesn’t decide whether AI assistants can find you, name you or link to you today. Those are different controls, and mixing them up is how businesses vanish from AI answers by accident.

Training vs grounding, in plain terms

• AI training means using your content to change a model’s learned parameters. That’s the working definition from the IETF group behind the new signal, quoted in the draft. Opting out affects future models, not what an assistant reads when it answers a question.
• AI grounding means using your content “as input to help inform AI-generated responses,” in Microsoft’s words. When Copilot or Gemini looks something up and cites a source, that’s grounding. It’s how a business gets named and linked in an answer.

A local bakery, dentist or law firm that wants to be recommended by AI cares most about grounding and search. Training is a separate question about the content they publish.

What was announced

The shared AI training opt-out in robots.txt

The signal comes from an Internet-Draft, Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and published October 30, 2026. It defines one cross-company “policy token,” __usage_ai-training, used in the User-Agent line of robots.txt. This is the draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, that says “no AI training” for the whole site except /press-kit/. A few details matter in practice:

• It’s a preference, not a crawl block. The draft says the token “is not a directive for crawling behavior.” Your rules for Googlebot, Bingbot, GPTBot and the rest work exactly as before.
• User-Agent: * doesn’t count. A wildcard group must not be read as matching the token. If there’s no __usage_ai-training group, no training preference has been stated.
• Company-specific rules still apply. The draft says specific rules “SHOULD take precedence over general ones,” and companies may treat their own tokens as more specific.
• It isn’t a standard yet. The datatracker page says an individual draft “is not endorsed by the IETF and has no formal standing.” This version expires May 3, 2027. Nate Hake, a publisher who took part in the working group, told Search Engine Roundtable that “some significant areas of dispute remain.”

Here’s what each company said:

• Google plans to read the signal as if you had disallowed Google-Extended, as long as you haven’t set Google-Extended rules yourself. The blog says support is coming “in 2027,” and Google’s crawler documentation now says “early 2027.”
• OpenAI says it will honor the signal and aims to add support “in the coming months.” Its existing crawler controls “will continue to apply.”
• Microsoft will start honoring AI training preferences from robots.txt on February 1, 2027. It is also adding page-level overrides (more below).
• Anthropic: Search Engine Land and Search Engine Roundtable report that Anthropic is adopting the signal, and its engineer wrote the draft. We didn’t find an Anthropic announcement with a date.

The Google catch: Google-Extended also covers Gemini grounding

This is the detail that’s easy to miss. Google’s crawler documentation says Google-Extended controls whether your content is used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google’s announcement says sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”

So when the shared opt-out arrives, Google will treat it as a Google-Extended disallow unless you’ve set Google-Extended rules yourself. Based on Google’s docs, that can cost you grounding and clicks in Gemini Apps too, not just training. Google also says Google-Extended doesn’t affect inclusion or ranking in Google Search. AI Overviews and AI Mode have their own control in Search Console, which we covered in what Google’s AI Overviews off switch actually removes.

Bing’s split: AI Training and AI Grounding

Microsoft’s post adds a separate grounding preference next to the shared training signal:

• AI Training can be set site-wide or by path in robots.txt with the shared token. It can also be set for a single page with a meta tag or HTTP header, which overrides robots.txt. Microsoft’s example: <meta name="Bingbot" content="ai-training:disallow">.
• AI Grounding is set page by page only, with a meta tag or X-Robots-Tag header such as <meta name="Bingbot" content="ai-grounding:disallow">. Microsoft says “there is no site-wide robots.txt grounding preference.” There are three options:
  • Allow (the default, no action needed)
  • DisallowExtended, which keeps your content in Microsoft’s own experiences (Microsoft lists AI Mode, AI Overview, Copilot and Microsoft 365) but keeps it out of grounding services Microsoft sells to third parties, such as Web IQ and the grounding API in Foundry
  • Disallow, which means your content isn’t used to inform generated responses at all
• Bing search results aren’t affected. Microsoft says these settings “do not change eligibility for traditional Bing search results” and “govern use of content, not crawl access.”
• The old tags are being retired. The new preferences replace NoArchive and NoCache. After March 1, 2027, Bing won’t read those tags as training or grounding preferences.

Microsoft also notes that neither setting “guarantees inclusion, traffic, or payment.”

What each control blocks, and what it doesn’t

Control	What it blocks	What it doesn’t block	Who honors it	When
Shared training signal (__usage_ai-training in robots.txt)	Use of your content to train generative AI models, at participating companies	Crawling, Google Search, Bing search results, and each company’s own crawler rules	Google (as a Google-Extended disallow), Microsoft, OpenAI; Anthropic reported	Microsoft Feb 1, 2027; Google early 2027; OpenAI “coming months”
Google-Extended disallow	Training future Gemini models and grounding in Gemini Apps and Vertex AI	Inclusion or ranking in Google Search	Google	Available now
Search Console “Search generative AI” control	Your content in AI Overviews and AI Mode	Regular blue-link results	Google	Available now
Bing AI Grounding: Disallow (page-level)	Your page informing answers in Bing, Copilot, Microsoft 365 and third-party grounding services	Bing search results, crawling	Microsoft	From Feb 1, 2027
Bing AI Grounding: DisallowExtended (page-level)	Third-party grounding services such as Web IQ and the Foundry grounding API	Microsoft’s own AI experiences, Bing search results	Microsoft	From Feb 1, 2027
GPTBot / ClaudeBot disallow (robots.txt)	Crawling for OpenAI and Anthropic model training	ChatGPT search (OAI-SearchBot) and Claude search (Claude-SearchBot), which use separate bots	OpenAI, Anthropic	Available now
NoArchive / NoCache	Bing’s older training and grounding controls	n/a	Microsoft	No longer read for training or grounding after Mar 1, 2027

Sources: Google, Google crawler docs, Google Search Console help, Microsoft Bing, OpenAI crawler docs, Anthropic crawler docs, IETF draft.

The timeline

• October 30, 2026: Google, Microsoft and OpenAI announce support for the shared training signal, and the IETF draft is published.
• December 15, 2026: Bing Webmaster Tools opens temporary grounding preferences. According to Microsoft, these are available only to sites that still use NoArchive or NoCache on that date. Every other site has nothing to do.
• February 1, 2027: Microsoft starts honoring the new AI Training and AI Grounding preferences as it crawls pages.
• March 1, 2027: The deadline for NoArchive/NoCache sites. If you want to keep your restrictions, publish the new page-level preferences and record a temporary grounding preference in Bing Webmaster Tools before this date. Microsoft says a site that misses it defaults to Allow for grounding until Bing recrawls the page.
• Early 2027: Google plans to support the shared signal. Its blog says only “in 2027.”
• May 3, 2027: The current version of the draft expires unless it’s revised.
• September 30, 2027: Bing’s temporary grounding preferences expire.

An EU court adviser draws the same line

On November 3, 2026, Advocate General Maciej Szpunar gave his opinion in Like Company v Google (C-250/25), the first generative AI case to reach the EU’s top court. A Hungarian press publisher says Google’s Gemini chatbot used its articles. The opinion draws the same line as Bing’s new controls and the training-only robots.txt signal: training is one question, answers are another.

On training, he suggests that using protected content to train a model is, as a rule, allowed under the EU’s text and data mining exception, if its conditions are met, including that the rights holder hasn’t opted out. Answers are different. Per the Court’s press release, a chatbot that fetches a news article in real time to answer may infringe the publisher’s rights if it got around a paywall or anti-scraping measure, or if the answer reuses how the article is written so recognisably that the reader no longer needs the original. Plain facts aren’t protected.

The opinion is not binding. The Court’s judges are now deliberating and will rule later. The Kluwer Copyright Blog notes he also suggested the conditions for liability don’t appear to be met in the example before the Hungarian court.

For a business, the takeaway is the same: what counts is showing up in answers. If the Court follows this line, AI companies might sign more licensing deals with news publishers, or quote news less in EU answers. Those are possibilities, not predictions.

Our take: what a local business should do

1. If you want to be recommended by AI, leave grounding and search crawlers alone. Microsoft says turning off grounding means “reduced brand visibility in AI experiences.” Keep Googlebot, Bingbot, OAI-SearchBot and Claude-SearchBot allowed, and leave Bing’s grounding preference on its default. We compare these bots by name in GPTBot vs OAI-SearchBot vs Google-Extended: which AI crawlers to allow.

2. Opting out of training is a fair choice, but make it on purpose. If you don’t want your content used for training, the shared signal will eventually cover several companies with one group in robots.txt. With Google, though, the price may include Gemini Apps grounding, because Google-Extended covers both. If Gemini visibility matters to you, choose your Google-Extended setting explicitly instead of leaving it to the shared token.

3. Don’t use a blanket “block AI” setting. One-click bot blocking in a CDN or plugin can sweep up search crawlers along with training crawlers. If your site runs on Cloudflare, check which AI crawlers actually visit and which ones to allow before you change anything.

4. Search your site for NoArchive and NoCache. An old template or plugin may have added them without anyone noticing. If you want to keep them as restrictions, switch to the new preferences before March 1, 2027. If you never meant to restrict AI use, you don’t have to do anything: after that date, Bing stops reading them as training or grounding preferences.

5. Watch the result, not just the setting. Publishing a preference doesn’t apply it right away. Microsoft says it only takes effect when Bing recrawls the page. Bing’s AI Performance report shows when Copilot cites your site (here’s how to read it). To see whether ChatGPT, Perplexity and Google AI actually name your business, the free Beket AI audit runs 25 prompts across 3 engines. Beket AI Pro ($49/month) re-audits daily, starting at 50 prompts across 3 engines, so if a robots.txt change costs you mentions, you’ll see it in the next day’s results.

What’s still unclear

• Exact launch dates. Microsoft gave firm dates. Google said “2027,” OpenAI said “the coming months,” and we found no Anthropic date.
• Whether other companies treat the token as a crawl signal. The draft allows a company that crawls only for training to skip pages that opt out, but it doesn’t require it.
• The final standard. The draft could change before it becomes an IETF standard, and people in the working group say disagreements remain.

None of this changes the basic advice. Training and answers are separate decisions, so set each one deliberately.