Anthropic, Google, Microsoft and OpenAI Back an AI Training Opt-Out. Their Crawlers Rarely Check Robots.txt.

Anthropic, Google, Microsoft and OpenAI have backed a one-line robots.txt opt-out from AI training that an Anthropic standards lead drafted; only Microsoft has named a start day.

The line would replace a separate instruction for each company’s crawler, but it works only if crawlers read robots.txt, and in one measurement 0.15% of their requests were for that file. Support also arrives company by company, on different timetables.

What does the token do?

A publisher adds a short entry to its robots.txt file, the text file that tells crawlers which pages they may fetch. The entry names __usage_ai-training as the crawler and marks the whole site, or chosen paths, Disallow or Allow.

Kevin Kelley, a standards lead at Anthropic, wrote the proposal and submitted it to the Internet Engineering Task Force (IETF), which sets internet standards. It is a draft from one author, not a working-group document, and the IETF has not endorsed it or given it standing.

When does it start?

Microsoft has the first fixed date: Feb. 1, 2027, when it will begin honoring the preference as its crawlers visit pages. The preference replaces Microsoft’s NoArchive and NoCache tags, which covered training and grounding, the use of web content to inform AI answers. Publishers relying on those tags must set the new preferences by March 1, 2027, Microsoft said.

Google said it plans to support the token in 2027. Where a site has set no Google-Extended rule, its existing control over AI use of a site’s content, Google plans to read the token as a disallow of Google-Extended. OpenAI said it will honor the token in the coming months. Kelley wrote that Anthropic has started implementing support.

Do the crawlers read the file?

Crawlers run by the four companies made about 1 billion requests across more than 50 publisher properties in 30 days, according to a published report citing data from the measurement firm ScalePost. Only 0.15% of those requests were for robots.txt.

Google’s documentation says its common crawlers always respect robots.txt rules, which sets a company’s claim against a count. The proposal offers no way to enforce the token, the report said.

What does opting out cost?

Microsoft’s guide says publishers who disallow grounding will see reduced brand visibility in AI experiences. The token covers training only; Microsoft adds a separate grounding preference, set page by page, so a site can block training and still be cited.

Alex Springer, technical lead at SPUR, a publisher coalition, argued the draft treats content as open to training unless a publisher says no, a default he said should not become normal, the report said.

The next dated checkpoint is Microsoft’s Feb. 1, 2027 start; Google’s crawler documentation points to early 2027 for its own support.