OpenAI, Google, Microsoft and Anthropic have said they will adopt a common standard that lets websites opt out of AI training collection in one step. A site operator can add a single shared signal to its robots.txt file and tell the AI crawlers of all four companies that it does not want its content used for training. Until now, each AI company had to be addressed separately, and the new signal is meant to cut that work.
| Item | Details |
|---|---|
| What | A shared ‘policy token’ signal placed in robots.txt |
| Companies | OpenAI, Google, Microsoft and Anthropic (four in total) |
| Proposed by | Kevin Kelly, Anthropic’s technical lead for standards, through the Internet Engineering Task Force (IETF) |
| Timing | Anthropic has already applied it; Google and Microsoft plan support next year; OpenAI says within the next few months |
| Extra Microsoft option | Publishers can refuse AI training and AI grounding separately; Bing search visibility stays the same either way |
What happened
The four companies said they will introduce an AI scraping opt-out standard for web publishers. The news was first reported by Search Engine Roundtable, a trade outlet that covers the search industry. At its center is the ‘policy token’, a shared signal. If a website puts the token in its robots.txt file, the AI crawlers of the four companies will restrict access to that site once they detect it.
The policy token was proposed through the IETF, the body that develops internet standards, by Kevin Kelly, who handles standards work at Anthropic. Robots.txt is a long-established file that tells crawlers which parts of a site they may collect, whatever the purpose: search indexing, archiving or AI training. The new signal sits inside that existing framework rather than creating a separate mechanism.
Microsoft also plans a finer-grained setting than the policy token alone. Publishers will be able to refuse AI training and AI grounding independently. Grounding is the process of having an AI model refer to verified sources of information to reduce hallucinations. Microsoft said that whichever option a publisher picks, its visibility in Bing search results is not affected.
Why it matters
Until now, a site operator who wanted to opt out of AI training had to state that preference to each AI company separately. That meant looking up each crawler’s name and writing an entry for each one in robots.txt. With a policy token, one signal can convey the same preference to all four companies at once.
The weight of the participants matters too. OpenAI, Google, Microsoft and Anthropic run major AI services: ChatGPT, Gemini, Copilot and Claude. The four have said that when they find the policy token, they will restrict their AI crawlers’ access. For a publisher, that means one setting can apply to several of the biggest AI companies together.
Background
The announcement comes as publishers struggle with the volume of AI crawler traffic. AI crawlers have been collecting website content to feed AI models, and publishers have been looking for ways to control it.
There have been earlier attempts to push back on unauthorized collection. The RSL (Really Simple Licensing) standard lets publishers charge crawlers that visit their sites. Cloudflare, the web infrastructure company, chose to block AI crawlers by default. The policy token differs from both approaches in one respect: the AI companies themselves have committed to supporting it.
What to watch next
Timing differs by company. Anthropic has already applied the policy token. Google and Microsoft plan to support it next year. OpenAI said it will implement it ‘within the next few months’. That means the policy token alone does not yet cover all four companies, and until each one finishes adding support, it is worth keeping any existing per-company settings in place and checking them.
If you run a blog or website, these are worth checking:
- Which AI crawler rules your robots.txt currently contains
- When Google, Microsoft and OpenAI actually begin supporting the policy token
- How Microsoft’s separate settings for AI training and AI grounding will work in practice
Because the policy token is still at the proposal stage at the IETF, it is also worth watching what form the standard document takes. Whether each company delivers support on the schedule it has given will be a key measure of how effective the policy token turns out to be.
