Biggest AI names back shared signal to help publishers block scraping

A shared site setting

The proposed policy token comes from Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force. It would be placed in a website’s robots.txt file, which already communicates rules to web crawlers.

Today, publishers must communicate AI training preferences separately to each company. The shared signal is intended to let them state those preferences across multiple AI providers at once.

What companies plan

OpenAI, Google, Microsoft and Anthropic say their AI crawlers will avoid sites that use the token. Anthropic has already adopted it; Google and Microsoft plan to support it next year, while OpenAI says implementation is expected in the coming months.

Microsoft says publishers will have detailed choices, including opting out of AI training or AI grounding, which directs a model to verified information to reduce hallucinations. Microsoft says those choices will not remove websites from Bing search results. Google has similar controls for AI search features and training that do not affect search visibility.

Publishers seek more control

The proposal arrives as publishers face continued scraping by AI crawlers, which collect material for AI models. Other efforts include the Really Simple Licensing standard, which allows publishers to charge crawlers, and Cloudflare’s default blocking of AI crawlers.