The biggest names in AI are backing a new way to opt out of scraping

OpenAI, Google, Microsoft, and Anthropic have agreed to support a new shared "policy token" signal that publishers can use to opt out of AI scraping. Proposed by Kevin Kelley, a technical lead for standards at Anthropic, within the Internet Engineering Task Force (IETF), the token will live in a website's robots.txt file—the long-standing mechanism for signaling which web crawlers may access a site. Currently, publishers must set AI training preferences separately with each AI company; this shared token lets them apply their preferences across multiple companies at once. The four companies say they will restrict their AI crawlers from accessing a site when they encounter the token. Microsoft will additionally offer granular controls allowing publishers to opt out of AI training or AI grounding separately, and notes that sites will still appear in Bing search results regardless of these preferences. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the "coming months." The move comes as publishers struggle with a surge of AI crawlers, with some turning to alternatives like the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers, and Cloudflare's default blocking of AI crawlers.

(Source：The Verge)