Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare will block mixed-use crawlers by default starting in 2026, forcing AI companies to distinguish between search and AI training traffic.Summary
Cloudflare is updating its default security settings to block crawlers that combine search functionality with AI training and agent services, effective September 15, 2026. This policy change targets crawlers that currently scrape data for both purposes, effectively requiring AI model providers to separate their traffic or face being blocked from sites hosting advertisements. CEO Matthew Prince stated that this move aims to create a more sustainable ecosystem by giving website owners better control over their content and monetization opportunities. Additionally, Cloudflare is expanding its “Pay Per Crawl” marketplace into a “Pay Per Use” model, allowing publishers to charge AI companies when their content generates value, with early partners including Ceramic.ai and You.com.
(Source：TechCrunch)