On 2 July 2026, Cloudflare announced a new policy that, starting 15 September 2026, will block bots that scrape ad-supported websites for both search indexing and AI training at the same time. The goal is to force AI companies to split their crawlers into separate bots with clear purposes, so publishers can choose which ones to allow.
The change sounds technical, but it matters. Right now, many AI companies use a single crawler to collect content for everything: training new models, answering user questions, building search results, and more. This makes it hard for publishers to say yes to search but no to AI training. Cloudflare's policy targets exactly that confusion.
For publishers, the new approach is welcome. Many news sites, blogs, and online communities have complained that AI companies take their content without permission or payment, then compete against them in search and traffic. By forcing a cleaner split between crawlers, Cloudflare gives websites a real choice: you can let in the search bot, block the training bot, or charge for access to either.
For AI companies, the policy is a headache but not a wall. They can still crawl, but they will have to be more transparent about what each bot does. Some may choose to negotiate licensing deals rather than fight through blocks. Others may try to evade the rules, though Cloudflare has a strong track record of catching that kind of behavior.
The wider signal is that the bargain between AI companies and publishers is being rewritten. Content owners want payment, attribution, and control. Infrastructure companies like Cloudflare are now helping them enforce those demands. Expect more tools like this, and expect the pricing of web content for AI to become a serious market.