Cloudflare will filter out web crawlers that serve AI companies

Cloudflare is shifting its default policies to automatically block crawlers that combine search indexing with AI training or agent use, particularly on pages with ads, starting September 15, 2026. The move affects new customers and existing users on free accounts unless they opt out, enforcing a split between search-focused crawlers and those used for AI training. This follows a 2025 "Pay Per Crawl" feature, now rebranded as "Pay Per Use," which shifts compensation from crawling activity to payments when content appears in AI chatbot responses. Cloudflare’s CEO Matthew Prince framed the change as a response to non-human traffic dominating the web, aiming to create a "sustainable ecosystem" by incentivizing transparency and separation in crawler functions.

The policy directly targets Google’s Googlebot, which indexes content for search while also training models like Gemini and powering AI Overviews. Google allows publishers to opt into a search-only crawler (Google-Extended) but lacks an option to exclude content from AI training if they want to appear in AI Mode. Cloudflare’s stance pressures mixed-use crawlers—like Googlebot—to disentangle functions, aligning with growing publisher demands for control over content usage. By partnering with smaller AI firms (Ceramic.AI, You.com) and introducing financial incentives, Cloudflare positions itself as a mediator in the escalating conflict between content creators and AI companies over data rights.

Cloudflare’s approach risks fragmentation in web crawling standards, as AI firms may develop separate crawlers or challenge the policy legally. Google could face regulatory scrutiny if forced to split Googlebot, while publishers using Pay Per Use might prioritize monetization over open access. The success of this model depends on adoption by major AI players and whether it reduces disputes over content licensing. Additionally, the 202