Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare's move is aimed at forcing AI companies to separate their web crawlers, which are used for traditional search purposes and AI training, and pay publishers for their content. This change will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. The company claims that most website owners want their content to be discoverable via search and AI services but also want protections against having their intellectual property given away for free. Cloudflare is specifically targeting the "world's largest search engine," Google, which it claims has access to about 2x more information than other AI companies due to its dominance in search and AI features. Google has pushed back against this generalization, noting that it provides a bot called Google Extended that lets site owners opt out of having their content used for training and AI products.

The broader context is that the internet has recently reached a milestone where bots have surpassed human traffic online for the first time, with over 50% of crawl traffic from AI crawlers spent re-fetching unchanged pages. Cloudflare is capitalizing on this trend by releasing a range of tools to give publishers more control over their content in the AI era. The company's new policy is part of its efforts to establish a sustainable ecosystem where AI companies pay for the content they use, rather than relying on free crawling. Cloudflare is partnering with two companies, Ceramic.ai and You.com, to implement its "Pay Per Use" model, which will allow publishers to charge AI companies when their content creates value.

The implications of Cloudflare's policy are significant, as it could disrupt the way AI companies access web content for training purposes. AI model providers may need to adapt their strategies to comply with Cloudflare's new rules, which could lead to increased costs and changes in the way they operate. Additionally, the "Pay Per Use" model may not be widely adopted, which could limit its effectiveness. This development is worth watching as it highlights the ongoing struggle between AI companies and content creators over the value of web content.