Cloudflare gives AI crawlers a September deadline: pay publishers or get blocked
Cloudflare has set the AI industry a deadline. From September, it will block the crawlers that hoover up content for AI training. Any page that carries ads becomes off-limits, unless the site’s owner says otherwise. The pitch is simple: stop giving the web away for free.
The company sits in front of a large share of the world’s web traffic. It announced the change on Wednesday. From 15 September, new Cloudflare sites will keep letting search engines index their pages. They will block AI training and AI agents from any page with advertising by default.
The rule also catches “mixed-use” crawlers, the bots that blend search, training, and agent tasks into one. If a crawler will not let a site owner separate those uses, it gets blocked on ad-supported pages.
The defaults apply to new customers and to new sites from existing customers. They also cover every free user who...
Copyright of this story solely belongs to thenextweb.com. To see the full text click HERE