Cloudflare to block cynical search-and-scrape bots from ad-supported web pages
Some crawlers gather data for both search and AI training, so when publishers block them to protect content they risk disappearning from search results ...
Cloudflare on Wednesday said it will soon prevent mixed-use crawlers from accessing ad-supported customer websites by default, part of its ongoing efforts to give site publishers more control over how they engage with AI services.
Apple, Google, and Microsoft's Bing operate crawlers that could fall afoul of Cloudflare's decision, although each of the tech giants offers an AI opt-out that may allow them to escape sanctions.
Web crawlers make automated network requests to websites for various purposes. Google has used them for decades to visit websites for inclusion in its search index.
Over the past few years, many crawlers have started visiting sites to harvest content for training AI models. This has prompted various countermeasures – publishers feel they're not being fairly compensated for...
Copyright of this story solely belongs to theregister.com. To see the full text click HERE
