Google Joins OpenAI And Anthropic In Promising New Scraping Controls
The new robots.txt directive will prevent the companies from scraping content for generative AI training.
Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley.
As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley's proposal will change that by getting multiple AI companies to recognize and honor a single common signal. Publishers have to set up a new user agent in the robots.txt file that's specifically for AI training, which will look like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That will...
Copyright of this story solely belongs to www.engadget.com. To see the full text click HERE
