Cloudflare's New Marketplace Will Let Websites Charge AI Bots For Scraping (techcrunch.com) 7
An anonymous reader quotes a report from TechCrunch: Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their site's content. The marketplace is the final step of Cloudflare CEO Matthew Prince's larger plan to give publishers greater control over how and when AI bots scrape their websites. "If you don't compensate creators one way or another, then they stop creating, and that's the bit which has to get solved," said Prince in an interview with TechCrunch.
As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial. A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. [...]
As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial. A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. [...]
As usual (Score:2)
So, the websites get cash, and the people who actually wrote the stuff get nothing.
As usual
Re: (Score:2)
Probably. But let's take two examples of sites that may use this service.
New York Times may use something like this. The content on New York Times belongs to them, whether it be through paid employees creating content, or the EULA you sign before posting comments on articles. So nobody is likely owed any dues by NYT should NYT decide to monetize on the AI crawlers' access.
Another example is something like WP Engine. Their content really doesn't belong to them. Should they try to monetize the AI use of
Re: (Score:2)
give them the right to sell the content.
To be clear, they aren't "selling the content".
They are allowing access to the content.
They already allow access to human readers and crawlers building indexes for search engines.
This is about allowing access to a crawler that is crawling for a different purpose than the currently allowed crawlers.
Re: (Score:2)
Agreed. I think we're on the same page. I should have said "selling the right to view the content", because that's what they would be doing. The sentence was referring to WP Engine having something in their EULA for lower tiers that would allow them to monetize the content. I'm too lazy to set up a WP Engine account and jump through all the hoops to see the EULA for their lowest account ($20/month), so this may completely be nothing. But I can conceivably see them squeezing something like that in the c
Re: (Score:2)