Cloudflare's New Marketplace Will Let Websites Charge AI Bots For Scraping (techcrunch.com) 12
An anonymous reader quotes a report from TechCrunch: Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their site's content. The marketplace is the final step of Cloudflare CEO Matthew Prince's larger plan to give publishers greater control over how and when AI bots scrape their websites. "If you don't compensate creators one way or another, then they stop creating, and that's the bit which has to get solved," said Prince in an interview with TechCrunch.
As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial. A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. [...]
As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial. A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. [...]
As usual (Score:2)
So, the websites get cash, and the people who actually wrote the stuff get nothing.
As usual
Re: (Score:2)
Probably. But let's take two examples of sites that may use this service.
New York Times may use something like this. The content on New York Times belongs to them, whether it be through paid employees creating content, or the EULA you sign before posting comments on articles. So nobody is likely owed any dues by NYT should NYT decide to monetize on the AI crawlers' access.
Another example is something like WP Engine. Their content really doesn't belong to them. Should they try to monetize the AI use of
Re: (Score:2)
give them the right to sell the content.
To be clear, they aren't "selling the content".
They are allowing access to the content.
They already allow access to human readers and crawlers building indexes for search engines.
This is about allowing access to a crawler that is crawling for a different purpose than the currently allowed crawlers.
Re: (Score:2)
Agreed. I think we're on the same page. I should have said "selling the right to view the content", because that's what they would be doing. The sentence was referring to WP Engine having something in their EULA for lower tiers that would allow them to monetize the content. I'm too lazy to set up a WP Engine account and jump through all the hoops to see the EULA for their lowest account ($20/month), so this may completely be nothing. But I can conceivably see them squeezing something like that in the c
Re: (Score:2)
GenAI profiting from AI bots! (Score:2)
The genie's out of the bottle & we have to decide how we're going to proceed so that our languages & cultures aren't drowned in oceans of GenAI content on the interwebs pipes. We'll all be fine. The internet won't.
Could GenAI simply be the death of human trust in the internet, a la "dead internet theory"?
Re: (Score:2)
Yay! This won't have any negative consequences like websites publishing reams of GenAI "content" to charge AI bots for access to scrape, will it?
Let them. It'd be the quickest route from here to losing all faith in AI for the entirety of society outside of the people making stacks of cash, and they'll lose faith once the money stream dries up.
The genie's out of the bottle & we have to decide how we're going to proceed so that our languages & cultures aren't drowned in oceans of GenAI content on the interwebs pipes. We'll all be fine. The internet won't.
And at this point? Do we care. It's a valid question. The Internet was useful for a lot of reasons. Now, some of those reasons still exist, but it's mostly a tool for corporate messaging and political shit-stirring, while being data-sucked by the AI and Ad companies. Information truly did want to be tied up and
Good for spam sites (Score:2)
Good for the spam sites. Provide content (use ChatGPT for that, it's free!) and then charge for lettings someone scrape it, who thinks it may be valuable.
And Cloudflare is the one making some money with a marketplace without even providing content (real or fake) or running a scraper.
This is not how this should be done (Score:1)
While this is a good thing for Cloudflare customers, and ultimately Cloudflare, the rest of the internet suffers from not having an option without them.
Also what percent does Cloudflar