

Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers (cloudflare.com) 18
Cloudflare today announced a "Pay Per Crawl" program that allows website owners to charge AI companies for accessing their content, a potential revenue stream for publishers whose work is increasingly being scraped to train AI models. The system uses HTTP response code 402 to enable content creators to set per-request prices across their sites. Publishers can choose to allow free access, require payment at a configured rate, or block crawlers entirely.
When an AI crawler requests paid content, it either presents payment intent via request headers for successful access or receives a "402 Payment Required" response with pricing information. Cloudflare acts as the merchant of record and handles the underlying technical infrastructure. The company aggregates billing events, charges crawlers, and distributes earnings to publishers.
Alongside Pay Per Crawl, Cloudflare has switched to blocking AI crawlers by default for its customers, becoming the first major internet infrastructure provider to require explicit permission for AI access. The company handles traffic for 20% of the web and more than one million customers have already activated its AI-blocking tools since their September 2024 launch, it wrote in a blog post.
When an AI crawler requests paid content, it either presents payment intent via request headers for successful access or receives a "402 Payment Required" response with pricing information. Cloudflare acts as the merchant of record and handles the underlying technical infrastructure. The company aggregates billing events, charges crawlers, and distributes earnings to publishers.
Alongside Pay Per Crawl, Cloudflare has switched to blocking AI crawlers by default for its customers, becoming the first major internet infrastructure provider to require explicit permission for AI access. The company handles traffic for 20% of the web and more than one million customers have already activated its AI-blocking tools since their September 2024 launch, it wrote in a blog post.
The great internet paywall begins (Score:5, Interesting)
Re: (Score:3)
I did see a lot more Cloudflare prompts lately on sites which had no issue with Pale Moon before.
I prefer my method (Score:2)
Re: I prefer my method (Score:2)
Re: (Score:2)
That's what CF was already doing.
So when are the lawsuits coming? (Score:2)
No doubt Meta[stasize], Google, OpenAI and all other major AI shops will whine about having to pay for anything and conjure up some reasoning why this system is illegal because reasons and sue Cloudflare to tie them up in litigation - so my question is: when is that happening?
Re: (Score:3)
Re: (Score:2)
You mean like people on here and elsewhere who brag about stealing music/movies/software because they don't want to pay?
If it's okay for you to steal someone else's work, why is not acceptable for these companies to scrape available content?
Re: (Score:3)
will whine about having to pay for anything You mean like people on here and elsewhere who brag about stealing music/movies/software because they don't want to pay?
"brag" != "whine"
If it's okay for you to steal someone else's work
WTF here said that in this discussion. Provide citations.
why is not acceptable for these companies to scrape available content?
Comparing apples to oranges.
Multi-billion$ AI companies scrape content, then repeatedly sell access to services that use that content at scale without compensation to the creators, without whose content those companies would have nothing to offer in the first place.
Quite different than some individual "stealing" a song for their own use (sure there's some level of deprivation of funding to the creator, but they're not making money of
Interesting but too late... (Score:2)
Had this existed three years ago, it might've been interesting.
That said, there's something missing. There's two ways a crawler can work. Either they request content, get told a price and have to reconnect and agree to the price OR they can declare in ad
Re: (Score:2)
That's an interesting and dangerous idea. Do you know that when you allow ad script, different advertisers "bid" for the space in your browser? They try to determine who you are and what you may buy (that's why there is so much tracking) and then there is a high-frequency bidding system who is willing to pay the most for the ad space.
Now imagine a system in which site and bot bid about the price for the access. "You want to be included on the user's result page? For 3 cent you go there, I also get 100 pages
but muh profits! (Score:2)
So AI only crawls spam content for AI from now on (Score:1)
Former it was like: AI might hurt itself when it gets more and more ai generated input
Now its like: AI will only get ai generated content at all
Re: (Score:2)
But not the spam behind cloudflare...
Re: (Score:3)
But in especially the spam behind Cloudflare.
If a site can charge per access without telling before what's the content, what will they do? They will create clickbait for bots. Generate a page that looks like the outgoing links are worth to pay for, then hope the bot pays for access.
Any bot owner who agrees to pay for accessing content makes himself a target for such spams/scams.