Perplexity Says Cloudflare's Accusations of 'Stealth' AI Scraping Are Based On Embarrassing Errors (zdnet.com) 96
In a report published Monday, Cloudflare accused Perplexity of deploying undeclared web crawlers that masquerade as regular Chrome browsers to access content from websites that have explicitly blocked its official bots. Since then, Perplexity has publicly and loudly announced that Cloudflare's claims are baseless and technically flawed. "This controversy reveals that Cloudflare's systems are fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats," says Perplexity in a blog post. "If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic."
Perplexity continues: "Technical errors in Cloudflare's analysis aren't just embarrassing -- they're disqualifying. When you misattribute millions of requests, publish completely inaccurate technical diagrams, and demonstrate a fundamental misunderstanding of how modern AI assistants work, you've forfeited any claim to expertise in this space."
Perplexity continues: "Technical errors in Cloudflare's analysis aren't just embarrassing -- they're disqualifying. When you misattribute millions of requests, publish completely inaccurate technical diagrams, and demonstrate a fundamental misunderstanding of how modern AI assistants work, you've forfeited any claim to expertise in this space."
Denial probably AI-written. (Score:5, Funny)
Re:Denial probably AI-written. (Score:4, Insightful)
"Technical errors in Cloudflare's analysis aren't just embarrassing -- they're disqualifying."
That is 100% the sentence structure of an AI dress-down.
Re:Denial probably AI-written. (Score:5, Funny)
"Make it sound stern, technical, and authoritative."
Re: (Score:2)
Oh, I don't know. That sounds like something every lawyer under the sun* would say.
* And all those under the stones...
Re: (Score:2)
Do you suppose any humans actually work at Perplexity?
Re: (Score:2)
About half do.
Re: (Score:3)
I mean, someone probably said "It would be weird if we DIDN'T have our AI write this, right? Right? Anyone? Bueller?"
Sure that's going to work. (Score:5, Insightful)
As fucked up as Cloudflare is I'd believe them over Perplexity any day of the week.
Re: (Score:3)
I agree. A Perplexity coder is busy altering their user agent header in these scenarios to randomly choose from a large collection of various valid user agent strings.
Re:Sure that's going to work. (Score:5, Insightful)
there is nothing legitimate about a scraper that visits a site, scrapes the contents and uses it at their end.
Re: (Score:2)
Came here to say this.
At this point, AI crawlers are promoting global plagiarism, not actual "thinking" or "machine learning."
What? (Score:5, Informative)
Cloudflare are saying Perplexity is disguising its crawler bots as Chrome users.
Perplexity counters saying their crawlers don't do that, their other AI tools do it.
Seems like Cloudflare correctly determined that Perplexity are using automated tools to access website that requested to not be accessed by automated tool, despite Perplexity trying to hide their behaviour.
Re:What? (Score:5, Interesting)
Cloudflare are saying Perplexity is disguising its crawler bots as Chrome users.
Perplexity counters saying their crawlers don't do that, their other AI tools do it.
Seems like Cloudflare correctly determined that Perplexity are using automated tools to access website that requested to not be accessed by automated tool, despite Perplexity trying to hide their behaviour.
This is an interesting technicality. Perplexity is basically saying that robots.txt prohibitions only apply to training, either in terms of building page ranks for search or accumulating data for AI model training. So, if the data is immediately used, then it doesn't count as training, so robots.txt doesn't apply. Is this true?
Of course, the curious question is how Perplexity knows what's behind the robots.txt wall to serve as immediate on-the-fly answers if it didn't already previously crawl past the wall and store something in its database/model to indicate that the hidden data was useful.
Re:What? (Score:4)
The issue is: has an automated crawl initiated by an AI occurred or not. Intent is irrelevant, as is arbitrary redefinition of terms by Perplexity.
Re: (Score:2)
They basically say that their agent is an extension (not as in addon, but like in reaching farther) to the user's browser, which also neither identifies as robot nor accesses robots.txt.
Re: (Score:2)
Re: (Score:2)
You really don't know the difference between a crawler and a web search?
An AI agent is tasked by a user to collect info about a topic - I do this all the time. The agent then does a bunch of successive searches, each time distilling the important things relative to the user's task that may need subsequent followup, and then ultimately sums up everything they found on the topic. What doesn't happen is "Perplexity building up a database of said content". Crawlers try to visit the whole internet (minus site
Re: (Score:3)
It’s still a bot accessing content requested not to be accessed by bots. It’s alsonot identifyingitself as a bot.
Re: (Score:3)
Think really hard about the logic behind that before you respond. If you follow that thread it to its conclusion - then there is no
Re: (Score:2)
The point is, that users are free to choose their software. If you deny specific softwares to access content, websites will not only deny AI agents, but also browsers with adblockers. Or maybe just everything that is not Chrome. If I want to read your website with lynx so I don't have to see your ugly background color it's my right. If I let some AI website fetch and summarize it, it is the same. Yeah, you wanted me to read the whole thing, but I said TL;DR and created a summary out of it, just as I may cho
Re: (Score:2)
"The agent then does a bunch of successive searches", so an automated system to follow links and download all the content it finds.
Got it.
Re: (Score:2)
It is not unusual to use a browser user agent when accessing websites with a virtual browser.
Do you run a webserver? Ever noticed how an iPhone visits sites that are little known a few minutes after the Google bot accessed them?
It isn't even unlikely that it may be an iPhone, testing the load times and how good the page renders on mobile to decide on the ranking (especially in mobile searches). Still it doesn't identify as Google bot, probably because Google also wants to catch you if you serve other conten
Re: (Score:2)
Do you run a webserver?
Yes
Ever noticed how an iPhone visits sites that are little known a few minutes after the Google bot accessed them?
No
Still it doesn't identify as Google bot, probably because Google also wants to catch you if you serve other content to the Google bot than to actual users.
Websites do this all the time. They allow googlebot through their paywalls so they get indexed.
Re: (Score:2)
I could confirm that behavior on several domains. As said, I think they are testing for mobile rendering and to access content that is fetched via javascript and not accessible by a pure scraper that doesn't run a browser engine.
Soo, who to trust? (Score:5, Insightful)
The content delivery network people with 16 years of experience in this game or the AI liars that peddle a basically fraudulent product based on a massive piracy campaign?
Hmm. Difficult!
Re:Soo, who to trust? (Score:5, Insightful)
But Perplexity is basically admitting it:
"If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic."
It really doesn't matter if Perplexity thinks they are "a helpful digital assistant". That's not what the robots.txt file says. There's no flag in there to allow only the "helpful" ones to scrape. Just don't scrape, m'kay?
Re: (Score:2)
Define "scrape".
The general definition of "scraping" involves actually, you know, storing the content, not just briefly caching it then summing it up to the user who asked to browse it.
Re: Soo, who to trust? (Score:2)
Define "scrape".
Any automated collection of webpage content. It doesn't matter if the agent doing the scraping stores it verbatim, summarises it or throws it in the bit bucket. From the point of view of the web server, all of these are equivalent.
Re: (Score:2)
Instead of using a browser, they used an LLM to generate a fucking curl call.
It doesn't matter if the agent doing the scraping stores it verbatim, summarises it or throws it in the bit bucket. From the point of view of the web server, all of these are equivalent.
Christ, you ignorant fuck.
If you type a URL into your start bar or whatever stupid web integrated doohickeys you've got on your OS, and it shows you a preview of teh web page, do you expect it to have respected the robots.txt file before pulling that data?
User-initiated requests are not "bots". There are definitions for this shit. Your ignorance mixed in a bowl with your fee
Re: (Score:2)
It's not automated dipshit, it's requested by a user.
When I access a web site, I send my request directly to that website. Not through an LLM. Semantics about proxy servers aside, If you fiddle with the content while in transit, you are operating an agent which is doing the scraping.
And you need to medicate right now.
Re: (Score:2)
When I access a web site, I send my request directly to that website.
No, you don't. You use a user agent. Though I love the mental image of you trying to speak into your mouse.
Not through an LLM.
The LLM in this instance is a user agent.
Semantics about proxy servers aside, If you fiddle with the content while in transit, you are operating an agent which is doing the scraping.
Fascinating. So every browser in existence today that does AI summarization of a site for you is guilty of being a scraper, and should consult the site's robots.txt before doing so.
Chrome will summarize a page for you with Gemini. Further, boyyyy, let's talk about adblocking.
I think we can safely summarize 85% of user-initiated web traffic as scraping, then.
Re: (Score:2)
So every browser in existence today that does AI summarization of a site for you is guilty of being a scraper
Yes. Particularly if they have to cloak a curl call [slashdot.org] as a Chrome browser.
let's talk about adblocking. I think we can safely summarize 85% of user-initiated web traffic as scraping, then.
I doubt it's as high as 85% at this time. Usually only the tech savvy can do effective ad blocking. But as AI sites such as Perplexity are increasingly being used by the average user this number will rise. And that will break the web's business model as we know it. Be prepared to pay for everything you access. And if Perplexity does it on your behalf, be prepared for a really big bill from them.
Re: (Score:2)
Yes. Particularly if they have to cloak a curl call [slashdot.org] as a Chrome browser.
cloak?
Give me a break.
Every UA in existence lets you change how they identify themselves. There's a reason for this, and it isn't nefarious.
The web is open, and if you have a right to block based on a self-reported string, then I have a right to change that string to stop you from doing it.
Anyone who uses strange third-party browsers is familiar with "cloaking" their browser as Chrome, since if you don't do it- many poorly designed websites won't work right.
I doubt it's as high as 85% at this time.
We're talking about anything that affects or
Re: (Score:2)
But TFS says:
curl doesn't masquerade as Chrome. Smells like fraud to me.
curl "masquerades" as whatever you want, and perplexity doesn't scrape or crawl. They're not in that business.
That's the actual fucking point of their reply.
Re: (Score:2)
Re: (Score:2)
If I type, "curl https://slashdot.org/ [slashdot.org]", should it have asked robots.txt before pulling the data?
If I type "https://slashdot.org" into my start bar, should it have asked robots.txt before pulling the data?
If I type it into an LLM chat window, should it have asked robots.txt before pulling the data?
The answer to all 3 questions is no.
If you would like to look less stupid among experts in the future, you can read here. [ietf.org]
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
No, it does not.
I side with Perplexity on the access being no crawling, not being used to train AI and not being a request that needs to obey robots.txt.
But it *IS* scraping by the very definition, i.e., fetching content automatically to parse it and extract information. And most scrapers do not respect robots.txt, as it is made for crawlers, which are autonomously following links, what scrapers don't.
Of course sometimes crawlers and scrapers are combined, e.g., in the bots that search for content for AI tr
Re: (Score:2)
The LLM is operating as a user agent, as in the RFC 3986 form of teh term.
It is not acting as a scraper/indexer.
You have an opinion, and it's adorable- really, but it's wrong.
Cloudflare is wrong.
If you want to deny certain types of user agents that are not bots (as defined in RFC 9309) from accessing your server, that's a different discussion entirely.
Sounds like the accusations are true. (Score:5, Insightful)
Sounds to me like the accusations are true, and Perplexity is deflecting by saying they're harmless and even helpful.
On my own servers, I see a pattern of behavior of something hitting my robots.txt (which both has a blanket denial for all user agents, AND specific denails for all known bots), and then suddenly a variety of IP addresses start hammering my site. It's bad enough I'm either going to put my servers behind Cloudflare, or at least one of the gatekeeper challenge systems.
Perpexity's shitty response really does nothing for me but confirm the accusations.
Re:Sounds like the accusations are true. (Score:4, Interesting)
Re:Sounds like the accusations are true. (Score:4, Informative)
Re: (Score:3)
Cloudflare will block if you ask them to block. If your desire is to let AI do what they want on your site, Cloudflare won't get in the way.
Many to most websites are business supported by ad revenue in addition to ecommerce - Letting AI become ad block for your end users is not ideal. Letting AI go haywire and purchase random shit instead of what the agent was instructed to do is also not ideal, for that matter.
Each site deserves the right to say "yes", "no", or "on these terms..." to AI crawlers and agents
Re: Sounds like the accusations are true. (Score:1)
Cloudflare of obviously only blocking AI crawlers if you tell them to. Not sure why you would get pissed on them for doing what you told them to do...
Re: (Score:2)
In my case, I have this in robots.txt, on a forum I don't want to get absolutely destroyed by bots that don't rate-limit themselves:
User-agent: *
Disallow: /
As for agentic actions, it's still an AI performing the actions. I said "No" to robots accessing my site. That they then pivot and do something that ISN'T not accessing my site is inappropriate no matter the reason.
Re: (Score:3)
The concept behind BBS's, maybe that is a proper way to deal with AI scrapers. AI companies are clearly unwilling to acknowledge or respect any terms or "no scraping allowed" for any website, besides their own.
So, those that know how a BBS works, should be smart/capable enough to pay (money/capabilities/favors) for access to it. The rest of the world none the wiser, and through obscurity and a vigilant BBS admin AI companies will have literally nothing new to scrape anymore.
Given that many types of data bec
Re: (Score:2)
It depends. Does the Perplexity agent faithfully relay the personalized messages from sponsors that would have otherwise been presented adjacent to the information on the website?
Re: (Score:2)
Re: (Score:2)
Depending on the agent, the output may never be seen by a human. "Constantly monitor eBay for a good deal on a waffle iron, and trigger a notification when found", for example. No eyes will ever see the pages the agent loads. It's just consuming eBay's compute resources. A much better written prompt will also ignore promoted eBay listings, inline advertisements, and so on.
Even if it's an action taken on behalf of a person, it's very unlikely the ads on the page will be delivered to the user. Keep in mind, t
Re: (Score:2)
Some website publishers have threatened publishers of ad-blocking software pursuant to the anti-circumvention provisions of the Digital Millennium Copyright Act (DMCA), the Computer Fraud and Abuse Act (CFAA), or other similar legislation. See "A Copyright Claim Was Reportedly Used to Stop Ad Blocking, But It’s Complicated" by Rhett Jones [gizmodo.com] for details.
Re: (Score:2)
If a human asks an AI agent to do this for them, why would the AI agent access robots.txt? Did you check Robots.txt before posting on Slashdot?
Re: (Score:2)
By that logic, anything executed by a person (such as, say, deploying a scraper) can just ignore robots.txt - a person started the task, after all.
Robots should respect robots.txt.
Re: (Score:2)
Robots should respect robots.txt.
- Asimov's 4th law
Re: (Score:2)
My point exactly. I was demonstrating how the GP's logic is messed up. Either a robot respects robots.txt or it shouldn't bother reading it. In either case the pattern being displayed is illogical.
Re: (Score:2)
How many scrapers do you know that respect robots.txt? That's a standard for crawlers, not for scrapers.
An example: I had for some time a tool running, that scrapes webpages from RSS feeds, to amend the headline-only feed to a fulltext feed. That's a scraper, as it gets a fixed list of URLs to process, fetches data, transforms data and then serves it to the user. But as the list is given fixed and no new links are added, it doesn't (need to) access robots.txt. The list is to be fetched, as the user said it.
Re: (Score:2)
All scrapers, crawlers, and other 'bots' SHOULD respect robots.txt. The original intent was to block what was termed at the time (1994) as "crawlers", but that has evolved as the Internet has evolved.
Justifying crawling behavior by saying it's "just scraping, and then loading additional pages..." is... Well. Fucky logic to say the least. Following your logic here: If I access a single page, then extract all the links in it and add it to an RSS feed, I'm free to then access all those subsequent pages because
Re: (Score:2)
The difference is, that a crawler is discovering new content itself. A RSS bot fetches a list of links. The site provides the list, the scraper fetches the links, that's it. It will never fetch a link of another site. A crawler on the other hand can start at the RSS feed and end up on a site I never knew it existed.
Re: (Score:2)
And I'll add one more point, which I meant to make in my previous comment:
Agents, AI or otherwise, aren't just pulling down a single page to present to the user. They're performing logic on that page that was accessed, and probably accessing additional pages. AI agents can be given instructions to "collect all the content on slashdot.org". I did that, and here's what ChatGPT did: https://chatgpt.com/share/6894... [chatgpt.com]
That's behavior that should respect robots.txt, but it apparently uses Chrome, so... It clearly
Re: (Score:2)
I thought about it, and I think the conclusion is that robots.txt should apply to following links. Before you follow a link that was not in your list of starting links, you need to check the robots.txt, while you don't for the links you were instructed to fetch. I guess that's basically the same as you are saying, a "Collect all content on ..." that spawns access recursively is similar to a crawler that just puts all links into a queue.
And somewhere one needs to draw the line, because if agents can act auto
Re: (Score:2)
The Truth of the person asking the agent to do the thing has no bearing on the Truth of an automated agent scraping the site. Your question is an irrelevant distraction. Have a nice day. :)
Or, to sink to your level, yes, it is still an AI agent scraping data regardless of any attempts to cloud the issue.
Re: (Score:2)
An LLM instantiating a web request on your behalf is no different than a browser, a start bar, or a fucking script.
You are trying to redefine scraping to mean, "accessing my web server using a user agent that I don't like.".
Re: Sounds like the accusations are true. (Score:2)
Re: (Score:2)
I think that if the agent identified itself properly in the user agent string rather than trying to masquerade as a person, there would be less of an issue.
Perhaps. But what if you, as a user, ran across a site that blocked Chrome because they don't like what that browser stands for.
Would you change your UA, so that sites didn't prejudice your choice of user agent?
Now, what if a user asked Perplexity's LLMified search tool to use a different agent?
However, it is being deliberately deceptive.
Or is deliberately trying to make their UA work.
For example, what if you, as the maker of Brave, tried to make your UA look like a regular Chrome UA so that pages would behave correctly since your renderer was bas
Differences (Score:2)
Re: (Score:2)
This is starting to split hairs. There are lots of legitimate reasons for pages to be fetched by a "bot". If you post a link to a social media platform, that system will fetch the page to access the HTML meta tags to find things like the page title, an image to represent the page, a description, etc, and that is what is displayed in the post instead of just a plain URL. That request also doesn't result in "eyeballs" and ads are not served. Browsers can pre-fetch URLs on a page, again, not resulting in the
Re:Differences (Score:4, Informative)
Yes, undoubtedly there are a lot of legitimate reasons for automated requests like the ones you have listed.
But individual requests for content for AI assistants can be even more problematic than a traditional scrape. Getting scraped once a day isn't that much traffic. But if 2 million people ask an AI assistant want to know what foo actor starred in bar movie that generates a request to movie database, those 2 million eyeballs aren't seeing the ads for upcoming movies and the host has to serve all that extra traffic.
It's enough of an issue for web devs that there is now even an open source AI blocker [techaro.lol] that devs are including in their sites as F5 reports about half all requests are coming from AI bots [zdnet.com]
Re: (Score:2)
Re: (Score:2)
If Cloudflare is misrepresenting the data to make it appear as if data is being scraped for training purposes when it is not then that is indeed something different.
You can guarantee that what ever the trigger for Perplexity's bots collecting data, they'll use it for training too.
If they were navigating through a site like a person would, I doubt it would be triggering Cloudflare's bot logic.
Re: (Score:2)
Re: (Score:2)
You are entirely correct.
This is problematic because it has a statistical effect on the monetization of views for them.
It still is not, and never will be expected to adhere to robots.txt as traditionally understood.
This problem exists for ad blockers, text-only browsers, etc. It's only at a different scale given LLM adoption.
Hey wait... (Score:5, Insightful)
Cloudflare claimed that perplexity's AI was able to answer questions about content on pages that perplexity was prohibited from accessing.
How exactly does perplexity explain that phenomenon?
Re: (Score:3)
Re:Hey wait... (Score:4, Insightful)
Don't worry, the data is still going into their future models.
Sounds like they're using their users to moderate the content the bots scrape. "Use our agent to browse the web for you, and tell us how good it's performing"
Re: (Score:2)
Perplexity mostly uses others models, like the ChatGPT API.
Re: (Score:2)
Perplexing, isn't it? That's Clairvoyic AI, the latest advancement. It learns things without having to read them. Knows without training. It's a whole paradigm shift that you're too feeble to understand.
They're guilty (Score:5, Insightful)
Perplexity's accusatory and belligerent tone is as good as guilty plea in my book. They sound like someone who is trying to insult the other party into submission so that they won't have to come clean.
Re: (Score:2)
They sound like someone who is trying to insult the other party into submission so that they won't have to come clean.
Perhaps the statement was written for them by Karoline Leavitt?
Re: (Score:2)
Well, they already admit it themselves. "It's not a malicious scraper, it's our AI tool searching the web when web search is enabled". So, "we're doing it" but it's legitimate because... well, they had AI assist in the response so I guess they needed some help answering that one.
Cloudflare has it right (Score:1)
Any AI access is malicious (Score:2)
Robots.txt not about threats, Perplexity is wrong. (Score:4)
Any automated assistant is still automated regardless of the intention. It's the whole reason robots.txt exists.
Also, anyone expecting robots.txt to hide content from public view or public use is just an idiot.
Cloudflare is big and has plenty of resources to make automated travel through content covered by robots.txt to be more hassle than it's worth. Once again anything on a networked computer becomes a battle of resources. Security is losing the battle, many forums lost the battle long ago, now AI is flooding anything left with both unending traffic and unending gibberish content. We lost, we now stand in shit.
Flip it around (Score:2)
"If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic."
If you can't create a "helpful digital assistant" that anyone can easily distinguish from a malicious scraper, then you probably shouldn't be creating traffic on the internet.
Explanation (Score:2)
Perplexity has three types of access to websites.
The first is crawling. Most of that is not done by them, as they mostly use models trained by others.
The second is web search. When the AI decided what search terms may find the information for your question, it asks a search engine (e.g. using the bing API). The search engine crawled the site and a request may even trigger a recrawling.
The third is agentic access. It doesn't crawl information, but only accesses it as proxy for the user. This means when the s
Me thinks she doth protest too much (Score:2)
Proplexity is pretty snippetty and judgy in their rebuttal. I think they're caught and trying to misdirect.