Cloudflare Has a New Plan To Fight Bots -- and Climate Change (techcrunch.com) 28
Cloudflare is ratcheting up its fight against bots with a new "fight mode," which it says will frustrate and disincentivize bot operators from their malicious activity. From a report: Bots are notorious for scraping websites and abusing developer access to download gobs of user data. All too often bots try to game the system by scraping concert or airline ticket prices to buy in bulk at their lowest price and sell them off for higher. Worse, some imitate real users and brute-force their way into websites with lists of stolen passwords. Cloudflare gets three billion bot requests each day. Now the company said it's "decided to fight back."
Its new "bot fight mode," which Cloudflare today enabled as a free opt-in feature for all accounts, will detect and serve bots with deliberately computationally intensive challenges. As the bot tries to crunch the impossible puzzle -- effectively a small bit of code only visible to the bot -- the bot's server will max out its processing power, churning up cloud resources and driving up costs for the bot operator. While the company says its efforts will dissuade bot activities in the long run, it recognizes its efforts in the short term will result in cloud servers working overtime, thus consuming more electricity and requiring more cooling -- all of which contribute to greater energy consumption.
Its new "bot fight mode," which Cloudflare today enabled as a free opt-in feature for all accounts, will detect and serve bots with deliberately computationally intensive challenges. As the bot tries to crunch the impossible puzzle -- effectively a small bit of code only visible to the bot -- the bot's server will max out its processing power, churning up cloud resources and driving up costs for the bot operator. While the company says its efforts will dissuade bot activities in the long run, it recognizes its efforts in the short term will result in cloud servers working overtime, thus consuming more electricity and requiring more cooling -- all of which contribute to greater energy consumption.
Robots And Climate Change Are Attacking Cloudflare (Score:2)
Bots (Score:4, Interesting)
I run a pretty popular blog and I get hit from bots constantly scraping my site, or trying to fill out forms with bogus information. 90% of this traffic comes from India. Which finally led me to just block any India IP address from filling out forms.
Sucks, but that was the only thing that finally put a stop to it.
India Spam (Score:5, Interesting)
Scrapers aren't malicious (Score:4, Insightful)
Scrapers aren't malicious, unless you run a site designed to fleece suckers. They provide the information that the people want, like price comparisons. And if you have the lowest price, they will result in traffic and sales. And if you know how to use caching, they shouldn't do any special harm to your site.
A better solution for people not trying to take advantage of ignorant users might be to make sure that scrapers get the information they want without having to load your whole page, which means actually outputting HTML, and not shoveling all the content into the page after the fact with JavaScript like a total noob.
Re: (Score:2)
Since they talk of putting code into the page, I assume they are going after the kinds of bots that like to fill out their forms and automatically interact as if they are a person doing things (meaning they need to run javascript).
I run a scraper for images (for reverse searching image sources), mostly snorting up opengraph data. This will not affect my scraper in the slightest.
It's rampant (Score:3, Interesting)
Nice. How are Slashdot's efforts going to fight troll farms gaming their once-sensible mod system, built on the notion only well-meaning moderations will occur?
Re: (Score:1)
The beauty of Slashdot's moderation system is that it prevent circlejerks from downmodding an unpopular opinion into oblivion. Fuck off back to Ars Technica if you want that.
It Happens In The Firehose (Score:5, Interesting)
A win-win scenario (Score:3, Funny)
Well all know that cows cause global warming, right?
So to offset the carbon increase from the Cloudfare Battlin' Bot mode, the answer is right in front of us - everyone should switch to eating burgers made from bot operators!
Re: (Score:3)
Reselling plane tickets? (Score:2)
Fighting Climate Change? (Score:2)
Sounds more like they're encouraging it.
Forcing botnets to use more electricity isn't actually "fighting" climate change, but I guess as long as you call it "fighting climate change" then you've accomplished your goal of reducing AGW....
Re: (Score:2)
Sounds more like they're encouraging it.
Forcing botnets to use more electricity isn't actually "fighting" climate change, but I guess as long as you call it "fighting climate change" then you've accomplished your goal of reducing AGW....
I know reading the article is hard, but you should totally due. Because they like totally covered this dude.
Like their CAPTCHA block? (Score:5, Insightful)
Cloudflare already has a mode where it will chain an infinite number of recaptcha challenges together if it decides it doesn't like your browser agent. When it pops up on sites, I just change browsers and I'm instantly not a bad person.
Re:Like their CAPTCHA block? (Score:5, Interesting)
The same often happens to me when I disable scripts. The future of captcha seems to be that you either let them run their spyware on your machine or they just decide that you're a bot.
Re: Like their CAPTCHA block? (Score:2)
Old idea (Score:5, Interesting)
robots.txt trap (Score:5, Interesting)
I used to put a "Disallow" line in my robots.txt file that lead to an endless series of programmatically-generated pages filled with random crap and lots of juicy (but fake) email addresses.
The bad bots that read the robots.txt file would follow the link and get mired in shit for hours (and sometimes for days!).
After a while I modified the code and started feeding the bots their own email addresses, so they probably sent themselves quite a bit of spam.
It was hilarious to watch the bots painfully crawl page after page after page in a fruitless quest that never ended...
Re: (Score:2)
That is a pretty good idea. Any idea if it still works, or did the bots get smarter?
Re:robots.txt trap (Score:5, Interesting)
Oh yeah, they still dutifully follow the Disallow links and thrash around in the La Brea Link Pit for a few hours.
I used to keep an eye on it just to see what was going on, but if you've seen one bot trip over its own dick, you've seen 'em all.
Lololol (Score:2)
It'd be cool if you took your manual spam blacklist and fed it out. As soon as someone gets into your inbox they're done. For good measure you could post it to usenet too. Alt.sex.stories; email goes to sender, subject, and body.
Reminds me of the Borg... (Score:2)
... and the plan Starfleet had to intriduce a very similar concept to destroy them. I wonder if Starfleet had followed through, if the Borg's carbon footprint would have spiked before they went kaput. Oh well, suppose we'll never know.
Re: Reminds me of the Borg... (Score:1)
HCF, or better (Score:5, Interesting)
It would be awesome if there were a way to serve the mythical Halt-and-Catch-Fire (HCF) instruction to the bots. Or, equivalently, just a HALT instruction. Or a WAIT instruction that would never get its interrupt. My inner 5-year-old imp is nearly as titillated with their solution of serving computationally difficult programs (presumably obfuscated in random ways so that the bots aren't able to recognize and drop them).
The adult in me thinks serving them something actually useful, such as problems from the BOINC network like SETI@home, would be an even better solution.
Re: (Score:2)
Well I mean they can shrink the TCP window down very small. You could then even protect your services by keeping the TCP window unusually small for just a little bit. They could also tell the remote system they wanna use gzip compression to send compression bomb with a gif bomb or something that's been run through a fuzzing framework, there's all sorts of opportunity to give to jerk them around all different ways at once.
If they wanted a long game they could start fuzzing different graphics libraries unt