How the FSF Sysadmins are Blocking Botnets with reaction (fsf.org) 23
For nearly two years the Free Software Foundation has been fighting web crawlers (including many aggressively scraping training data for AI models). A botnet controlling about five million IPs hit one system for six months in 2025. Their systems administrator wrote this week that they view these as distributed denial-of-service attacks.
How are they fighting back? We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection. We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app... We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet.
We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules... We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules...
We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom... After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed...
Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online.
How are they fighting back? We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection. We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app... We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet.
We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules... We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules...
We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom... After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed...
Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online.
CGNAT (Score:3)
Much of the world is forced to use CGNAT to access legacy IPv4 sites, and many users are stuck with either no IPv6 or very lousy implementations (eg rapidly changing prefixes etc)...
You have one infected machine and it gets the shared gateway blocked, and then other users of the same provider are unable to access anything.
A lot of people are affected by this, but often don't know the reason. Site operators in developed countries often don't care or don't understand how users elsewhere have to suffer with CGNAT.
Example: https://www.reddit.com/r/Conve... [reddit.com]
Re: (Score:2)
Mapping legacy addresses to v6 addresses wouldn't achieve anything, since you'd still have multiple users with the same address.
Marketing companies don't track users by IP, that's done with cookies, browser fingerprints, sessions etc. Marketing companies don't care about CGNAT.
This tracking doesn't work with malware, because malware doesn't keep cookies and each instance appears as a new user. It is the constant influx of new users from shared addresses which trigger responses like blocking or captchas. If
Re: (Score:2)
You have one infected machine and it gets the shared gateway blocked
The issue here isn't the architecture of any IPv6 implementation. The issue is service providers (I'm looking at you, CloudFlare) using IP based blocking, when far better methods exist.
Re: (Score:3)
Far better methods such as what?
Several ISPs use IPv6 prefixes which are frequently rotated and recycled which can cause issues. A prefix which is static, or at least stable until the user explicitly releases it works much better.
The idea of dynamic addressing made sense in the dialup days when most of your users were disconnected most of the time. It doesn't make sense now.
Marketing companies like to perpetuate this myth that they track you by ip, but they don't and never have. It's just to distract users
Re: (Score:2)
Marketing companies like to perpetuate this myth that they track you by ip, but they don't and never have.
The MAFIAA certainly does though.
Re: (Score:3)
ISPs using CGNAT still log traffic in order to track users, they do significantly more detailed logging for exactly that reason.
Similarly CGNAT breaks p2p, so applications like bittorrent effectively turn into a client-server model making it much easier for such orgs to go after the servers.
Re: (Score:3)
I find IPv6 is worse than CGNAT for these anti-bot measures. YouTube is a good example. With IPv6 enabled all I ever see is "log in to prove you are not a bot". With it disabled and sitting behind CGNAT, I can usually watch some videos.
Re: (Score:2)
Hence my comment about poor implementation, your ISP is doing something wrong.
For most of the world, cgnat causes all kinds of problems with youtube and other sites.
Re: (Score:2)
I wouldn't say they are doing it wrong, I'd say that there is a fundamental conflict between privacy and anti-bot measures.
For privacy reasons I don't want a unique IP address. I want a shared one, and if it's IPv6 I want it to rotate frequently. That's one of the reasons why I use a VPN. ISPs probably also like it because it means that without extensive logging, for which there is no business justification, they can't identify who downloaded some movie that the MAFIAA et. al. want to sue over.
But of course
Re: (Score:2)
The idea that companies track you based on IP is a fallacy, they use cookies and similar technologies. Even if everyone had static addressing at home and companies like google could guarantee this was the case globally, people still travel so the same device can pop up from multiple different locations with wildly different source addresses. I have multiple devices at home, but on the same VLAN they originate from the same IPv6 /64. Sites like google consider them different users, and even guess completely
Re: (Score:2)
My browser shreds cookies as soon as I leave a site in most cases, as well as all other site date. These days the tracking works based on multiple signals, so even if you delete the cookies, if the IP address and browser signals like user agent and screen resolution match, they will re-associate that identity with you. You need to screw with a lot of metrics to throw them off.
In my country a spam lawsuit against 50 people where only one of them is possibly "guilty" of a civil offence with a relatively small
Re: (Score:2)
The primary reason botnets target random home devices is because of shared ip addresses.
Earlier botnets targeted servers (more bandwidth etc), but these are trivially blocked. Once you have shared addressing you can't blanket block it without upsetting users, so you end up with login requirements or operators being forced to accept some level of malicious traffic.
If you have static addressing, and your address space earns a reputation for non malicious activity then you have a much easier life. This still d
Good job (Score:2)
And it's nice to see groups like FSF share their best practices with the world. It appears that ClownFlare has fallen on its face again.
Re: (Score:2)
Sure but seriously, this is the FSF and they just found out about ipset which I have been using for decades?
From TFS:
and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules... We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules...
Also no, five million entries in ipset do not generate five million firewall rules. That's the whole concept. Only one rule in the firewall can check if the src ip, dst ip, ports etc. is in the ipset. It's basically just a lookup in a hashtable that requires only 1 firewall rule.
Follow the money (Score:4, Interesting)
Paying someone for data scraped by compromised smart TVs to train your AI sound if not downright illegal like several lawsuit waiting to happen. Not laying this at the FSF doorstep, but asking is there any movement on this from anywhere?
Re:Follow the money (Score:4)
Oh, no, it's perfectly "legal." You "agreed" to allow your TV to be used like this in the EULA when you installed the app. ( https://blog.includesecurity.c... [includesecurity.com] )
Re: (Score:2)
Thanks! That was a twist to the story I really didn't anticipate.
Re: (Score:2)
By installing this app you may find your IP blocked by more and more websites.
In 2026, you can still get slashdotted (Score:1)
Routing tables? (Score:2)
I have been playing with routing tables. I have a few virtual machines around the globe that have full BGP routing tables and that gives me great IP -> ASN details. That means someone comes into a server and that might get redirected to as1221.my2026.example.com but that is a virtual web server with a few hundred thousands others. Everything nice and fast until there is a problem. Once that hits, that ASN gets redirected to the slow scrape box.
Also there is ipasn.net created at APNic. host -t txt 8.
Centralise the karma (Score:3)
Where do the lists created by fail2ban and reaction ultimately go? I understand that these are local, but would it make sense to make a karma server where people can look up whether an IP is likely untrustworthy? Such a karma list may help prioritize traffic and help ISPs/providers identify that they are infected.