Mystery Malware Destroys 600,000 Routers From a Single ISP During 72-hour Span (arstechnica.com) 56
A widespread outage affecting over 600,000 routers connected to Windstream's Kinetic broadband service left customers without internet access for several days last October, according to a report by security firm Lumen Technologies' Black Lotus Labs. The incident, dubbed "Pumpkin Eclipse," is believed to be the result of a deliberate attack using commodity malware known as Chalubo to overwrite router firmware. Windstream, which has about 1.6 million subscribers in 18 states, has not provided an explanation for the outage. The company sent replacement routers to affected customers, many of whom reported significant financial losses due to the disruption. ArsTechnica adds: After learning of the mass router outage, Black Lotus began querying the Censys search engine for the affected router models. A one-week snapshot soon revealed that one specific ASN experienced a 49 percent drop in those models just as the reports began. This amounted to the disconnection of at least 179,000 ActionTec routers and more than 480,000 routers sold by Sagemcom. The constant connecting and disconnecting of routers to any ISP complicates the tracking process, because it's impossible to know if a disappearance is the result of the normal churn or something more complicated. Black Lotus said that a conservative estimate is that at least 600,000 of the disconnections it tracked were the result of Chaluba infecting the devices and, from there, permanently wiping the firmware they ran on. After identifying the ASN, Black Lotus discovered a complex multi-path infection mechanism for installing Chaluba on the routers.
ISP should not be allowed to force you to use ther (Score:4, Interesting)
ISP should not be allowed to force you to use there shity router
Re: (Score:2)
No indication that they do force you to use their router, it's just that 99.99% of customers do.
Maybe it was botched malware. Was supposed to steal traffic, but ended up bricking the routers.
Re:ISP should not be allowed to force you to use t (Score:4, Informative)
They're calling it a "router" but it's really a cable modem/router/wireless AP. Putting it into bridge mode so you can hang your own router off of it isn't going to help much when they brick the cable modem.
Re: ISP should not be allowed to force you to use (Score:2)
Thereâ(TM)s no indication that theyâ(TM)re forcing them to use their cable modem.
Re: (Score:1)
There's no support for 3rd party equipment. If you're paying for a supported service and there's no support unless you use their modem; they're pretty much forcing you to use their modem. A basement dwelling slashdot low life could cry otherwise, but that's all they would be doing... crying.
https://www.windstream.com/sup... [windstream.com]
Re: (Score:2)
Re: (Score:2)
It maybe the problem is the remote management of the device. if the remote management keys/password/security is stolen then someone else can manage the device and do whatever to the device. If there were no remote management features then whatever malware would need to get a foothold on your network and potentially not even then be able to destroy the device then since hopefully there aren't global keys/password/security.
Re: (Score:2)
Management from the ISP is usually done on a management network not directly accessible from the Internet. You first need access to the ISP network to access ISP management on the router.
Windstream allows you to use your own modem (Score:5, Interesting)
In fact, if you don't, they charge you a modem rental fee for using theirs. Which is well worth it if you live in a lightning-prone area.
(Ask me how I know!)
You can also place their (or your own) modem in "bridge" mode which makes the modem into a dumb VDSLEthernet conduit that cannot interact with the network in any way. You then need another router (or pc or whatever) with a PPPoE client to establish connectivity.
There is only one instance that you are stuck using their modem in non-bridge mode -- if you get a static IP block, you have to usse this funky "unnumbered PPPoE" mode. And even if you tell it otherwise, their standard modems still perform completely unnecessary stafeful connection tracking that gets overwhelmed by even non-broadband traffic levels.
(Ask me how I know!)
Re: (Score:1)
No indication that they do force you to use their router, it's just that 99.99% of customers do.
Maybe it was botched malware. Was supposed to steal traffic, but ended up bricking the routers.
Possibly. We could also speculate that certain hacker linked countries have been starting sabotage [theguardian.com] and various western leaders have been showing tendency to appeasement which might tempt this kind of action. Definitely its a moment when any companies that want reliable survival should have proper offline backups.
Re: (Score:2)
Possibly. We could also speculate that certain hacker linked countries have been starting sabotage [theguardian.com] and various western leaders have been showing tendency to appeasement which might tempt this kind of action. Definitely its a moment when any companies that want reliable survival should have proper offline backups.
Yes and yes. This may, for example, have been an experiment to see how long the ISP would take to get these people online again and how much the government would get involved. As to organizations that do not have offline or WORM backups at this time, that is basically asking for death.
Re: (Score:2)
Sounds like it. Malware is often customized by people with low or very low skills after stealing it from other attackers. Some malware I have seen had several generations of such pretty incompetently done modifications on it. Hence it is entirely plausible that the attacker messed up. The scale of the problem is a bit unusual though. Possibly the ISP did not patch an already older vulnerability, making them part of the problem.
Re: (Score:3, Insightful)
Nothing worse than what already happened.
Re:ISP should not be allowed to force you to use t (Score:5, Informative)
What could possibly go wrong?
What a fantastically stupid comment. Bring-your-own-router has been a thing since the late 90s with dialup and 1.0 rollouts of broadband. Even Windows 98 had NAT routing included as a service you could turn on called "internet sharing".
If the ISP sets up their network properly, absolutely nothing other than a single customer potentially cutting off their own access because they don't know what they're doing is "what could possibly go wrong."
Re: (Score:2)
If you're having problems with SMTP/25, then that's not a problem with your router if you've mapped TCP/25 to an internal address through the NAT. That's a problem with your ISP blocking SMTP servers on their residential service in order to reduce the amount of spam traffic being generated on their network.
Try using a "business" account where they don't do that kind of thing, if you really must run your own email host.
You're going to have to explain a little better how a layer-3 router has anything to do w
Re: (Score:2)
Again, please point out where your grievance-filled cry about layer 4 concepts has anything to do with layer-3 routing.
If you think that you can just run SMTP on 25 on a residential segment on most large ISPs, you just haven't been paying attention for at least 10 years. And that still has absolutely nothing to do with your router, or who provided it.
And you might try dropping the ad-hominem attack if you want to be taken seriously by anyone. You clearly don't know what the hell you're talking about.
Re: (Score:2)
Bring your own technically happens on Cable Modems and DSL *BUT* you often have to buy an approved device and since the ISP's appear to be able manage/upgrade some of those devices there is some sort of remote management ability.
Best guess is the issue was that the remote management password/key/access was compromised.
I am going to further guess they asked both of them to pay ransom to not access the devices and either the price was too high, or they did not consider that remote access would allow the rapid
Re: (Score:2)
With cable modems, there's a standard for how they provision when they come up and attach to the network, which is the DOCSIS protocol [wikipedia.org]. This protocol allows the modem to download a configuration file that tells it what frequency channels to use for data service, how much download bandwidth it's provisioned for, how much upload, etc.
Usually the "approved" devices are ones that conform to whatever version of DOCSIS that their cable modem terminators employ.
Re: (Score:2)
For most customers, that means they would buy equally shitty routers, or worse.
My ISP-provided Optic Fiber device is in pass-through mode and I use my own router, but that's applicable to maybe a couple percent of the user base.
Rather, ISPs should be held accountable for any disruption caused by their own infrastructure, including routers they imposed to customers.
Re: (Score:2)
ISPs always provide devices from the lowest priced hardware provider, not from the provider with the best price/performance.
Also ISPs usually have a "branded" firmware that they can control aspects of from the outside, something that goes against general security standards. Sometimes this means that the router requests config from a server with regular intervals, but if there's a DNS poisoning then the router will get a false config that nobody knows what it's doing except the malicious intruder.
Re: (Score:3)
Nobody is forcing anyone. What likely happened is users signed up with the ISP, and the ISP provided them a wifi router, and the vast majority of ISP customers have no use for any router functionality past that, and wouldn't even know what it would be to begin with.
The vast majority of broadband subscribers are fine with "the thing the ISP gave me gets me on the Internet, and that's all I need to know about it" even if it's a piece of shit and inadequately secured by the ISP.
Re: (Score:2)
No, it's worse than that.
I tried to buy my own cable modem to use with XFinity. I bought one of the cable modem models that they explicitly listed as supported. But when it didn't immediately work, and I called customer service, they said, "Sorry, we don't support third-party cable modems." They refused to help in any way, ultimately forcing me to go back to their rented cable modem.
These are the games they play. "Sure, you can bring your own cable modem, but we won't guarantee it will work!"
Re: (Score:2)
You could probably have solved it with MAC address cloning.
I'm an unhappy Windstream customer... (Score:3)
They do not force you to use their equipment. That being said, the Sagemcom modem/router is actually pretty good in bridged mode to a real router. The problem with Windstream is that even if you had 10 gigabit fiber, it would be just as bad as the 5x1M DSL as their network is crap and the datacenter all their local CDN nodes are at uses wet spaghetti to connect to the rest of the network.
Re: (Score:2)
In most cases, you can turn it into a L2 (bridge) device and put your own router behind it. That would effectively make the device un-attackable AFAIK.
Not much information in the article... (Score:5, Informative)
So I read the article (no, really!) and it didn't seem to have a lot of useful information. It appears this attack targeted a specific make and model of a router that's placed on the customer premise. The software used, Chaluba, seems to have exploited a zero day on these devices and then rewriting their firmware so that a reset is not possible. Each of these devices would need to be flashed with new firmware to make them work again and even that might not be possible, depending on the new software that's been installed.
This could easily be a test run by a nation-state that wanted to see how disruptive not having Internet would be for the affected individuals. This attack was also done only to devices connected to a single ASN, further leading me to believe this could just be a test. Now imagine if this same unknown attack group were to figure out the simple majority of routers that your average American household uses and then works to exploit them in an orchestrated attack. Then have that happen about a day or two before the November elections. It could definitely be a major problem for the USA.
P.S. I'm guessing this was a zero day since the article mentions they don't yet know how exactly the routers were even infected. This leads me to believe the attack found an unknown bug and exploited it. Interesting stuff.
Re: (Score:1)
The router's configuration network isn't (or at least shouldn't be) directly hooked into the Internet. Just because my router/switch passes information doesn't mean its firmware is addressable. My cable modem does not get an IP, only the computer/router connected to it does (which the router section could be internal to the modem, but isn't usually the same thing).
So likely this was an attack internal to the network, hence it only affecting a subset at a single ISP, based on the description someone (probabl
Re: (Score:2)
Interesting thing is that many fiber modems are accessible by the ISP.
They MIGHT use TCP/IP, but in reality they could use another protocol for that and hope for "security by obscurity", but most likely they are just using a separate management VLAN for their devices.
However if the user traffic goes as the default VLAN (unencapsulated) it's possible to craft packets that looks like they are VLAN encapsulated and can send data into that VLAN, but there's no channel back for that traffic. For UDP traffic it m
Re: (Score:1)
They do use something similar, but it is basically a management VLAN that is indeed security through obscurity and a bit of encryption. It is in the DOCSIS standard somewhere. In the very early days this traffic wasn't encrypted and customers could basically instruct the modem to change their subscription (because things like traffic management were done in the modem based on the number of channels it was allowed to use), so if you purchased 128k (back then) and wanted the full 10M it was capable of, you co
Re: (Score:2)
P.S. I'm guessing this was a zero day since the article mentions they don't yet know how exactly the routers were even infected. This leads me to believe the attack found an unknown bug and exploited it. Interesting stuff.
Yeah, or, these being ISP routers, it could be that there were multiple vulnerable services left installed. Let's wait and see.
Re: (Score:2)
How about this:
Router vendors can provide a ROM version of firmware that will get loaded if they get a dhcp address in a certain range and from there only take a tftp patch, as a failsafe.
That would require a minimal unpatchable bootloader which could potentially be a problem, but maybe the risks of a full flash overwrite are larger.
Imagine the ISP logistics of unbricking millions of devices.
Re: (Score:2)
Directly to the trash, go in the loss column so may be deductible for the ISP. I'm sure those 600,000 routers are in a landfill site.
Re: (Score:2)
Sending them to a landfill would be illegal in many places, so they are probably sent to a dismantling site overseas that then re-sells the chips in the devices so that they can end up in even crappier devices.
Re: (Score:3)
It is really not hard to do: Require any firmware update to have a valid public-key signature. With that, you can even push firmware updates securely without any other measures at all. You could have an FTP drop or the like on the router, for example. Yes, there may be DoS risks from that, so generally you want something a bit more complex, but still. My take is that the people writing the firmware of these routers do not really understand how IT security and cryptography works.
Re: (Score:2)
I still remember that one time AT&T bricked everyone's spare modems by making the modems use a weird proprietary authentication scheme so if they weren't online to get the new certificates before the old ones expired they couldn't connect to get the new ones.
Sure they could have just issued update files so people could just manually load the new version but this is att.
Eventually someone managed to pull the update file from one of the working modems and posted it online so it was possible to unbrick the
Re: (Score:2)
Yep, some companies are really "special". Fortunately, my ISP just delivers optical GbE to me, everything from the fiber socket onwards is my own equipment.
Re: (Score:2)
Yeah we moved from AT&T to the city munifiber which had a outdoor ONT to 100Mbps ethernet, they're still using the same equipment, it's 20+ years old now, no config or anything just straight public IP 10/10Mbps, aside from losing a couple Ethernet ports over the time we used it, it was rock solid equipment.
We since moved to Optimum cable 200/20Mbps, they let us use our own modem (their rental is so locked down as to be unfit for purpose in business use) it hasn't been nearly as reliable and optimum make
Re:Not much information in the article... (Score:5, Informative)
This sounds like a "chain of escalation" vulnerability, where the bad actor uses a series of known low-impact vulnerabilities to gradually elevate their privileges to a point where they can take control
These can be difficult to predict, since they usually make assumptions on what the user already has. A vulnerability that "requires user to have a local account" may appear to be ignorable on a router, until you find another pair of vulnerabilities that allow you to change the password on or enable an existing local account. This makes it possible to leverage otherwise unexploitable weaknesses.
Re: (Score:2)
Re: (Score:2)
"Then have that happen about a day or two before the November elections."
The phrase "...And nothing of value was lost..." comes to mind. No more capability to cheaply push those last minute lies. Or to coordinate tourists.
OTOH, everyone on older media would start pointing fingers at everyone else, each with "incontrovertible" evidence that {those-rat-bastards-on-the-other-side} are attempting to subvert the election to enable {some-unlikely-possibility-that-we-just-thought-of}.
So, yeah, another variable inp
Complex multi-path infection mechanism (Score:2)
What was this complex multi-path infection mechanism? From the provided graphic, an already compromised SOHO runs a script. Would the script have worked if the ISP did not have remote access to the routers.
Malice vs Incompetence. (Score:2)
New DDOS technique (Score:2)
"We lost $1500" (Score:5, Insightful)
"Did do you have business class?"
"well...no"
"Do you have an SLA?"
"well...no"
"So your home internet connection; which by agreement is for home usage; has impacted your business usage?"
"Y-"
"I urge you to really think about confirming your commercial business has been impacted by an outage of your home internet"
"......"
"Thanks for calling."
Re: (Score:2)
"Did do you have business class?"
"well...no"
"Do you have an SLA?"
"well...no, But I have a .357 Magnum"
HAHAHAHA (Score:1)
Billions of CVE Routers still kick'in (Score:2)
In civilised countries they canâ(TM)t. (Score:1)
Only where corporations write their own rules.