Belkin Router Owners Suffering Massive Outages 191
An anonymous reader writes: ISPs around the country are being kept busy today answering calls from frustrated customers with Belkin routers. Overnight, a firmware issue left many of the Belkin devices with no access to the customer's broadband connection. Initial speculation was that a faulty firmware upgrade caused the devices to lose connectivity, but even users with automatic updates disabled are running into trouble. The problem seems to be that the routers "occasionally ping heartbeat.belkin.com to detect network connectivity," but are suddenly unable to get a response. Belkin has acknowledged the issue and posted a workaround while they work on a fix.
Oh hey, consumers! (Score:5, Funny)
Re:Oh hey, consumers! (Score:5, Funny)
I'm going to be naming all my deadman switches "heartbeat" now. (:
Re:Oh hey, consumers! (Score:5, Insightful)
I think the most awesome revelation about this whole thing is that you can stop a lot of people from accessing the internet if you simply DDoS heartbeat.belkin.com.
Re:Oh hey, consumers! (Score:5, Funny)
Mod parent up. (Score:5, Insightful)
Yeah! Who the fuck thought that was a good idea?
Sounds more like "all of the internets is broken because this one site won't work" complaint I get all the time.
It's a ROUTER. If the physical link is up then try pushing packets through it. That's all.
If you want to show connectivity to a specific site then show that in the diagnostic page on that router. But keep pushing packets.
Re:Mod parent up. (Score:5, Informative)
Re: (Score:2)
The issue, specifically, is that it breaks DNS if it can't get it's heartbeat. It's still stupid but the device continues to be a router.
Why would switching over to Google's DNS fix this?
Re: (Score:3)
I'm typing www.slashdot.org, but it's the website that's broken.
Re:Mod parent up. (Score:5, Interesting)
Apple does that too, though on end-user machines. When connecting to wifi, it doesn't enable the connection until it first verifies you're really connected. It does that by trying to pull a specific known Apple URL. If it doesn't get the expected contents, it guesses you're behind a wifi hotspot's login wall, and pops up the "please log in" page. The intent of this is to make sure apps like Dropbox and your email and whatever don't think they're back online and start failing connections, in the time between when you connect to a hotspot wifi and when you log in. But it also means that if Apple's URL goes down, wifi connection will end up with extra hoops to jump through to get it to work.
Re:Mod parent up. (Score:4, Informative)
"Apple does that too, though on end-user machines. When connecting to wifi, it doesn't enable the connection until it first verifies you're really connected. It does that by trying to pull a specific known Apple URL."
I'm sorry, but there must be more to this than your description.
Just for jollies I unplugged my cable modem and fired up my Macbook. It connected to Wi-Fi and I was up and running on my local network same as always.
A.
Re: (Score:2, Informative)
iPhones and iPads definitely exhibit this behavior, but I think only on Wifi networks that do not have any security enabled (which would be most public hotspots). When the login page pops up, it always has a apple URL before quickly switching to the login page's URL.
Re:Mod parent up. (Score:5, Informative)
And adding to the list, Windows does it as well. It's called Network Connection Status Indicator [technet.com].
NCSI is designed to respond to changes in network conditions, and examines the status of a network connection in a variety of ways. First it uses an active probe to determine the status. For example, in an active probe NCSI tests connectivity by trying to reach http://www.msftncsi.com/ [msftncsi.com] a simple Web site that exists only to support the functionality of NCSI. Eventually, as other programs begin generating Internet traffic, NCSI switches to a passive monitoring process that assumes responsibility for detecting changes to the network status.
Every time a network configuration event occurs (meaning that something has changed in the network configuration), the NCSI process performs several tests to identify the network's connectivity status. The first step NCSI performs is a DNS query for www.msftncsi.com. The second step is and HTTP get request for http://www.msftncsi.com/ncsi.t... [msftncsi.com]. This file is a plain-text file and contains only the text "Microsoft NCSI." Last it will perform a DNS query for dns.msftncsi.com.
The URLs used by NCSI can be changed via Group Policy, i.e. you can have it check for the presence of some local server, so that it doesn't bug the shit out of users on a network without external connectivity. Several weeks ago, Microsoft was having global DNS troubles, and many users reported seeing the "trouble" icon in the tray even though their internet connection was working just fine; the problem was that msftncsi.com wasn't resolving. Whoops.
Re: (Score:3)
But all that means is that the user sees a yellow splat in the system tray. Not really a big deal. You can still actually access the Internet on Windows even if NCSI thinks it's down. The Belkin issue is a much bigger deal.
Re: (Score:3)
The important difference is that Windows just puts up an icon saying "your connection looks broken", it doesn't stop resolving DNS like Belkin routers do.
Re: (Score:3)
Yeah! Who the fuck thought that was a good idea?
IIRC it was Cisco. [crn.com]
Re:Mod parent up. (Score:5, Interesting)
Re: (Score:2)
I don't know how much traffic Microsoft really sees (I assume it's quite a bit), or BofA would put out (probably a fair bit as well), but if I was running a network and saw a range of IPs pinging me all day every day I would be pretty hard pressed to not block them. I mean, why is Microsoft paying for BofA's internet connectivity testing?
Re: (Score:3)
Re:Mod parent up. (Score:5, Funny)
Re:Mod parent up. (Score:5, Informative)
No, it's a BELKIN. They've been pulling stupid shit like this for the last 10 years at least , so why anyone should still be surprised is beyond me.
Remember that - whatever you buy from them, it'll always be a Belkin first and <device> second.
Re: (Score:2)
This. The biggest train wreck of a router I ever tried to use was a Belkin. It rebooted every few hours, depending on load... when it was new.
Useful? Maybe if you could disable this 'feature' (Score:3)
It might be useful to have a way to disable this 'feature' on the client side.
The bad? There isn't.
The good? This 'feature' already broke connections for anything going through the campus NAC even before their heartbeat server crapped out. SOP for any Belkin-involved problems became "Belkins break RFC2616, they are officially unsupported, go return it and get something that doesn't suck." ...so there aren't any of them still in use to be broken today. Yay!
Re: (Score:3)
It seems as though installing DD-WRT/OpenWRT/Tomato/other-non-OEM-firmware will fix it on at least some routers made by Belkin.
Re: (Score:3)
Re: (Score:2)
Be reasonable. It may have been *sys.
Re:Oh hey, consumers! (Score:5, Funny)
It was a part of their advanced simulation program that slipped out. This particular feature was designed to simulate what it's like to be a Comcast customer.
Re: (Score:2)
If I was bored, and wanted to do something for teh lulz, I would organize an ongoing campaign to run a DDoS against heartbeat.belkin.com. If I was that type of person, anyway.
Re: (Score:2)
Live by the cloud, die by the cloud. (Score:5, Insightful)
Did Belkin tell you their router was dependent on their site being up?
"When I die, the world ends." - Belkin policy
Re: (Score:2)
I'm confused as to what DNS server these Belkin routers were originally pointed at.
And why not using that DNS server has fixed the heartbeat ping issue.
Re:Live by the cloud, die by the cloud. (Score:5, Interesting)
Why not using that DNS server has fixed the heartbeat ping issue.
Their router may be trying to distinguish, as Windows and most things that connect through WiFi now have to, between real Internet connectivity and fake Internet connectivity. Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad. So, many devices try to connect to some known site which produces a known response to verify that they can connect to the outside world.
It's the choice of "known site", and not having alternatives for it, that's the problem.
Re: (Score:2)
That portable devices do that for the reasons you stated, I can understand.
But for a fixed, non-mobile router? WTF Belkin?
Re:Live by the cloud, die by the cloud. (Score:5, Interesting)
Fake Internet connectivity is when some WiFi access point hijacks all DNS requests to take you to some login web page or ad.
So my company presents at trade shows. Trade shows often have Internet service available at ridiculous prices, and frequently, performance is horrible. Often, rather than pay that ridiculous price, we have a laptop set up with the same configuration as our servers, and run with a recent backup copied onto the laptop. This lets us demonstrate our products with a "sandbox" - same as we use for development - without having to bother with the on site Internet.
Our mobile "server" is set up to wildcard DNS to a locally hosted copy of our website. Other vendors, of course, see our hot spot and figure they can use it to get Internet service on somebody else's dime. When they find that all they can get to is our website and product, it's typical for them to get upset - more than once we've been accused of hacking!
Now, set up the hot spot with an SSID like "NoInternetHere" as a way of discouraging trouble.
Re: (Score:2)
I'm mostly repeated what ArcadeMan said, but why would this make any sense for a router? It does make sense for mobile devices which use WiFi to connect to the internet, thanks to all these shitty hotspots that require you to click "agree" or whatever before you can use the internet. However, a router is not like that, it isn't even connected to WiFi (not on the WAN side), it's connected directly to the ISP through a modem. Unless there's some shitty ISPs giving you fake internet connectivity every day u
Bad Belkin (Score:5, Insightful)
No router should ever be dependent on phoning home to a server in order to work. (No router should be engaging in any communication at all that I haven't told it to!) This is BAD - Broken As Designed. I'm awfully glad that I don't use Belkin stuff.
Re:Bad Belkin (Score:5, Informative)
Agreed. I am astonished that this happened. I thought the router companies would have learned their lesson after the SNTP nonsense over a decade ago. Clearly I was mistaken. For those that do not know about that incident, here you go:
http://pages.cs.wisc.edu/~plon... [wisc.edu]
Re:Bad Belkin (Score:5, Interesting)
It's a feature. By pining Belkin's servers they can keep tabs on their customers. See how long they keep their old routers for, what reliability is like, if they replace it with another Belkin etc. Even just knowing the number of active users is valuable marketing data for them.
Re: (Score:2)
Um, I didn't think ICMP pings could transmit all this information. Unless TFA is misusing the word "ping" and it's really some other type of packet.
Re: (Score:2)
This is probably, as usual, the marketing department overriding the decisions of the technical department.
Re: (Score:2)
I've had routers hang and stay hung overnight. DD-WRT has a configurable heartbeat ping. I have that turned on.
The issue is not so much the pinging, but the fact that it had a heartbeat that was not nuclear-bomb-proof and not re-configurable.
Go for the 100% Open Source option (Score:2)
I've had issues with the last several routers, so I recently bought the very first, 100% OSS router [fsf.org]. My thinking is that if it's open source, it's probably high quality code, and it's more likely to get updated than proprietary firmware, where they are cash incentivized to just have you buy the new router rather than fix old bugs.
As far as hardware goes, it's mid-range router hardware, N300 Wifi with respectable antennas and a ho hum 100 Mbit hardware switch. The UI was a little odd, more complex and far
Re: (Score:2)
Yes, and that usage makes sense. But we aren't talking about endpoint devices like phones or tablets. We're talking about routers. On routers, this makes zero sense and is objectively bad.
I quit using Belkin years ago, (Score:5, Informative)
any type of device, they won't get my money for even a power strip.
They earned my boycott honestly [slashdot.org] years ago. I still haven't let them off the hook. Comments on that article exposed other reasons not to even give them the satisfaction of wiping your ass their products.
Re: (Score:2)
Be ready to boycott linksys too. Belkin now owns the linksys brand
Re: (Score:2)
No Belkin, no LinkSys... who's still making routers that you can find in regular stores?
Re: (Score:2)
I switched to an Asus wireless router at home a few months ago. Can't speak to the quality of the firmware as I loaded dd-wrt on it, but it has worked nicely so far. Picked it up in a local store, though I don't recall which.
Re: (Score:2)
My current router is Buffalo - think I bought it from Newegg, but you can find this brand at Fry's, if you are lucky enough to have one nearby. It came with dd-wrt preloaded. After one firmware upgrade, it has been reliable as anything (470 day uptime).
I have a separate device that monitors internet connectivity (DSL here) and will reboot the modem if an outage is detected. It monitors FOUR remote addresses to avoid the stupid problem Belkin has inflicted on its customers. Yes, sometimes one or the othe
Re: (Score:2)
Re: (Score:2)
Thanks to this thing called "the internet", which you may have heard of, you don't need routers to be available in local stores any more. Instead, you can buy them online from places like Amazon.com and Newegg.com and countless other online vendors, which offer lower prices and better selection than almost any local store.
I hear Buffalo routers are excellent and include DD-WRT firmware from the factory.
Re: (Score:2)
I guess I can't be too surprised that they'd pull a cunning stunt like this, just because they got caught with their hands in the cookie jar already, with the "spam router" fiasco. It's hard to believe it's been over a decade since that, and they're still baking stupidity into their routers.
It's sad to see they snapped up Linksys, but Linksys was already on a downward spiral anyway. In any case, I'm not buying a router unless I can install DD-WRT or OpenWRT on it. Of course, with Comcast now pushing integra
Re: (Score:2)
In any case, I'm not buying a router unless I can install DD-WRT or OpenWRT on it.
This is what I do. I can't think of a company that makes routers whose firmware I trust, so I only use routers where I can install my own.
Of course, with Comcast now pushing integrated router/cable modem setups
You don't have to use their equipment. You can get your own cable modem for under $200 and use that.
Re: (Score:2)
> Of course, with Comcast now pushing integrated router/cable modem setups
You don't have to use their equipment. You can get your own cable modem for under $200 and use that.
That may work for now, but for how long? They can decide any time to change that policy and force you to use their equipment, whether you like it or not. What are you going to do, change to another ISP?
Personally, though I have Comcast now, I'm going to be moving out of the state soon to a state where Comcast isn't present, and I
Re: (Score:2)
Linksys was done in during the days it was owned by Cisco, who apparently put the squeeze on to increase profit margins. The last Linksys router I bought had had the memory cut in half from the prior version in the series, just to shave a few pennies off the cost, but as a consequence it was flaky and unreliable compared to the previous model I'd bought during pre-Cisco days. It took a while to figure out that it was the router that was at fault.
So that explains today's Slashdot outage (Score:2, Flamebait)
In retrospect (Score:3, Funny)
It was probably short-sighted to write their main event loop like this:
while (!((date.day == 7) && (date.month == 10) && (date.year == 2014))) {
// rest of router code follows
}
Re: (Score:2)
Re: (Score:3)
while (date.year != 2014 || date.month != 10 || date.day != 7) {
Before and after 2014, you only do the year comparison, for 11 months of 2014 you only do the year and month comparisons, and only for the month of October 2014 do you have to compare all three. That's 31 days of doing three comparisons, 334 days of doing two, and infinity days of doing just one.
Your code first evaluates the day, which will equal 7 12 times each year, so you're doing tw
Re: (Score:2)
Just a nitpick, but it's more efficient to use OR...
while (date.year != 2014 || date.month != 10 || date.day != 7) {
Yes, by putting the year first, it would short circuit the rest of the evals, but my code's goal was to have it not run *only* today, so you definitely want to stick with AND, otherwise you're shutting it down for the whole year. ;-)
Re:In retrospect (Score:5, Informative)
I think you need to review your boolean logic. !(a && b) is equivalent to !a || !b
Re: (Score:2)
Re: (Score:3)
My code:
Is it 2014? No: Enter Loop. Yes:
Is it October? No: Enter Loop. Yes:
Is it the 7th? No: Enter Loop. Yes: Break Loop.
Your code:
Is it the 7th? No: Return False. Yes:
Is it October? No: Return False. Yes:
Is it 2014? No: Return False. Yes: Return True
Negate the return value of the compound logic.
Is the negated value True? Yes: Enter Loop. No: Break Loop.
It would be more clear in a flow chart, but your code does a lot more branching than mine, and branching uses CP
Re: (Score:2)
Re: (Score:2)
linksys (Score:4, Insightful)
Belkin Router Owners Suffering Massive Outages (Score:2)
Is this resolved? (Score:2)
I'm getting a reply back from heartbeat.belkin.com. I don't have a belkin router to test with.
Why a fixed hostname? (Score:4, Interesting)
Old cable modems sucked. Mine would often lock up, needing a power cycle to resume working. Very annoying when I was at work.
The quick and easy solution is to monitor the connection status and flip a relay to reboot the modem. But how to monitor the connection? Setting a single host or IP seemed like a bad idea because it would have added an extra, and totally unnecessary, single point of failure.
Instead, my home router (slackware box with 2 ethernet cards) collects the IPs that I connect to (by watching the conntrack stuff in /proc/ ), and if it can ping them, adds them to the ping list. It then pings random selections from that list to verify connectivity. IPs are removed if they are unreachable for a while (until it decides the connection is down; no point purging the whole list because of an outage).
Took me a couple of hours to set up and debug, back in like 2002 or 2005 or whenever I wrote it. I presume that there is some free software to do the same task by now.
Monitoring a single fixed hostname is foolish, at best. And this is like the 3rd or 4th big story (that I can think of) about home routers acting badly because of hardcoded values.
Re:Why a fixed hostname? (Score:5, Interesting)
I wrote scripts to do ADSL load-balancing / failover using kernel patches that allowed all kinds of things that aren't in the base kernel. Problem was that our ADSL modems were sucky and wouldn't bring the connection back up when it could have done, so I stuck a £5 Velleman electronics kit and a couple of relays into a box, and with USB control we could reset them from the router itself.
It ran a school for 5+ years, even able to stick 3G sticks into the list and let it failover to them when a dead connection was spotted. And, handily, the 3G sticks worked as a perfect "text-to-fix" receptor and also sent out and received text messages on behalf of the school at the same time. Hell, in one emergency, we even just bought a shed-load of SIMs and every time we hit 1Gb data on a SIM, it turned it off, we changed the SIM for the next and threw the first one away (to get around stupid low data limits). We literally didn't have anyone know we'd done it, from inside the school, except myself and the bursar who bought the SIMs. Everything just worked seamlessly.
But, just watch number of packets incoming on connection. It's much easier. If your external DNS is down, you aren't going anywhere anyway, without manual intervention. If the root DNS is down, you're fucked. If traceroute can't trace to your ISP's gateway, you're probably dead anyway. All of these work, there's no need to get too complex and ping-out.
So if you aren't getting DNS packets coming back from simple queries, you might as well consider the connection dead and move onto the next. That's what we did. Then a few second later you'd hear a click, the lights would flash on one of the modems, and in a couple of minutes it's would be back up and pass traffic again.
The biggest problem? We had to put TWO IP's on the external VPN list because you were never quite sure what line was up and handling the route for the VPN. It could be either. Plug both in, let the VPN client try both. End of problem.
Was so sad when I realised that I'd left the hardware and scripts for that at my previous workplace.
Re: (Score:2)
There's one or two devices (called "IP Swtiches") that do the monitoring and power cycling all in one box. They work basically the way you've described.
Google Public DNS? (Score:2)
Re: (Score:3)
No they won't.
From here: [google.com]
uh oh (Score:2)
Add this to the list of reasons I never buy Belkin (Score:2)
Arbitrarily breaking HTTP is a bad idea. Who knew? (Score:5, Interesting)
Entertainingly enough, I've run into this issue before. You will encounter the same issue when trying to connect the affected Belkin routers through the Cisco Clean Access NAC here (AKA Campus Housing), because devices are quarantined in the VLAN ghetto until successfully authenticated and associated.
So, these terrible, terrible Belkin routers try to phone home, and when they are unsuccessful they redirect all HTTP requests to the router's administration page. Since sessions are required to authenticate via HTTPS, there is no way to login. Extensive investigation revealed no way to disable this behavior on the client side, SOP for anyone calling with connection problems involving a Belkin router became "Officially unsupported. Return it and get something else that isn't a Belkin."
I am beyond pleased that this incredibly foolish decision on Belkin's part has come back to bite them in general, and hilariously entertained to see that Belkin's temporary workaround was effectively "spoof DNS traffic to heartbeat.belkin.com to a server on your local network that will pingback to fix your ISP's broken clients"
The industry needs to stop this (Score:2)
Can't wait to see what happens years from now should Microsoft's NLA site become unreachable to one or more address families as the world swashes between IPv6 and IPv4 connectivity as a result of failed NLA probes.
There is no need or benefit for this garbage certainly not by default and certainly no excuse to failure in this manner. Heartbeat is code for more excuses for vendors to be in the loop and collect data when they have no legitimate business doing so.
I don't Belkin - but can firmware be flashed? (Score:2)
I just went to www.dd-wrt.com and did a quick scan of their firmware and the results are a bit spotty but possible for some types of routers. But if your router is on the list then I'd get the hell out of the Belkin umbrella altogether. Might wish to check out Tomato, OpenWRT or FreeWRT too. If your router isn't upgradeable, smash the fucker in a fit of rage (it'll feel good, I promise) and after you collect enough empty bottles, go buy yourself a new one that isn't made by Belkin and IS on on of these firm
LOL Belkin, color me surprised (Score:2)
My favorite brand is Logitech. Their stuff is cheap, high quality and lasts.
Re: (Score:2)
For keyboards, I've found Corsair's K## line to be quite reasonably price
How to disable similar features on other platforms (Score:2)
Windows registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet
EnableActiveProbing = 0
Android terminal:
settings put global captive_portal_server 127.0.0.1
settings put global captive_portal_detection_enabled 0
Belkin is out (Score:2)
Spent close to 2 hours with this in the AM. First, the ISP; ie, TWC. They directed me to Belkin. So I tried calling Belkin. It took an hour to get tech support. Their phone system kept disconnecting me. Plus I was trying to access their web site via my phone. Of course, their site was hosed, as well.
When I finally got through, it was to someone in India. She was very thorough, but ultimately no help. (Now I know why.) She assured me I would get a call back in about 2 to 4 hours with a solution.
After finishi
Belkin? Unpossible! (Score:3)
This isn't the first time Belkin has implemented a hare-brained feature, only to have it cause backlash when it induces catastrophic failures across the world. I stopped buying anything with their name on it (except the occasional cable) over a decade ago, over this little feature [cnet.com].
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I've experienced rare Internet outages and usually wait 20 minutes or so to call to see if there's a problem and the auto answer will confirm that but also say that one can get more information about outages by going to Comcast's home page.
Dude, you should know, you always get better response from customer support about internet outages if you send them an email instead of calling.
I once had a Comcast call handler try to upsell me to Xfinity voice for my home phone while I was calling about a complete cable outage (no TV, no Internet).
Re:Ummm - did we forget the obvious? (Score:5, Funny)
Re: (Score:2)
My favorite email from my tech support days was, "I can't send email."
Ok. You can't send email so you sent me an email to tell me. And it worked so, clearly, you can send email. It was always an email address they'd mangled except when it was a .vacation message reply.
Step by Step Instructions (Score:3)
1) Unplug Cat5 from router.
2) Plug Cat5 into computer.
3)
4) Profit, er Internet.
I know, too complicated right?
Re:One rule comes to mind... (Score:5, Funny)
Dude, such a wasted opportunity to say "Master of your own domain ".
Re:One rule comes to mind... (Score:4, Funny)
Yea, I know, but I'm married...
Re:One rule comes to mind... (Score:4, Interesting)
This is true for all people who understand the code of OpenWRT in its entirety.
Else it's simply a choice of picking who to trust.
Re: (Score:2)
This is true for all people who understand the code of OpenWRT in its entirety.
Else it's simply a choice of picking who to trust.
Personally, I'd go with the software that has the open source which I can obtain and look at if I want over the "you will never get the code" from the router vendor. (At least at the consumer level.) I'll trust the one I could possibly verify over the one I never could.
I think your trust is well placed if you choose the ready build OpenWRT firmware... If you don't like that, build your own in a VM like I had to because they don't release firmware for my specific router.
Re: (Score:2)
Re: (Score:2)
Not sure I agree with you. I have a set of Netgear routers that you can get used for under $50 each that work with OpenWRT well and are not that far out of date. (Netgear actually used OpenWRT so they work well given that they have to release the source code.) Now if you are looking for something more than a router, say a media server or VPN gateway, they might be a bit light, but as a standard wireless router they are quite adequate. They have a lot of flash, good memory size and reasonable performance t
Re: (Score:2)
I'll probably replace my current PFSense box with the next gen of these kinds of boards.
Re:First! (Score:5, Funny)
Looks like you were behind a Belkin router.
Re: (Score:2)
Re: (Score:2)
interesting.
Re: (Score:2)
Re: (Score:2)
dd-wrt lets you manipulate the routing tables, which I have used to quash one device that kept "phoning home" but works fine when "home" no longer responds.
Re: (Score:2)