Time Warner Cable Experiences Nationwide Internet Outage 133
Wolfrider (856) writes Reports are coming in from multiple U.S. states that TWC is having a major Internet outage since early this morning. ... TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment, and no ETA yet.
According to CNET, most locations should be more-or-less back online as of 7 a.m. EDT or so (my TWC connection came back around 7:30 a.m. EDT). TWC says it's maintenance gone wrong: In response to a query by CNET, Time Warner Cable issued this statement: "At 430am ET this morning during our routine network maintenance, an issue with our Internet backbone created disruption with our Internet and On Demand services. As of 6am ET services were largely restored as updates continue to bring all customers back online."
shoot the admins (Score:5, Funny)
this level of incompetence clearly deserves the penalty of death
my kids woke up this morning and couldn't watch their Phineas and Ferb cartoons. my oldest kid had to actually read a book to pass the time before camp and a trip to the museum
Re: (Score:2)
Re: (Score:1)
my kids woke up this morning and couldn't watch their Phineas and Ferb cartoons.
I hope you reminded them that there is only 5 days of summer vacation left before school comes along just to end it. And that the current problem for them will be trying to figure out how to spend it. Without Cable/Internet. ;)
Re: shoot the admins (Score:2)
Well done you.
Re: (Score:2)
Depends where. In San Antonio, public schools started this Monday and private schools started last week.
Re:shoot the admins (Score:4, Funny)
Same with Cincinnati, OH. The little crumb crunchers are already done with screwing around for the summer.
Re:shoot the admins (Score:5, Funny)
I hope you seized the opportunity to share at least 2 or 3 eyeroll-inducing "back in my day..." stories. Otherwise, I don't know how else you could have possibly salvaged such a fiasco.
Re: shoot the admins (Score:3)
Re: (Score:1)
That musta been you I saw at the Flintstones Museum [creationmuseum.org]
Re:shoot the admins (Score:5, Interesting)
I get the sarcasm, but this outage almost screwed me - I was scheduled to work from home today so I could be here for a service appointment. With no data service, I can't do that.
There are real uses for home Internet connections besides porn and Twitter, you know. =)
Re: (Score:3, Insightful)
I get the sarcasm, but this outage almost screwed me - I was scheduled to work from home today so I could be here for a service appointment. With no data service, I can't do that.
There are real uses for home Internet connections besides porn and Twitter, you know. =)
If it is important you don't get screwed then a redundant connection at home might be the answer rather than trusting a consumer cable company. Unless you were exaggerating with the "screwed" and could just afford to take the day off.
Re: (Score:2)
Buying a redundant net connection on the off chance that your net is down on the one day you REALLY can't leave home seems a little over the top to me.
Re: (Score:2)
Last time we had a DSL outage, we had to use our phones and tablets for connectivity. In many ways, I'm really getting to like the 21st Century.
Re: (Score:2)
Buying a redundant net connection on the off chance that your net is down on the one day you REALLY can't leave home seems a little over the top to me.
I agree. If it is CRITICAL, have redundancy. If it is not CRITICAL, you are not screwed if it fails. He stated he was screwed so I assumed it was CRITICAL.
Re: (Score:2)
No backup Internet service? I have dial-up for these Internet emergencies. :)
Re: (Score:2)
" I was scheduled to work from home today so I could be here for a service appointment"
Was it a service appointment with Time Warner?
Re: (Score:2)
I was scheduled to work from home today
Meh. You are just a fraction of a percent. A rounding error. In this new world of global dominance and market share, fractions of a percent do not count. That is what the MBA folks say anyways. :)
shoot the admins (Score:2)
Re: (Score:2)
A technician told me they are making network changes "because Comcast bought us". Apparently nobody told him it wasn't a done deal.
Re: (Score:3)
While for the standard home user, 4:30-7:30 outage, means you have to forgo your morning entertainment. However there are a lot of small/mid sized businesses dependent on TWC for their operations.
I just wish they would send me an email about the outage so I know not to try to reboot my wireless router then my cable modem. Then plug my PC to the cable modem to see where the connectivity down.
(BTW I was kidding about the email notification)
Re: (Score:2)
TWC just wants your family to buy their TV service to watch cartoons. :P
Re: (Score:2)
TWC just wants you to send them increasing amounts of money. They don't care for what.
I was affected (Score:5, Informative)
My time Warner went out sometime around 4:30 EST this morning, oddly enough it came back up before I was awake (IRC logged my disconnect) and with IPv6 service finally, it's a good chance they had a hiccup doing a mass switch to IPv6
Re: (Score:2)
it came back up before I was awake (IRC logged my disconnect) and with IPv6 service finally
Guess it's time to turn that on on my router...
Re: (Score:2)
Same here with my friend and me in Southern CA, USA: ... ...
[03:58am] * Ant is away: (Auto-Away after 120 mins) [BX-MsgLog On]
ï½ï½ï½ Connection closed from irc...: Remote end closed connection
ï½ï½ï½ BitchX: Servers exhausted. Restarting.
03:58AM ï½ï½ï½ Ant [~ant@...socal.res.rr.com] has joined #linux
ï½ï½ï½ Topic (#linux): http://preview.tinyurl.com/muy... [tinyurl.com]
ï½ï½ï½
Re: (Score:2)
I was affected (Score:1)
I can't believe how much chat there is about this outage, and so little talk about what diagnostics showed the problem was. It was a DNS issue, NOT a general routing issue.
It hit us here in Austin TX. It looked like a DNS outage... but I was using Google DNS. Routing was NOT down... I could still access a selection of web servers by direct IP address, and ping and traceroute. Rebooted my modem and router repeatedly. Modem acquired a link quickly, and status page showed it had a valid configuration. Modem'
Comcast (Score:5, Funny)
TWC: "Surely only Comcast can save us! Please allow the merger!"
Government: "Well golly gee, what a great idea!"
Re:Comcast (Score:5, Insightful)
Comcast: Mwuhahahaha ... no data caps(*), bitches [slashdot.org].
(*) You can buy all the data you want like suckers, and we'll keep getting away with crap like this once we're the only game in town.
Re: (Score:3, Insightful)
Don't worry, in 20 years your only choice will be google.
Think that's hyperbole?
http://fiber.google.com/about2 [google.com]
I'm a big fan of google right now, but let's see how long "don't be evil" lasts once Larry and Sergey have moved on, and MBA brain damage is calling the shots.
Re: (Score:1)
Some of us would argue they gave up on "don't be evil" quite some time ago.
I don't disagree, however, that a future Google will be even more evil.
Re: (Score:2)
Don't be evil? That packet left the switch awhile back....
Re: (Score:2)
Re: (Score:2)
Well, we saw what happened when the Bell System got too big for its britches. Heavily regulated for fifty years or so and then broken up. Too bad the days of the trust buster and anti-monopolist are gone.
Re: (Score:2)
The lack of an edit button is a feature and not a bug. It mimics verbal communication in that someone can correct their statement with an additional statement but their first statement still was heard and processed by the listener.
Having the ability to edit, even one that marks posts as edited or even shows revisions, makes conversation shaky. Posters often have to QFT in order to have their statements be resistant to potential edits. Conversations can still be parsed and followed but it is nowhere near
Re: (Score:2)
If edits were allowed (Score:2)
As a Time Warner User (Score:5, Informative)
I am on Time Warner at home and when I got up this morning I found I couldn't check my email like I do every morning. I spent a bit of time on it trying to figure out what they had done to the connection and found I could not connect to their DNS servers, but I could connect to remote IPs and even trace route them just fine.
I had to head into work, so I didn't stay to actually call them... But I'm hardly surprised if they have screwed up their DNS. I had changed my DNS settings to use Google's DNS servers for a long time because theirs had issues. More recently I rebuilt my PC and I hadn't changed the settings again so they still use Time Warner's DHCP settings to pull DNS server settings.
Re: (Score:2)
I'll add that this was the case at around 6:30 AM EST when I was getting ready. The TW comment says "As of 6am ET services were largely restored", so it may be DNS was just the only thing left not working when I was still at home.
DNS? (Score:2)
Re: (Score:1)
Comcast runs DNS servers? Wow, perhaps I did not need to run my own for all these years! On the other hand, I have not had any problems at my home LAN due to DNS going down.
Can't wait to hear what happened (Score:1)
Re:Can't wait to hear what happened (Score:5, Informative)
Re: (Score:2)
They probably have a peering agreement for edge servers to lighten the load.
Re: (Score:2)
The Hulu co-los are.
The ones hosted at Comcast are hosted at Comcast and so forth and such as.
Re: (Score:2)
This was NOT DNS, multiple sites nation wide on static IPs went down for us as seen by AT&T and downdetector saw nearly every point they monitor within the TWC network as down [multichannel.com].
Re: (Score:1)
Re: (Score:2)
With stuff like "Internet Backbone" I'm hoping for some major BGP messup.
I know the B stands for backhoe, but what do the G and P stand for?
There have been worse outages (Score:4, Informative)
I mean, a couple hours isn't the worst TWC has ever done. Not even in the past year. This is just a wide problem. No one even bothers to note 12 hour fuckups on a regional scale.
Re:There have been worse outages (Score:5, Interesting)
The level of stress and anxiety on the admins during these outages would break most people. I used to do that and literally could not take it, so I switched to a less stressful job. At least with Cable companies it's only TV and Internet, I had to deal with phones as well. We literally had a man DIE during one of our outages because his family couldn't call 911. Even with data lines only all kinds of things break... Police stations data interconnects... hospitals lose access to databases... power companies might not be able to dispatch. Even small outages are a very big deal, and are taken very seriously. The residential side is irritating but not that big of a deal. It's the commercial side that makes it hard to sleep at night sometimes. The guys that do this, do care... if they didn't they sure as hell wouldn't put up with the miserable job.
Re: (Score:1)
Once again showing why analog is better for phones than digital. When was the last time you ever heard of an analog phone line not able to transmit a call? Now compare that to how many times you've have dropped calls or, in this case, no ability to make a call, on a digital line.
But let's keep saying the move to digital is better than what we had, despite the overwhelming evidence to the contrary.
Re: (Score:1)
Re: (Score:2)
Or maybe this guy was going to die no matter what. This sort of thing happens. While our modern infrastructure is important and we should angst about major fuckups (especially if they involve TWC and friends), even the old unbreakable Bakelite phone / POTS system screwed up from time to time.
The other thing to realize about the current 911 system (at least in the US) is that it is a fairly robust, fairly smart system. In the 'old days' when we had to make sure that paths were clear of T. Rex and raptors,
Re: (Score:2)
Once again showing why analog is better for phones than digital. When was the last time you ever heard of an analog phone line not able to transmit a call? Now compare that to how many times you've have dropped calls or, in this case, no ability to make a call, on a digital line.
But let's keep saying the move to digital is better than what we had, despite the overwhelming evidence to the contrary.
It was, in fact, and analog POTS line. POTS lines go down all the time... all over the place. The difference is, IP traffic is like an airliner going down... when it happens, it's big and happens to lots of people, so it gets more attention. POTS outages usually effect only a few hundreds people, but there are lots and lots of them all over the place all day long.
ALL telephone service, being analog or or IP phone gets MUXed at the DSA and turned digital within 30k feet of your house anyway. The biggest prob
Re: (Score:2)
Here (in upstate NY) Verizon runs the phone system - and they can't keep that thing running for 10 minutes it seems - if you can even get it (my appartment building is not wired for Verizon anything - phone or other). In my old place an hour away I had Verizon phone and DSL (don't get me started), that's where the phone kept going down. Cable is rock solid, and home cell service is a good backup (they even offer a home phone box that has a nice built in battery backup that can run for a few hours).
On a side
Re: (Score:2)
I once lost it and snapped at a manager during an outage, in front of many suits in the "war room" and on the conference call. It went something like "I am actually trying TO FIX THE PROBLEM so BACK OFF!" H
Re: (Score:2)
This is why monopolistic behavior should be stopped, middle management doesn't even know how bad they are breaking their market - they suffer. I can't imagine this fellow was any more useless than the rest of them. What is needed is that they listen to their technical staff before things shatter like this, as you say the network engineers would rather have things run smoothly.
It's not monopolistic at all. There are plenty of competitors. If there weren't we wouldn't be having layoffs and sending support overseas. Customers want the cheapest rates possible and will except sub-par service if that's what it takes.
Coincidence? I think NOT! (Score:2)
Customer service.. (Score:1)
"TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment"
So it's just a normal day then? It seems that whenever I call one of these companies all I get is "due to unusually heavy call volume we are experiencing extended hold times".
Re: (Score:2)
Try calling their cancellation lines after an unannounced rate hike(like I was even using the TV cable, and I'd want to pay more for it).
Is this news? (Score:3)
I haven't had TW for years, but when I did, it was newsworthy when it was actually working.
Re: (Score:2)
I've been using them since early April of this year, and so far so good. Aside from this incident, I think there was one other instance where I was down for maybe 15 minutes.
That said, they're overpriced and I don't trust them at all. If the Comcast thing goes through I'll be looking at my options, but unfortunately I don't think I have many good ones.
Re: (Score:2)
Earthlink over TWC cable here (Score:2)
Couldn't even ping 8.8.8.8, so I suspect it was more than just a problem turning URLs into numerical addresses.
It was so weird to have problems and still have all the lights on the modem lit up (except standby).
Unfortunately there's no novelty factor in having problems with it in general.
Re: (Score:2)
Re: (Score:2)
I work for an ISP. Trying to figure this stuff out from the consumer side is kind of pointless. When things break this bad, we're bringing things down and back up again faster than you can blink. Even if the problem is just DNS, we might end up turning all the interfaces on the core routers up and down. So your tests might have just been showing you the recovery efforts. Trust me, these things are never as simple as the right person noticing and flipping a switch. I guarantee you there were 20 or 30 bleary
Re: (Score:2)
It won't stop a lot of us from trying. Many of us base our careers largely on the ability to find and analyze problems, and it's almost instinctive by now.
Re: (Score:2)
The lights on the modem only mean you're talking to the CMTS on the other end of the neighborhood wiring, and that you were able to provision through DOCSIS.
If they fuck the routing over, you'll have all the blinkinlights in the world, but no packets get where they're supposed to.
Re: (Score:2)
It's just that all the many many other times I've had problems, I could see that at least one of the lights that were supposed to be on wasn't on. This was the first time nothing "looked wrong".
And they're the LESS evil giant cable company (Score:5, Interesting)
Screw-up? (Score:5, Interesting)
Did the NSA screw up again while tapping in into the TW infrastructure? I mean, if the manage to knock whole countries of the Internet by doing so, I wouldn't be surprised if they knocked a few ISP offline now and then.
Re: (Score:2)
Did the NSA screw up again while tapping in into the TW infrastructure?
Yeah, NSA is busy upgrading all their gear. They took out Charter earlier this week. Today it was Time Warner's turn. They'll get to Comcast in a day or two.
jumped the gun on the comcast merger? (Score:3)
Did they jump the gun on the comcast merger?
Comment removed (Score:5, Funny)
Re: (Score:2)
- Robert Marcus, CEO (who cares a lot about you)
Re: (Score:2)
I loved this whole post but this part was the gem
seriously most of the phone tree is a circular loop. we have maybe 4 phones in this office
Re: (Score:2)
One Pingy Dingy! (Score:1)
Irving P. Freely,
SVP of Customer Relations, AT&T
Chicago was hit by this. (Score:2)
Now I'm inclined to believe the typical explanation of "A wizard did it". God dammit, Dresden, I thought we told you to stay away from the datacenters.
It is probably your hosts file, try sending :80 th (Score:1)
Re: It is probably your hosts file, try sending :8 (Score:1)
This is a non-story (Score:1)
If we didn't hate TWC for other reasons, this would be dismissed as a quickly-corrected outage resulting from human error during some maintenance operation. But since we hate TWC, people make a big deal out of it and declare conspiracy and yell about bad customer service.
Nothing to see here, people.
Epic fail... (Score:3)
The fact they have ONE backbone connection is an utter and epic failure. Who designed their network because that person needs to be fired.
Each major section or city needs it's own backbone connection. At least that is how we did it back in the old @home days, one failure can not bring down all services across the nation. Some cities had multiple backbones going into the area's OTN.
Re: (Score:3)
The fact they have ONE backbone connection is an utter and epic failure. Who designed their network because that person needs to be fired.
That would be the MBA who said no we can't afford to buy that much connectivity, cram it all through one connection. 'cause we need more PROFIT! MOAR!
Re: (Score:2)
The fact they have ONE backbone connection is an utter and epic failure.
Wait, you're not believing what the PRBot says, are you?
Re: (Score:2)
then you explain how a single point failure will take out service across the country. There is no way in hell that can happen if the network was designed properly.
Re: (Score:3)
If the routing protocols (e.g. BGP) get screwed up, it doesn't matter how many peer connections are up, the network doesn't know where to send the traffic.
You haven't done any serious networking, have you?
Re: (Score:2)
Epic fail, routing protocols on 170+ POP routers all at the same time? Again, epic fail on design and implementation.
Don't worry. (Score:2)
This will never happen again once Comcast buys TWC.
Overwhelming call volume (Score:1)
TWC's customer service reps are reportedly a bit overwhelmed by call volume at the moment
Hopefully, their remaining customers, who evidently are willing to wait on hold for at least two hours under normal circumstances, will understand and will patiently wait on hold for ten hours for the privilege of being told "We're looking into it but there's nothing we can do right now."
Luckily, TWC has competition in my area. Every time I see an ad from them that seems cheaper than what I'm paying to their competitor, I remember the horrifying customer service, and the urge to save money goes away.
Auto-Qualification (Score:2)
Same bleep as ever (Score:1)
I'm a TW customer, and I didn't notice the difference because TW always sucks (and so does their sole competition in the area. F oligopolies).
Re: Or as we call it, Wednesday (Score:1)
Really? Moderated as flamebait. Really? Fuck you.