Confirmed Gmail / Google App Outage 189
mbone writes "Earlier today there was a confirmed Google outage which got a lot of attention from network operators. From a post to NANOG after everything calmed down: 'Google ack'd a maintenance on their core network did not go as planned-Forced traffic to one peer link that was unable to handle all the traffic. Maintenance has been rolled back. Issue has been restored.' This is exactly what makes me nervous about cloud computing and data storage. It's bad enough when I screw up a config and it takes down my mail, but what about when it happens to the entire globe at once?" Several readers also point to CNET's coverage of the outage.
Update: 05/14 19:25 GMT by T : CWmike adds this: "Steven J. Vaughan-Nichols writes that what may be happening is a massive DDoS attack. Based on the size of the attack that would be needed to interfere with Google, I believe that it's quite likely to be the result of an attack from the controllers of the Windows worm, Conficker. Another theory that has been put about — that the problem was due to AT&T NOC routing problems — does not appear to hold water, writes Steven."
Update: 05/14 21:01 GMT by T : Google's put up a low-detail explanation on their blog that says "An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam. As a result, about 14% of our users experienced slow services or even interruptions."
Google Voice Issues (Score:5, Informative)
1) Text messages sent from the web got duplicated. One person got near 10 duplicates in quick succession. I also got duplicate messages back.
2) My number doesn't work. If you call it you get a "Currently unavailable"
3) A few calls that came in before the outage aren't showing up in the Received/Missed calling list.
Re:comments from google (Score:4, Informative)
This outage was a sign that Gatus and Joba are having some success in their quest to thwart Googol the Destroyer [slashdot.org]; apparently, not only are they getting converts in their efforts to have all the world's sorcerors collaborate on the One True Operating System with Global Search, but they (or someone else) has launched an effort to delay the work on the Rite of a Million Targeted Ads by Googol the Destroyer and his acolytes.
Who is responsible for the guerrilla attacks on Googol the Destroyer, his acolytes, and his crack team of evil underlords? How are Gatus and Joba faring on their recruitment of the world's sorcerors? What has Stallmanx been working on in his secret laboratory*? Will we ever find out what lies beneath his Beard of Druidic Prowess? Answers to these questions and more will be revealed in the next two episodes!
Remember, kids, tune in to next week's episode (look for it Tuesday or Wednesday) of Googol the Destroyer!
Re:Mail Servers (Score:5, Informative)
>>And what's -really- the difference between a server going down locally that affects you and a server going down globally that affects you? Nothing.
Actually, I disagree. There is a difference. If it's local and I own it, I have to fix it. If it's outsourced and Google owns it, I sit back and let Google fix it. Which is nice.
ThePlanet.com had a bad switch install a few days ago which brought down part of our cloud. Our website was down, as was our access to Google DNS gave an IP down there for Google. If you look at the last year, the cloud solution has had a better uptime than what I was providing computing in planned maintenance, patching, updates and all.
It was nice to leave at 5pm, knowing ThePlanet would fix the switch and get us back up. And they did. It's a lot easier to gripe about the cloud being down and sit back, than to manage and fix your own local servers switches and such. When you get to managing hundreds of servers, it becomes time to know what to outsource.
Re:Then download your google mail (Score:5, Informative)
Re:Then download your google mail (Score:4, Informative)
I know it is poor form to reply to your own posts but I have just read the full article above and discovered that us in the UK seemed to be ok. Also not affected was the West coast apparently.
Maybe someone told Google I was on holiday tomorrow and needed a nice quiet day to clear my desk :)
Re:It must have been the Klingons (Score:5, Informative)
ack'ed is short for acknowledged, by way of TCP (which sends ACKs and NACKs). In the networking world, saying ACK as shorthand is pretty common.