Google Blames Gmail Troubles On Maintenance Goof

Slatterz writes "Google has apologised for the two-and-a-half-hour Gmail outage on Tuesday morning, and admitted that the cause was down to data center maintenance. 'Lots of people around the world who rely on Gmail were disrupted during their waking and working hours, and we are very sorry. We did everything we could to restore access as soon as possible, and the issue is now resolved,' said Gmail site reliability manager Acacio Cruz in a blog post. Google had been testing new code designed to keep data geographically closer to its owner, which brought about disruption when maintenance in one data center caused another facility to be overloaded. This had a cascade effect, according to Google, and it took the company an hour to get it back under control."

Re:Beta = Test Environment

[sloth jr]

There are all manner of tests, and sooner or later, you do have to test in production. It's important to know that in cloud computing, there are certain kinds of tests that are only possible in production; production load is the surest way to characterize your application and platform. Who knows where in the deployment lifecycle this happened? Someone at Google, certainly, but not us.

Re:Beta = Test Environment

[jadin]

I don't think they are testing it on their corporate users. My domain is signed up for google apps which includes email, but not the pay for premium version. When I read on slashdot that gmail was finally adding an option for 'always use https connection', I looked in the options where people said it would be, and found nothing. Logging into the "official" gmail I was able to find it right away. It took some time before it showed up in my domain's gmail client.

My conclusion is they test all the code on the official gmail users to make sure it's stable enough before updating the corporate clients etc.

Actually another problem may exist...

[Puppet Master]

Not sure how wide spread this is, but I use OpenDNS both at home and at the office as my resolving name servers. Recently some ass hat apparently set gmail.com on OpenDNS's filters. Labeled it as a Webmail client. So, for the past 2 days I couldn't get logged on to my Gmail account while at the office, kept saying login failure. But at home it would work fine. I changed to the company's internal DNS servers for resolving and suddenly my Gmail would connect... So, anyone using OpenDNS and still not able to connect might look into that. I have sent OpenDNS admins a request to re-check that filter... It's kinda pointless to just block everything that someone *thinks* should be blocked.