Outages Leave Google Apps Admins In the Hotseat

snydeq writes "This week's Google outages left several Google Apps admins in the lurch — and many of them are second-guessing their advocacy for making the switch to hosted apps, InfoWorld reports. The outages, which affected both Gmail and Apps, 'could serve as a deterrent to some IT and business managers who might not be ready to ditch conventional software packages that are installed on their servers,' according to the article. 'If we began to experience a similar outage more than about two or three business hours per quarter, we'd probably make Google Apps and Gmail a backup solution to a locally hosted mail system, if we used it at all,' said one Apps admin. 'And it would likely be years before we'd try a cloud-based collaborative system again from any vendor.' Coupled with recent Apple and Amazon cloud issues, these Google outages are being viewed by some as big wins for Microsoft."

why "big win" for microsoft ?

isnt there any other vendor out there providing business solutions ? its not like everyone is going to jump into exchange wagon because they couldnt do with google apps. geez.

Re:why "big win" for microsoft ?

I agree. Openoffice is still "locally installable" and 100% free on the applications front. And any business that relies on an outside free webmail service for their corporate email needs is just asking for trouble...loss of the service from time to time is but one of the gotchas.

Cheers,

Re:why "big win" for microsoft ?

Google Apps Premier is not free - it's 50$ per year per account.

I'm using it for my private mail. I like it. But i don't expect 100% uptime - especially for just 50Ã per year per account.

Re:why "big win" for microsoft ?

Precisely. And while I use Gmail most of the time, and the rest of my office use an Exchange server hosted in same said office, guess who has the better uptime...?

That would be me. They frequently (3-4 time a month) loose half a day, as the under resourced, high maintanence, auto-destructing, sorry updated, blackhat honey pot splutters in the corner. I've lost two half days in the however many years I've used Gmail.

Re:why "big win" for microsoft ?

No matter how good the admin is, running it on an old gaming machine provided by the CEOs Son won't give you five nines. You can be happy to get 80%.

How is that the fault of Exchange?

I've backed up our main Exchange engineer for over five years now in an enterprise environment, and out of our 10+ servers I've seen 2 outages. One was due to the system board on the server failing, so that leaves one where Exchange was at fault (one of the databases became corrupt and had to be restored from a backup).

I attribute this to three main factors:

- We run it on enterprise-class hardware.
- Despite rumours to the contrary, most of Microsoft's enterprise-level software is pretty solid, unless it's a 1.0 or 2.0 release.
- Our Exchange implementation was engineered by someone who knew what he was doing, and is now supported by someone who knows what he's doing.

Anyway, this article just makes me more convinced that we've done the right thing by sticking with our own system instead of using a hosted product.

Re:why "big win" for microsoft ?

Because it's a refutation of Google's business model (cloud based, for want of a better way of phrasing it) compared to Microsoft's (locally based tech).

I remain sceptical, as it it would seem that Google would have to be less reliable than local kit in order to make it worth switching back, even before you take into account extra costs for doing it locally. (How much more do you want to spend to get an extra hour per quarter in reliability?)

Nevertheless, IANASA so I don't know the data behind this decision.

Re:why "big win" for microsoft ?

There are only two IT solutions out there in the minds of too many people: Microsoft, and non-Microsoft.

To go with Microsoft is the easy, sure road. It is the standard. It is what is expected, what is known to be safe, what will always work. Any problems you encounter here are met with "well, computers always have problems don't they?"

To go with non-Microsoft is hard and uncertain. It is not expected, nor "the standard", and suspected to be extremely unsafe. The smallest problem will be countered with "you and your stupid ideas. Now go and call LocalRetailerInc for a certified Microsoft solution, and be glad I don't fire your ass over this fiasco!"

Google is not Microsoft, so according to the business logic described above, if it doesn't work the only possible alternative is to use Microsoft.

Re:why "big win" for microsoft ?

Non-Microsoft can be :

• IBM mainframe
• IBM System/i
• IBM AIX
• Sun
• HP
• Solutions using Red Hat or SUSE (however despised they might be)
• I am pretty sure there are other solutions...

I think the basic problem is impatience. I can understand that people want for business purposes something that is quickly implemented, but my experience is that when a Microsoft implementation is chosen, you have two long-term issues : you will time and time again have to solve the same problems over and over, and you can be sure that Microsoft will try to pressure you into upgrades, willing or not.

My experience with Linux and associated programs is this. Over time, everything gets better and better. Sometimes, you might need some time to investigate a problem and solve it, but once solved, it will not recur again (be sure that you have a good system to record such findings, but that would be same when using Microsoft).

I have already three people (not much, yes, but important for me) using Linux : my father, my brother (who shares with my father's PC) and my sister. Unless there is a hardware problem, I can be sure that I do not have to solve software issues on a regular basis, only help them with functional questions : what software to use and how to use it.

They use on a regular basis :

• OpenOffice
• GIMP
• SANE based scanners
• HP Deskjet printing
• Firefox (Iceweasel)
• Evolution and Sylpheed-Claws
• Skype
• Google Earth

I am pretty sure that for most parts of a business, this would be enough.

Now, I think that the usage of Exchange is more of a perception thing, than a real technical obstacle. At my work, Lotus Notes was swapped for Exchange, but I do not consider this a progress, as it reminds me too much of PCTools 4.0 or 5.0 (about 1990) : I really do not see anything innovative in this area (and while some people here seem to loathe Lotus Notes (mostly without any reasons given), it was much faster than Exchange, I find speed very important for computer programs).

Anyone here which as implemented or is using alternatives to Exchange ?

Incredible Expectations

When my boss tells me he wants 0 downtime (or even five-9 downtime), I show him a quote for the 7-figure cost of creating such a system.

Apparently Google is expected to hit that level of uptime all while charging either nothing for their standard edition or $50 per user per year for the premier.

I wonder how much downtime the companies that are using Google Apps would experience if they had to pay for their own redundancy?

Re:Incredible Expectations?

Expecting five-9 or 0 downtime for a system used by only ONE company might be a very high expectation with a high cost vs. usage obtained from it afterwards.

But how many companies rely on Google's systems? When you offer your application or suite to the whole nation or WORLD, and campaign for its use - then YES, you do need to keep a very near-0 downtime to be really successful.

No planned downtime?

we only have one or two unexpected downtimes per year

What about your planned downtime? If you're running Windows, you're rebooting to install patches on a regular basis or you're running unpatched systems. What about software installs?

In the context of the article, do you think the users of Google Apps (or any users) would be happy with, "Oh, no you don't understand. This is PLANNED downtime. This doesn't affect you or our downtime numbers."

you can have 0 unexpected downtime with a single server, if you are lucky.

You can win the lottery too, if you are lucky. How many people win the lottery though?

Google will release app servers

It is not a big win for Microsoft, it is a big win for corps hosting their own app servers. I would think that eventually Google will release google apps on a server that corps could install in their own data centers.

Google's Service Level Agreement

Google has a Service Level Agreement. [google.com] If they have excessive downtime, you can get up to 15 days of free service. No refunds.

Tell that to your boss. It's not your problem. That's what the company signed up for. Welcome to "cloud computing".

Where are the stories about the outage itself?

I scan Slashdot nearly every day and didn't remember seeing anything about outages at Google this past week. A search through the story history confirmed that fact. So I thought I'd visit google.com and see what Google itself had to say. Nothing on the blog; nothing in the press section.

So why is this the first time these outages have been discussed here? From reading the article it appears we're talking about multiple outages over the past couple of weeks. Doing a Google search for "google outages" brings up one blog posting about these recent events. The blog posting includes this unsourced quotation, "Google spokesman Andrew Kovacs said via e-mail that 'a small number' of Gmail users and 'some' Apps users were impacted by the problem, which is still outstanding and being worked on as of 5:30 p.m. U.S. Eastern Time on Friday."

So all these events seem rather shrouded in mystery. How big was the outage? What explanations did Google give for the outage? I've certainly had servers go down, lost network connectivity, etc., etc., but I don't maintain huge server farms with enormous redundancy and multiple high-bandwith connections to the Internet. I don't recall search on Google ever going down; what's up with gmail and Apps?

The suspicious among us might start to think that outside parties might be responsible. After all, if companies start migrating to the "cloud," disrupting those services could have a substantial, economy-wide impact.

I did not even notice it...

We use google Apps for email and to be really honest no one here noticed the issue and I trust the email that could not reach gmail server during the outage will be retransmitted in time. The reliability of our mail server was far worse when we hosted it ourselves, particularly when some of the SEA-ME-WE cables got cut and our provider lost all connectivity for a couple of days. I am certain that if I had big money to waste I could build my own email servers farms and target the five nines but right now we are paying $0 and we are getting a pretty decent service.
Those IT manager using the free service and expecting mission critical uptime should really go out more often and get a grip on reality.
Let's see, to set up my own five/nine email servers I would need at least two hosting location on different backbone, each of them should have at least two redundant servers. And of course I should have one spare that I can ship express whenever one fail.

Fixed Cost (Investment)

• Decent server (RAID, Redudant PS): $3,500x5= $17,500
• Operating system license: RHEL Standard subscription: $799 (optional of course)
• Software license: $0 (sendmail etc..)

Monthly Recurring Cost

• Hosting with decent SLA: $500x2= $1,000
• Email Administrator (IT Admin): $10,000(?)
• Replacement Parts: $100(?)

Implementation time

• 2 to 6 months (including, research, documentation)

Of course I pulled the numbers out of my hat but it should be enough to show that there is no way a SOHO will ever have the mean to do it and that it is unrealistic to expect that kind of service for free or cheap.

The Illusion of Control

We have all seen it. Ebay a couple of years ago going down due to Oracle corruption. Royal Bank of Canada failure due to an improper software upgrade. Now, Google with Gmail and other Google Apps failing. All of these organizations were geared towards having the highest uptimes available and failed spectacularly.

Whether you host your own or use someone else its the illusion of control that somehow clouds our judgment into believing that it would somehow be different if I did it. Example: Is it better to drive or fly? Pure numbers state that its safer to fly on a commercial carrier by an order of magnitude but somehow we feel safer when we drive. Whether we choose to acknowledge it or not the world is full of 6 sigma events. As long as you are doing everything you can and within your budget when your hosting your own apps or auditing your provider to ensure they have, backup systems, redundancy, offsite bunker, etc. then you have done everything you can to prepare for this inevitability.

In a lot of ways designing systems is like playing poker. You can play your hand perfectly, design all the systems redundancy and recovery you like, but sometimes even after all that your opponent (risk) draws a lucky card on the river to beat you. Just because you got beat doesn't mean you shouldn't continue to play the same way, it just means you hit one of those events that you cannot plan.

power

Two weeks ago a transformer blew out in the building I work in. First there was no power for 3 hours, then temporary power as a large generator was hooked up, but it was not big enough to run the AC, so we did no turn on the servers. It took another day to get a large enough generator (about the size of a tractor trailer). In total, our business was shut down completely for a day and a half due.

I don't think you can even get a SLA from the power company.

Google Apps went down for 3 hours.

Shit happens.

Rethinking Google

We ran into one of these "gotcha" features in hosted Gmail that's been giving me fits and it all started with a simple mistake. I misspelled a user name. You can change the spelling in the admin module, but it doesn't change the spelling in the contacts and the misspelling still showed up when she logged in. So I tried deleting the user name and recreating the account.

Big mistake.

When you delete a user name you can't recycle it for five days, which pushed us past our roll out date. Their crip work-around is creating a mailing list with that user name. But that has its own set of problems, especially when trying to migrate a large number of users. There's no support unless you get the premium edition. So now we're stuck in the position of paying for support on a service we're not certain will work for us. I'm not inclined to throw money at something to see if it will work when what we're already paying for is working.

Unfortunately, it was one of our key sales people who already had that account name on her business cards. Rolling without her is a non-starter.

It's frustrating because I'm the one who recommended Google and I feel really let down. It's a stupid problem that shouldn't exist in the first place. Even if there's a good reason for it, there should be a giant warning banner with a flashing red neon border warning you that deleting a user results in a five day lock out. Actually, it's been more than five days and I still can't recreate the account.

This one niggling little incident is making me rethink hosted applications. So, yeah, it does sort of benefit MS. Not in our case, we're using hosted SendMail instead of Exchange, but if this type of "feature" deters other companies already using MS solutions, then yeah. Who wants to take a chance on looking bad? There will still be outages with any solution but no one gets fired for recommending MSFT. There's a certain period of time that users are looking for an excuse not to like a new service, just because it's different. If you can get past that time frame, then a small outage can be overlooked. But those first few months have to be smooth. Maybe not flawless, but close to it.

It would almost be better if the free version was a trial and corporate users could get support from day one. This is just maddening. Shape up, Google.

Give me a break

In my company Google Apps is the most reliable thing we use. Microsoft products are my biggest headache. We have clients that need their work done and I don't have any more time to waste on these crappy machines. We will be switching to Apple for all mission-critical machines in the next three weeks.

If my MS computers could have only 3 hours of downtime a quarter I would be really happy. I used to work for an IT company and they primarily used MS servers for their clients. Big mistake. MS products are a nightmare. Their clients would have been happy with 3 hours of downtime instead of days and days down dealing with MS server issues. I would only avoid cloud computing if there were serious concerns with privacy or hacking.

The uptime percentage is very clear...

The sign-up page for Google Apps Premier says you get 99.9% uptime. That's about 1/3 of a day downtime per year, or a couple of hours per quarter.

Google seems to be managing to hit that 99.9% uptime, just not exceed it. VERY few in-house e-mail systems actually manage 99.9% uptime, especially when you consider scheduled maintenance and downtime (remember, Google's 99.9% is for all downtime)

In fact, I have seen very few Exchange systems that manage much more than 99% uptime. However, for those organizations, there are other compelling advantages to Exchange.

Why blame Google ?

I love being the asshole, but let's be honest here: how many in-house systems actually deliver better uptime than Google ?

Not that many. If they did, all us sysadmins would be out of a job. Apps are not perfect. The fact that you can pay Google a few pennies to manage your email, even with some downtime, makes it several orders of magnitude cheaper than an in-house solution for most people.

Give them a break, people can survive without email for a few hours.

You can't do it better than Google

Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible. I know how much it sucks when your system is down and there's nothing you can do but wait on some status dashboard to from Red to Green. That said, we should recognize that while being frustrated at this lack of control is normal, that doesn't mean you actually could do it better. It's easy to say "this would have never happened if we were self-hosted" while never thinking about the bullets you dodged by running hosted applications.

That means you, as a single customer, are insignificant. And that shows daily when dealing with any large service provider.

The only thing that my service provider should care about is the availability of the platform. I am completely insignificant, but the only reason my hosted app would be down is if the platform is down, and that sure as hell is significant to them. The advantage of hosted applications and cloud computing is that no one needs to ever look at or touch my app, the platform is all that matters.

Re:You can't do it better than Google (no troll)

Our mail platform has beaten google in uptime and security "bugs" for the past 40 months. Why? I attribute it to using proven technologies and not everyone wanting an account being able to get one: we charge every system user. You would be surprised how much this cuts down on spammers/excessive usage.

Google has had their mail in beta for years. The last time I checked SMTP was ratified as an RFC over a decade ago.

Re:You can't do it better than Google

Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible.

Do I believe it? You betcha! While my company doesn't have 100% uptime for every employee all the time, we haven't suffered an across-the-board outage of a critical system (i.e. email, ERP, core business applications, etc.) in the 11 years that I've been here. Sure, we'll lose an email server once in a while, but we have many such servers, so the loss of a single system only impacts a few thousand employees tops. That's far better than impacting ALL our employees if Google has an outage. And don't get me started about the idea of not being able to do word processing just because a WAN link is down. How on earth could you run a business that way???

And it's possible to provide uptime even in the event of widespread events, such as flooding, tornadoes, etc. We have multiple datacenters, geographically dispersed. Each center has multiple Internet connections through multiple providers carefully chosen such that the lines go to different cities (i.e. one link to Chicago, one to Denver). Similarly, our power is connected to multiple grids, with the feeds coming in on opposite ends of the buildings. Critical centers have on-site generators spec'd to handle 100% load of the datacenter and requisite support stations, plus enough battery backup to allow for all systems to continue running between loss of grid feed and when the generators are spun up, not to mention on-site diesel sufficient for several days of operations and contracts to get more as needed.

Was this cheap? Not in the least. Was it worth it? Definitely. We kept our main datacenter running without interruption during a week that saw multiple weather events (i.e. tornadoes, flooding, lightning-related power loss, etc.) when every building around ours for multiple miles was without power.

Re:Networks crash just like software

Another issue is web/network attacks. They are going up big time and are even state-sponsored. Look at what Russia is, and has been doing to Georgia.

I don't understand how anyone in this day and age can justify going with remotely-hosted applications. The ability to reach remote servers can be taken away even by morons and botnets who might not like your company.

In my opinion, remote web hosting of applications that are presumably important for a company to be able to run is just asking for trouble. I wonder how many fingers will get pointed when some critical deadline looms and nobody can run their applications to be able to meet it.

It's reckless and risky for business to expose themselves like that. As others have pointed out, OpenOffice is free and it is good. Why waste money on training people on both the Google (or other) remotely-hosted application and OpenOffice (if that is your emergency backup). Just train people on OpenOffice and now you don't need a backup plan in case the network goes down and you can't run the remote stuff.

Remote applications may have been a solution before the Internet got nasty but these days, running business-critical stuff over it when you don't need to does not make sense to me.

Maybe I'm missing the huge economic advantages that justify the unknown and growing risk, but I see network (Internet) applications as being at huge risk for outages, a security risk, a data privacy risk, etc.