Slashdot Log In
Extended Gmail Outage Frustrates Admins
Posted by
timothy
on Thu Oct 16, 2008 02:40 PM
from the pulling-out-doesn't-sound-manly dept.
from the pulling-out-doesn't-sound-manly dept.
CWmike writes "A prolonged, ongoing Gmail outage has some Google Apps administrators pulling their hair out as their end users, including high-ranking executives, complain loudly while they wait for service to be restored. At about 5 p.m. US Eastern on Wednesday, Google announced that the company was aware of the problem preventing Gmail users from logging into their accounts and that it expected to fix it by 9 p.m. on Thursday. Google offered no explanation of the problem or why it would take it so long to solve the problem, a '502' error when trying to access Gmail. Google said the bug is affecting 'a small number of users,' but that is little comfort for Google Apps administrators. Admin Bill W. posted a desperate message on the forum Thursday morning, saying his company's CEO is steaming about being locked out of his e-mail account since around 4 p.m. on Wednesday. It's not the first Gmail outage. So, will this one prompt calls for a service-level agreement for paying customers? And a more immediate question: Why no Gears for offline Gmail access at very least, Google?"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
The benefits of cloud computing (Score:5, Insightful)
Someone else deals with all the problems, right?
Re:The benefits of cloud computing (Score:5, Funny)
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
That's no excuse. When I need a word processor, I need it NOW, not tomorrow. I do not want my software to be dependent upon anything except my Cl drive. No net connection required.
Parent
Re:The benefits of cloud computing (Score:5, Funny)
FWIW I don't seem to be having any problems getting gmail through my gmail-lite [sourceforge.net] install.
Parent
Re:The benefits of cloud computing (Score:5, Funny)
What's a chlorine drive?
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
Yeah, that's usually what I was saying about email when the exchange server at my previous employer went down every third week or so. Or when some rodent chewed through the line that handled the VOIP. Or when some transformer down the street got plowed by some idiot driver causing sporadic access to the mains for half a day. Word processors in the cloud may not make a ton of sense right now, but email is fundamentally useless without every machine in the chain working properly.
You try running your software without a power connection and let me know how it goes for you. Laptop batteries don't last that long, and desktop UPSs even less so (assume that the generator, if present, can only keep the servers online indefinitely, not the whole building).
Gmail being down for a few hours is a minor inconvenience at worst. If your dirt cheap or free and completely awesome email being unavailable for two hours a year is causing you to lose business, then you seriously need to rethink your operations. You have a landline, a cell phone, and a fax (among others), and if those are all also out of commission then chances are you've got bigger problems. You know, that mushroom cloud hovering overhead.
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
email is fundamentally useless without every machine in the chain working properly.
Ummm, no. SMTP was designed a long time ago when computers & networks were much less reliable than they are now. You can have multiple email servers on multiple internet providers on multiple backbones for outstanding reliability and graceful failover.
Of course, you do have to configure the redundancy - it doesn't magically appear.
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
You've got to love IT, where one-in-a-million events occur every other day...
Parent
Re:The benefits of cloud computing (Score:4, Interesting)
Yes well inside computers, billions of event occur every second. Hell something as insignificant as a cache miss is a one in a million event.
Parent
Re:The benefits of cloud computing (Score:4, Insightful)
I do not want my software to be dependent upon anything except my C: drive. No net connection required.
Different strokes. If you were traveling to see a client and your laptop got stolen, you might see the upside of having your documents online. Viewable and editable whether on a free library terminal or iPod Touch.
I can't count the number of times myself and co-workers have mislaid a USB thumb drive. Can I just VPN in? No, that's why I have the damn thumb drive.
And I use Photoshop, so there's no promise that my client will have a spare terminal in their office with the latest version installed. I guess that's why Adobe is shooting for online apps too. [slashdot.org]
Parent
Re: usb drives, clouds... (Score:4, Interesting)
C. Other.
The online copies are backups. When your laptop went AWOL, you go to some new computer, download them, then do your thing.
Go redundant. When your laptop isn't available, these new phones can sometimes process your actual documents. We're one generation short of proper usability on this front. That will be fixed in about 2 years.
Phone not an option? Get a "disposable desktop". You know, some piece of junk for $100. There's a huge influx of machines due to hit this maturity stage within the same next two years when HeavyOS drives upgrades.
My USB drive is my watch. It's strapped to my arm. So unless I'm a twit and take it off, it's essentially unloseable. Oh look, I lost it. Here's one on my car key chain. Awww, I got mugged. Maybe in 10 years they'll be doing subcutaneous mods. (Gee. My beer belly holds 4 terabytes.)
Parent
Re: Sunny! (Score:4, Funny)
Icarus, is that you falling?
Parent
Re:The benefits of cloud computing (Score:4, Informative)
Parent
Re:The benefits of cloud computing (Score:5, Informative)
Google Apps is a hosted service sold to businesses, Google are meant to provide that redundancy, and in theory they should be in a much better position to provide it than most small to medium size companies' IT departments.
In practice they seem to be sucking a bit.
Parent
Re:The benefits of cloud computing (Score:4, Funny)
Parent
Re:The benefits of cloud computing (Score:5, Interesting)
Parent
Re:The benefits of cloud computing (Score:4, Insightful)
Yes. In your organization how many times have your servers went down or had a problem... Compare that to Google Mail... You will probably find that there is a lot less downtime. The problem is just like flying on an airplane. You are statically safer flying an airplane then driving. However because you fate isn't in your control you feel more scared then if you could just drive there yourself. The same thing with SaaS models, you actually get better service however because you don't have the same amount of control you feel like it is riskier. But it isn't
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
because you don't have the same amount of control you feel like it is riskier. But it isn't
I have a real problem with "cloud computing" and the lack of control is just once piece. With google, there is no assurance that *my* problem is being worked on. *My* problem will get handled in the order in which it was reported. (if at all) To me, MY problem is the most important problem.
The problem with "cloud" computing, and probably the biggest IMHO, is the importance of "you" and your interests to the company providing your service. Suppose that you build your own business on a company providing virtual machine services. All is going well, you are profitable, and poof!! they decide to drop the service because it isn't profitable for them. What if they see what you are doing and say "hey, that's a great business idea, how does he do that, lets look at the code." and so on.
I could go on, but there is a lot to be said about "building" your own business, and my rule of thumb is: "Committing to a single vendor lock-in, in the long run, will always be worse than doing it yourself."
Parent
Re:The benefits of cloud computing (Score:5, Interesting)
With google, there is no assurance that *my* problem is being worked on.
While that is true, it is also true of your electricity and net connection. And any other utilities feeding your building that are critical to your business.
For most businesses, losing email for a little while is nothing compared to a snow day. It just means more telephone calls. And probably more productivity :)
Parent
Re:The benefits of cloud computing (Score:4, Insightful)
If you paid a marginal fee. Then you are a paying customer, and your problem gets priority.
It's not actually that absolute. What is your priority relative to all the other customers who are competing with you for finite rexources? The reality is that more profitable customers may move through the customer support phone queue faster than you, and their issues may be addressed before yours.
Also, it's not uncommon, particularly after a company acquisition, for customers to be reassessed and prioritized according to profitability. Companies decide to cede whole markets to their competitors if they're not sufficiently profitable or they decide to go in a different market direction.
The customer is often not told they are no longer a priority, they just find their rates go up, the quality of their support goes down, their packets are routed through over subscribed network fabric, etc.
The company won't actually tell the customer to go away; they'll continue to accept money until the customer figures it out and goes away on their own.
Parent
Re:The benefits of cloud computing (Score:4, Insightful)
Statistics aren't magical. It's entirely possible that a safe, conscientious driver is safer driving than flying (I haven't seen any statistics which break it down that way before.) There are a whole lot of considerations that need to go into a statistic like that for it to have any real meaning.
Parent
Re:The benefits of cloud computing (Score:4, Informative)
Yet amazingly enough planes crash. They crash quite often in fact. In fact from what I remember the chance to die per hour of travel is roughly the same between airplanes and cars. In other words the chance of dying from some random outside event is probably much higher in an airplane per hour of travel. So yes, a safe driver is much less likely to die in a car than in an airplane.
Parent
Re:The benefits of cloud computing (Score:5, Funny)
You are statically safer flying an airplane then driving
That's true, I always get shocked while riding in cars. Can't remember a time in a plane, though. Must be that the plane is off the "ground".
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
Parent
Re:The benefits of cloud computing (Score:4, Informative)
Sorry, but the total downtime I've ever caused ALL of my employers over my career has been a LOT less than 28 hours! Heck, even if you add up the downtime for all of the single systems I've admined their collective downtime is probably only close to that. I'm not bragging, I'm pointing out how bad of an outage this is. The only other outages I've personally heard of that were this bad are hosting providers who have critical systems physically damaged and a failed Exchange 2000 pilot at Cisco (They had a corrupted datastore that was so bad that MS and HP and EMC couldn't recover it so they had to fall back to a tape restore which took something similar to this gmail outage)
Parent
Re:The benefits of cloud computing (Score:5, Interesting)
Honestly, I've maintained my own mail server for 5 years, and my company's corporate server for 2 years, and I can count on one hand the number of times either of them have failed in that period. When they did fail (because I was being irresponsible about configuration changes, or hardware failures, etc), there was pretty much no way I was going to be getting in bed before I got them back up.
Granted, I don't have millions of users and petabytes of email. But I also am not any kind of real system administrator, I don't have a massive redundant data storage facility, and neither do I have millions of dollars and endlessly brilliant engineers working at my beck and call.
Some GMail downtime is, of course, to be expected. But these kind of high-profile outages from Amazon and Google are truly shocking. I don't think it puts the nail in the coffin of SaaS by any means, but it does indicate a significant necessity of SLAs for paying customers.
I would desperately love to divest myself of the responsibility for these mail servers, but I want to know that I can trust GMail's response time during crisis as much as I can trust my own.
Parent
the "small IT shops are worse than SaaS" BS (Score:5, Interesting)
Yes. In your organization how many times have your servers went down or had a problem... Compare that to Google Mail... You will probably find that there is a lot less downtime.
Bullshit- this is an often-repeated myth that small or medium-sized IT shops can't offer competitive uptimes. It's simply not true- I'm a sole sysadmin, and my server (~200 users) has only had one time when we had an outage , and it took us all of about 15 minutes to fix. We have a number of people who choose to use GMail, and I'm constantly reminding them that they should not be relying on Gmail so much.
The problem is not downtime- it's lack of any way to mitigate the problems, and a complete and total lack of any customer service from Google. There is NOBODY you can call when there's a problem. PERIOD.
Compare and contrast. Google:
Me:
The building I (and the server) are in in could burn to the ground, and I could have us back up in less time than this stupid outage at Google (I'm factoring the time to find/buy two commodity PCs, find/buy compatible tape drive/SCSI card, do an OS install, install the backup server, and fetch the off-site backups from across campus.)
If Google's datacenter burns to the ground, how long do you think you'll be without your GMail account?
Parent
Re:the "small IT shops are worse than SaaS" BS (Score:5, Interesting)
Yeah, got a rude awakening there myself. My gmail account was closed a month ago, no prior warning. No way to get to support. No way to actually contact a human.
I still have absolutely no clue why it was closed. I also lost blog, gallery, docs, calendar, site stats, rss reader, notebook... You get the idea. Luckily I didn't have any serious data there, but still. It's a lot of things I used daily just suddenly gone.
And google's ToS says "We can close an account for any reason. Absolutely any reason. Like, we didn't like the color of the sky today is a good reason." Which isn't helping the issue much either.
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
It most certainly is riskier. If you own your computers & data, and your company goes out of business, you no longer need access to that data. If you SaaS provider goes out of business, you probably still need that data.
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
Bill W. is probably taking heat because he sold google apps to his superiors as having 100% uptime with no disadvantages, which of course it does not.
Parent
Re:The benefits of cloud computing (Score:5, Insightful)
I started out as a personal gmail user. I was very happy with it, even routing my work email through it, but when the question for our mid-sized business came up, "should we outsource our email to gmail?" I said no. I said let's do due diligence, there are other outsourced solutions, this is something we really ought to get right.
Our CEOs (we have two, yeah..) both tried it and liked it, so we went with it.
So I'm in the unique position of having argued to management that we shouldn't risk anything on Gmail, and us doing it anyway because management wanted it. And you know what? I was wrong. Gmail has been a great productivity booster for our business, it's saving us money on salaries, and the downtime is less than we experienced when we were half-assedly running it ourselves.
Plus, when shit does hit, I just smile, and nobody tries to blame me. :-) On the ~two occasions that we had any noticable gmail outage, our CEOs weren't the ones complaining. They have realized that email may be important, but we can still get work done while gmail is futzing around with it.
Parent
Outage Outrage (Score:5, Insightful)
It's a risk you take any time you let someone else handle something for you.
Re: (Score:3)
This is what you get when you pick the cheaper option eh?
Re:Outage Outrage (Score:5, Informative)
That's just it, google DOES charge for the hosted apps version of Gmail. See http://www.google.com/apps/intl/en/business/index.html [google.com] for more info.
Parent
Stallman is laughing (Score:5, Funny)
Is that the sound of cloud computing advocates crying, or the sound of Richard Stallman laughing?
Re:Stallman is laughing (Score:4, Insightful)
No, it's the sound of one hand clapping.
What I mean to say is, what admin HASN'T had an outage like this?
Shit happens. I'd rather get email that works 99% of the time, and when there's a problem, google engineers are dealing with it, leaving me time to work on more important things.
Parent
Re:Stallman is laughing (Score:5, Interesting)
Seriously, here's a link to the Google Apps business page [google.com]. Look around. This isn't free stuff. I'm not sure why you scoff at this and not other business webmail applications. There...do you see that...do you see how Gmail isn't always free?
My point remains...if they paid for it, Google owes them an explanation.
Parent
If you need something done right do it yourself! (Score:5, Insightful)
Re:If you need something done right do it yourself (Score:5, Insightful)
Parent
Re:If you need something done right do it yourself (Score:4, Insightful)
You can't count on the USPS to deliver your mail...sorry buddy.
You can't count on Verizon to run your telecommunications...sorry buddy.
Every service you use was, at one point, decentralized and every large corporation ran it themselves. Then someone did a better job and companies slowly released the reins. Does Verizon's phone service go down? Yep. Does the USPS lose mail? Yep. Goes Google mail go down? Yep. But, in the end we've decided that we'd rather rely on these external services than continue to try to run increasingly large services with ever-diminishing returns for the individual business.
Parent
These guys need a brain transplant... (Score:5, Insightful)
"Admin Bill W. posted a desperate message on the forum Thursday morning, saying his company's CEO is steaming about being locked out of his e-mail account"
Run your own damned mail server if it's THAT IMPORTANT. Seriously, it's not hard to set one up, and you've obviously got the money to do it.
Once again, it's a case of rich people with more money than brains having the problems. Nothing important here, nothing of value lost.
it can go both ways (Score:5, Insightful)
I've been at plenty of places that run their own mailservers where uptime is considerably worse than Gmail's, so it'd be an improvement to offload it. The biggest problem seems to be at medium-sized shops: big enough for there to be problems, but not so big that you have some sort of massively redundant setup with transparent failover and 24/7 staffing. The ideal of the cloud-computing style of outsourcing is that you'd outsource to someone who was big enough to have a massively redundant setup with transparent failover and 24/7 staffing. However Google seems not to have delivered on that ideal.
Parent
Re:These guys need a brain transplant... (Score:5, Insightful)
Run your own damned mail server if it's THAT IMPORTANT. Seriously, it's not hard to set one up, and you've obviously got the money to do it.
Right. Because some nerdy 20-something admin with a copy of "Sendmail for Dummies" can do a much better job than all the engineers at Google.
This is a paid service offered by one of the largest and most knowledgeable technology companies around. They should be able to do a much better job than any internal IT department. There are arguments in favour of doing it yourself, but there are definitely arguments in favour of outsourcing to a competent provider, which Google should be.
This is a PR disaster for Google.
Parent
Beta means never having to say you're sorry. (Score:4, Funny)
Who puts important mailboxes on a beta service? Sheesh.
That's SaaS for you... (Score:4, Insightful)
There is an SLA for paying customers (Score:5, Informative)
Quote from article: So, will this one prompt calls for a service-level agreement for paying customers?
Paying customers of the apps Premium account level DO have a service level agreement.
Free customers do not however which is probably what they were trying to say.
Revised quote: So, will this one prompt calls for a service-level agreement for free customers in addition to paying customers?
From the terms of service for Premier account edition:
http://www.google.com/apps/intl/en/terms/premier_terms.html [google.com]
1.9. *Service Level Agreement*, or *SLA* means the Service Level Agreement located at the following URL: http://www.google.com/a/help/intl/en/admins/sla.html [google.com]
Downtime period is a period of ten consecutive minutes of Downtime
Service Credit is
three days of service added to the end of your term at no charge for monthly uptime percentage between 99.0 and 99.9
seven days for between 99.0 and 95.0
fifteen days for worse than 95.0 uptime percentage.
You must request your service credit. It is not automatic.
Google SLA (Score:5, Informative)
http://www.google.com/apps/intl/en/terms/sla.html [google.com]
There you go, the SLA for Google Apps. It's listed at 99.9%
But... the remedies for them failing that suck, only up to 15 days worth of service per month will be credited.
Also, it costs $50 per user per year
Banner ads to the contrary (Score:5, Funny)
In poignant irony, the banner ad I see above the story is a google ad that reads:
"So why not switch to Google Apps?
We maintain our hosted software 24/7 so you can sleep at night."
Re:You get what you pay for! (Score:5, Informative)
Doesn't anyone RTFA?
$50/yr for each user is not "free". Nor is it in the domain of "you get what you pay for". $50 per user is actually a rather significant sum when we're talking about 100+ user companies.
Parent
Re:Beta? (Score:4, Insightful)
You're paying less than $1 a week per user. If email was important to you you'd be paying a *lot* more than that. Stop being a cheapskate.
Parent