GMail Experiences Serious Outage 408
JacobSteelsmith was one of many readers to note an ongoing problem with Gmail: "As I type this, GMail is experiencing a major outage. The application status page says there is a problem with GMail affecting a majority of its users. It states a resolution is expected within the next 1.2 hours (no, not a typo on my part). However, email can still be accessed via POP or IMAP, but not, it appears, through an Android device such as the G1." It's also affecting corporate users: Reader David Lechnyr writes "We run a hosted Google Apps system and have been receiving 502 Server Error responses for the past hour. The unusual thing about this is that our Google phone support rep (which paid accounts get) indicated that this outage is also affecting Google employees as well, making it difficult to coordinate."
Indeed (Score:5, Insightful)
So much for handing your email over to Google because it's more reliable than hosting locally...
Not surprising (Score:5, Insightful)
Re:Indeed (Score:5, Insightful)
Re:Indeed (Score:5, Insightful)
It's spotty (Score:5, Insightful)
As I type this, I can get in to GMail just fine, but a friend in Texas can't (I'm in Nevada). Guess Google likes us better.
And kudos to the Google team for updating the status when they say they will. Looks like the script they use automatically puts current time + 1 hour in as the default next update time, and they're posting updates before that expire. Too many times, something simple like that gets overlooked.
Re:Wow (Score:3, Insightful)
Or someone will get congratulated and promoted. It depends on the response to diagnose and fix the issue, whatever it is. Major outages aren't always the fault of some apocryphal guy asleep at the switch.
Re:Indeed (Score:4, Insightful)
I don't know. Our local email has gone down a few times since I've been here, and this is the first I've heard of Gmail being down.
Also our local email search sucks horribly. I can find a trivial personal message from 4 years ago on Gmail in a fraction of the time I can find suddenly-important work email from six months ago, if I find it at all.
Depend on something... pay for admin (Score:5, Insightful)
I think Gmail is a great service for personal accounts...
but for business sorry you need to pay a real live person or support company who will actually be able to deal with your data
how do you get the data out of gmail to switch providers ?
ever serviced a discovery litigation from google ?
(you know where they judge you guilty of you dont come up with the data)
sorry but there is a good reason to keep this stuff on site and working...
regards
John Jones
Eating their own dog food (Score:5, Insightful)
It it's true that this outage is affecting Google too I have to say that is a good thing. Eating your own dog food, product-wise, is always a good idea.
eating your own dogfood (Score:5, Insightful)
Upside: shows confidence in your products; makes it more likely that your engineers will spot problems if they use the software and services themselves; can increase how motivated people are to improve the products
Downside: tainted dogfood kills the engineers who would have investigated the issue
Re:Anti-Slashdot Effect (Score:2, Insightful)
Before more users jump on the useless non-interesting "working for me"/"not working for me" posts, dont do it. No one really cares.
Re:Not surprising (Score:5, Insightful)
I think the real story here is that it outlines the downside to moving everything to The Cloud, as a lot of people are trying to promote these days. As you said, email outages are pretty common even at large enterprises. The difference is, CIOs like to be able to go and yell at someone in their office for an outage, and know that it's being worked on in some measurable fashion. They don't like it when your answer is, "I don't know what's going on. Ask Google."
The Cloud is great, as long as it always works. But, in my experience, downtime is far less tolerated in hosted solutions than it is in on-site infrastructure. And stories like this make executives nervous about this stuff.
Re:Wow (Score:5, Insightful)
Why would you fire the guy who caused it. He would probably be the most carful employee after that. People learn from mistakes firing people even for big mistakes isn't a solid business model and bad HR.
Re:Eating their own dog food (Score:4, Insightful)
It's a good idea as long as your allowed to do something about it. At some companies, you just use it and suffer. From what I've seen, the culture at Google allows people to make contributions (e.g., Labs) to fix things.
Re:Indeed (Score:5, Insightful)
Can you beat Google's uptime? I doubt it. Sure, it's not impossible, but you won't be doing it for less than $50/user?
Re:Depend on something... pay for admin (Score:5, Insightful)
Feel like I'm feeding a troll, but johnjones's ID is so low that I feel this silliness may be taken seriously:
how do you get the data out of gmail to switch providers ?
Same way you would do any remote hosted email migration. POP and IMAP. Additional tools are provided for Google Apps (their for-pay version).
ever serviced a discovery litigation from google ? (you know where they judge you guilty of you dont come up with the data)
sorry but there is a good reason to keep this stuff on site and working...
Umm, an hour of downtime doesn't mean your data is gone. I'll also echo earlier comments -- locally hosted email generally has more problems, as no company but the largest enterprise has the same magnitude of IT equipment and experience as Google.
I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?
Re:Indeed (Score:3, Insightful)
Except that instead of pulling your hair out to get it fixed you get to complain about it on slashdot.
Re:Anti-Slashdot Effect (Score:5, Insightful)
Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.
Hell, even the company I work for has outages for both proactive and reactive maintenance, and that's only for 5000 people.
To say that because you've never had an outage you never will have an outage is absurd.
On top of this, saying that google should "have a backup" is silly. Do you even understand how redundancy works? Do you even understand how web based mail systems work? I really don't think so from this comment. If the error has nothing to do with servers falling over and is an issue with routing then you can have all the redundancy you want, but it won't make a difference.
At this stage it's any comments are merely conjecture, until google make a press release advising of what happened comments like "have a backup" are just troll posts.
Re:Indeed (Score:5, Insightful)
Hi Boss -
I'm the guy that switched our email service to Google. See, it only costs us $50/year/user and this has been the first outage in over year. We used to pay a full time sysadmin to manage the mail server and would average about 12-20 hours of total downtime per year (maintenance, outages, etc.).
Obviously, the switch to Google has been much better for the corporate bottom line. Not to mention that we also get calendaring, wiki/sites, docs, and chat for the same price.
Ah, I'm glad you understand. You have a nice evening too.
Re:Anti-Slashdot Effect (Score:5, Insightful)
Yeah, that's funny. You know what's also funny? The treadmill I bought 3 years ago and never used is in mint condtion. I've never had a problem with it sitting there under the pile of clothes in the corner. I read that 24 Hour Fitness has TONS of problems with their treadmills going down, but mine just keeps going without a single issue. I guess they just bought the wrong brand. Stupid idiots.
Re:Not surprising (Score:4, Insightful)
This is front-page worthy because it let's us all know that we're not losing our frigging individual minds over this, that it IS a collective problem. The fact that google knew what the problem was and fixed it before it had time to hit the frontpage of /. just goes to show that they are trying and they do care.
Personally, this just renews my confidence in Goog, regardless of what the twats are doing inside the beltway...
Re:Depend on something... pay for admin (Score:3, Insightful)
Umm, an hour of downtime doesn't mean your data is gone. I'll also echo earlier comments -- locally hosted email generally has more problems, as no company but the largest enterprise has the same magnitude of IT equipment and experience as Google.
I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?
Could be in part that. Another explanation is that most that work as local IT folks (for any kind of business) know that when anything breaks, its always considered their fault (they are the people-facing shields, not the actual service providers elsewhere). And everything anything remote "breaks", or suffers any kind of troubles THEY will know it (because people will complain to them). Therefore, they both consider remote services less reliable than the average person (they know about more outages) as well as consider them less flexible (they can fix local problems, but are impotent to fix remote ones).
Re:ironically enough... (Score:2, Insightful)
... while the assumption has always been "backups are not necessary with gmail" ...
I really want to know who started that stupid idea. You always want multiple off-site backups of anything important, cloud resources such as gmail and google docs count as one of those sources, you still want others. You also want local copies of any data that you absolutely have to have, what if your local ISP is having troubles and you need to access important documents?
There are a hundred things that could go wrong with using any off-site service as your one and only solution from simple outages and lost passwords to having all your eggs in one basket if someone hacks in or uses elevated privileges to do malicious deeds. The idea that any cloud based resource or offsite backup solution is the be-all, end-all solution for secure data storage is stupid and extremely lazy.
No you don't. (Score:4, Insightful)
but you also have to pay google for the privilege of asking "so whats wrong?
Or, you could NOT pay Google at all, and when something goes wrong, realize that being able to call and ask what is going wrong is not going to get it fixed any sooner, and wait until they fix it.
Having someone soothe you over the phone during the process is a waste of money.
Re:Anti-Slashdot Effect (Score:2, Insightful)
are you really so naive to think we won't know what this means or how to pronounce it? most of the world population grew up on american movies, i'm pretty sure nobody needs such a patronising explanation
Re:Indeed (Score:5, Insightful)
Re:Indeed (Score:3, Insightful)
Why does it matter that the outage affects multiple organizations? If you're studying the reliability of an e-mail service (presumably to decide whether or not you want to invest in a local e-mail infrastructure, or use someone else's), shouldn't you care more about the reliability of that service as provided to you? What's the difference between you having X hours of downtime a year, versus you and 1000 other organizations having X hours of downtime a year? How is the latter worse for you?
Re:Anti-Slashdot Effect (Score:5, Insightful)
Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.
The parent poster's simple little postfix system doesn't NEED to serve millions of people. That's a feature: by not needing the immense complexity that goes along with running a web-based email system serving millions of people, his system is smaller, simpler, and less prone to problems.
It's impressive that Google's Gmail runs as well as it does given its size, but smaller, simpler solutions are almost always preferable. For company email (especially in a small company, not some behemoth company with 100k employees needing lots of mail servers), it simply makes more sense to use a small, simple mail server like the parent's postfix system, rather than to rely on some external vendor's multimillion-user system. Especially since the software needed to run that system is all available for free.
Re:Anti-Slashdot Effect (Score:2, Insightful)
I know you're being sarcastic, but there's a point here:
If you want the most reliable treadmill, it probably makes more sense to buy your own than to use the one at your local gym. Even if they're the same brand and model, yours is going to get much less use, and last far longer than one which is shared by dozens of people every day. Of course, 24 Hour Fitness probably has a dozen of these treadmills so you can switch to another one if one fails, but that's a little different from Gmail which is a single system (running on multiple servers, but still linked together).
It's the same with email: your personal postfix server (say, one which you use to serve email to your 50-employee company) is probably going to have fewer problems than Gmail, simply because it's a much smaller, simpler system (don't forget, a simple postfix server doesn't have to run some fancy web app for people to read their email on, since they use a local email program to download their mails from the server). So the postfix server is probably going to be preferable for uptime and reliability, especially if you put it on a decent server with RAID.
The simple solutions always work best.
Re:Indeed (Score:5, Insightful)
365 * 24 = 8760hrs/year "99.99% uptime" = 8759.124
So to meet 99.99% uptime, you have to have less than 52 minutes of downtime, total, planned and unplanned, in a year. That's really hard. Really. Think about it, few enterprise systems can rarely do that (Peoplesoft update in 50 minutes? HA!). But here, a 1.2hr outage puts them firmly out of the four nines club.
Re:Anti-Slashdot Effect (Score:3, Insightful)
Treadmill?!! (Score:2, Insightful)
Why on Earth would anyone buy a treadmill? Don't you have any streets, parks, ground of any kind where you are? Maybe you should have bought an albatross while you were at it.
Re:Anti-Slashdot Effect (Score:3, Insightful)
it simply makes more sense to use a small, simple mail server like the parent's postfix system, rather than to rely on some external vendor's multimillion-user system. Especially since the software needed to run that system is all available for free.
Have you ever tried setting up one of these things? I am not a novice and I can setup web servers without much headache, but things like postfix and exim are very old and have weird configuration files with their own syntaxes
The howtos you can find are literally hundreds of pages long and mostly outdated.
And if you have everything set up after two days, try adding a spam filter.
Do Programmers Dream of Electric Clouds? (Score:2, Insightful)
Gmail is single point of failure. Your data is stored at one location. When that location is unavailable your data is inaccessible. An opaque, inscutable black box is the correct analogy.
If you were actually applying the principles of cloud computing to email then your data would be replicated among several black box vendors. The outage of a single vendor would not cut off your access.
In short, we dream of the promise of cloud computing but very few true cloud computing services have been implemented. The most conspicuous example of a working cloud computing service is IP packet routing and IMHO it works great.