Google Apps Suffering Partial Outage 150
First time accepted submitter Landy DeField writes "Tried accessing your Gmail today? You may be faced with 'Temporary Error (500)' error message. Tried to get more detailed information by clicking on the 'Show Detailed Technical Info' link which loads a single line... 'Numeric Code: 5.' Clicked on the App status dashboard link. All were green except for the Admin Control Panel / API. Took a glance 2 minutes ago and now, Google mail and Google Drive are orange and Admin Control Panel / API is red. Look forward to the actual ...'Detailed Technical Info' on what is going on."
The apps dashboard confirms that there is a partial outage of many Google Apps. The Next Web ran a quick article about this, and in the process discovered there was an outage on the same date last year.
Yearly thing (Score:4, Funny)
Re: (Score:1)
Re: (Score:1)
As they teach you in HCI class, consistency is key for an ideal user experience. So this is just another small change by Google to improve usability.
Re: (Score:2)
HCl class? You mean chemistry class? That's like calling biology class 'frog class' isn't it?
Re: (Score:2)
When they can offer uptime comparable to what I already have, I might consider using them for important things. Not until.
Re: (Score:2)
If by "reliable" you mean "no failures for any users anywhere" then sure. If your service has 300 million users, you're going to have downtime for some of them more frequently than if you have 300 users. But if by "reliable" you mean "no failures for a randomly chosen user" then they're already quite good (and technology is just starting to mature in this space - it will get better).
Of course, most of the problems we've seen on /. have been for the free services. I'd hope the paid services are better.
But
Re: (Score:2)
"f by "reliable" you mean "no failures for any users anywhere" then sure. If your service has 300 million users, you're going to have downtime for some of them more frequently than if you have 300 users."
No. By "reliable" I mean "not being down in any given year, for any length of time, for any significant percentage of your user base".
And EVERY major cloud service so far has failed that standard. Every one that I know of: Microsoft Azure, iCloud (which is based on Azure), AWS, Google Apps, Gmail... the list goes on.
Re: (Score:2)
OK, so, what then? I've never worked anywhere that didn't have regular service outages for stuff like mail. The fact that most of the outages were planned doesn't make then any less down. And even the unplanned ones were at least once a year - usually due to being too cheap with the corporate internet connection.
Reliability is easy at very small scale - just run a single server with some OS that doesn't need regular reboots. But the cost per user is remarkably high, in comparison.
Re: (Score:2)
"Reliability is easy at very small scale - just run a single server with some OS that doesn't need regular reboots. But the cost per user is remarkably high, in comparison."
That may be true but it has nothing to do with my point. My point was: until the reliability is up to what I consider to be a reasonable level (and I consider reasonable to be the same as the services that I currently use), then I'm not putting my important business "in the cloud".
It may be that kind of reliability in "the cloud" is prohibitively expensive or difficult right now. I don't care. Unreliable is unreliable.
Re: (Score:2)
With that level of arrogance I can only assume your setup is fatally flawed in some way that you're blind to.
I don't care what your SLA reads, I can assrape you to hell and back if the data is compromised.
Well, having insiders leak information hurts just as much. If your data at rest is both encrypted and on servers that none of your employees have physical or admin access too, I'd rest easy. I'm always amazed that people encrypt some of their customer information, and I'm sure that will continue as data moves to the cloud, but you can't fix people.
I think you've worked for some fucking hacks.
No doubt. But still, that's the norm for corporate
Re: (Score:2)
When they can offer uptime comparable to what I already have, I might consider using them for important things. Not until.
What do you classify as 'comparable'?
gmail outage effecting 0.007% of users (Score:1)
4/17/13 9:09 AM
We are continuing to investigate this issue. We will provide an update by 4/17/13 9:55 AM detailing when we expect to resolve the problem.
This issue is affecting less than 0.007% of the Google Mail user base. The affected users are unable to access Google Mail.
Re:gmail outage effecting 0.007% of users (Score:5, Funny)
So, only James Bond is affected?
Re:gmail outage effecting 0.007% of users (Score:5, Funny)
He has a license to kill -9
Re: (Score:3)
Re: (Score:2)
Four different accounts impacted in California between my wife and myself. Calendars also impacted. Service restored around 8AM pacific.
At some point a backup system is a nice-to-have, but generally Google has given us the quality of service we would have needed to pay about $30k to keep the information in-house and achieve equal reliability. It has also saved us from a few issues in the office where we would have lost connectivity. Instead, we paid $0.
The $30k is not that bad, but the responsibility of
But outages don't happen in the interclouds (Score:2)
Re: (Score:2)
exactly how many are "many", when it's less than 1%?
Re: (Score:2)
Less than 1% of 1%.
Comment removed (Score:5, Insightful)
Re: (Score:2)
I'm sure no cloud contract has a 100% uptime guarantee, but it is "sold" as the perfect solution all the time.
I try to remind people that the cloud is not filled with magic beans. Sometimes it is just what is needed, sometimes it isn't; it depends!
Re: (Score:2)
I'm sure no cloud contract has a 100% uptime guarantee, but it is "sold" as the perfect solution all the time
And that, in a nutshell, is the difference between a signed SLA and marketing.
I'm sure it's inconvenient and unfortunate and maddening to the many affected, but it's not as if your internal fully-owned fully-self-managed infrastructure never have a backhoe network outage or a widely-deployed OS patch-o-death that knocks your operations into a loop for hours or days.
The cloud is just another variant
Re: (Score:2)
Hey, that's what I was thinking this morning!
I've got better things to do than baby servers along, so it sure is nice to have this be someone else's problem. Things were fixed in less than 30 minutes for us. Maybe 6 people at our company (roughly 60 people) even noticed that there was a problem.
And I got 30 minutes without anyone pestering me by email. Glorious.
Can't you guys read? (Score:5, Funny)
They sent an email explaining the cause of the... oh wait.
Re: (Score:2)
Re: (Score:2)
If... your e-mail doesn't work... how would you read the... um, said e-mail?
I tell you, there's MAGIC at work!
But I give you that: it DOES make some sense, because if your web-based e-mail isn't working, maybe the back-end works; maybe you can read e-mails on your mobile device; maybe POP push works, or IMAP, or whatever.
Re: (Score:2)
Two aliens land near a gas pump. One pulls out a laser gun and yells: "Earthling, kneel!" The other alien runs away. Big explosion ensues. The trigger happy alien emerges from the blast quite well smashed and asks the other: "why did you run away?"
The other alien says "when I saw his dick hanging out all the way to the ground and up again and stuck in his ear, I realized these earthlings are pretty tough".
True humor. Not really an element of truth in it. Wow.
Re: (Score:2)
The truth is that, to an alien, a gas pump nozzle and hose inserted in a gas tank might look like a being with his dick hanging out all the way to the ground and up again and stuck in his ear. The second truth is that you went all the way to a bizarre alien joke to try to make your point, and failed in epic proportions.
Re: (Score:2)
Did I at least make you laugh?
Also WHOOOSH.
Really now, you went all the way to analyze a bloody joke just to fuel an argument over the INTERNET? Good god.
Re: (Score:2)
And for the record, I can guarantee you I have an excellent sense of humor as verified by hundreds if not thousands of unsolicited reports from people I encounter from day to day. There is simply nothing funny
Re: (Score:2)
No amount of treatment can fix YOU.
Re: (Score:2)
Re: (Score:2)
Makes sense :)
In the mean time... (Score:3)
In the mean time...I'm working in my desktop machine, saving to my own disk (with automatic backup to my server AND my machine at work) and getting my mail into my own server not depending one ounce on any cloud services. Life is good.
Re:In the mean time... (Score:4, Insightful)
Sounds like you have a comparatively high number of potential points of failure compared to the cloud services.
Re: (Score:3)
Yeah, but the difference is, if something breaks, you can fix it. (Unless the something is your internet connection, I suppose. But then, if your internet connection died, you wouldn't really be able to use services in the cloud, either.)
Re:In the mean time... (Score:5, Insightful)
This is the real benefit with Google Apps. When Google Apps is down...I don't have to do anything! Life is good.
Re:In the mean time... (Score:4, Insightful)
Yeah, but the difference is, if something breaks, you can fix it.
It might make one feel like they're taking a more "active" role in the problem, but you're likely to spend as much time fixing your homegrown solution as Google is fixing Gmail. With the cloud solution when something goes wrong though SOMEONE ELSE fixes it.
Besides - Gmail actually has an "offline" mode available for Chrome users. For those really that worried about downtime they can use that.
Re: (Score:2)
Gmail actually has an "offline" mode
In which you can compose a new email, and look at old email you've received, but exchange no actual email?
Yeah. I know they didn't invent offline mode. But that doesn't change the fact that offline mode in an email program is like no-engine mode in a car. You can sit on a comfy seat out of the rain and listen to the radio, but you're not doing the one thing people actually want a car for: driving.
Re:In the mean time... (Score:5, Insightful)
Yeah, but the difference is, if something breaks, you can fix it.
Alternatively, if something breaks in Google's servers, it gets immediate attention by people who know a hell of a lot more than I do about maintaining a server. And things are multiply redundant, making something breaking comparatively unlikely.
Re: (Score:1)
my servers at home are more stable and reachable that google ;)
Its like the 3rd time in 2 years google fails.
Re: (Score:2)
It's .007% of their users, and last time it was a different small percentage. If you assume all of google is down when 1 in 10000 users are, it sounds bad. The reality of it is that It's not.
Re: (Score:2)
Your servers never need to reboot due to hardware failure or kernel updates? Your internet never goes down? Your power never goes down? Your DNS never gets hosed? You never have storms that knock down power and data lines? Sweet!
I run servers at my house too (for friends, non-commercially) and I'd guess that 100% of my users lose access to their email and web sites for ~8 hours a year, or about 3 9's of reliability. I bet that heads would roll at Google if they had that much unreliability.
I switched m
Re: (Score:1)
.007% is down for one day.
what percentage of in house email users do you think are having trouble today?
Re: (Score:2)
Re: (Score:2)
Have to agree. I work in a fortune 500 company and stuff like email/fileshares/network has outages on at least this kind of frequency, and I imagine it costs way more for a company to do it alone than to outsource it to Google.
I think the Google outages just get way more publicity. If some Exchange outage results in 10% of my company not getting emails for an hour, there is a good chance the other 90% won't hear about it at all, and many of the 10% won't notice either (away in meetings, and of course the
Re: (Score:3)
Google apps is sold with a 99.9% uptime guarantee - that works out to a maximum of 526 minutes downtime per year.
In the last three years that we've been using Google apps, I've never had more than one hour of cumulative downtime in a calendar year. I also haven't spent a single second configuring or monitoring email servers, backing up email data, or with an executive breathing down my neck while I work on a server problem.
I'm pretty happy about that track record.
Re: (Score:2)
Google apps is sold with a 99.9% uptime guarantee - that works out to a maximum of 526 minutes downtime per year.
In the last three years that we've been using Google apps, I've never had more than one hour of cumulative downtime in a calendar year. I also haven't spent a single second configuring or monitoring email servers, backing up email data, or with an executive breathing down my neck while I work on a server problem.
I'm pretty happy about that track record.
This. 1000x this. We've been using Google Apps (paid) for 5 years and I can't remember any significant downtime in that period. We've had more problems with our internet connection than Google problems.
Re: (Score:2)
Yeah, but the difference is, if something breaks, you need to stop whatever your doing to fix it.
It is not always an advantage... and I do prefer to work with my own resources than to rely on 3rd party "service" providers, but there are a lot of cases when choosing a provider is the sane approach.
Re: (Score:2)
I used to feel the same way as you and ran my own mail server. Over time, I realized that I was spending a tremendous amount of time playing email administrator cat and mouse with spammers and script kiddies.
Since I wasn't being paid for any of this, at some point I realized that paying* Google to manage it for me was actually a good deal. They're significantly more reliable than I ever was, it's some other admin frantically fixing problems a
Re: (Score:1)
so in the cloud there is no smtp? no imap? no http? ... do you expect things magically turn into cloud transfer protocol or what?
In the cloud you just have MORE potential points of failure: namely all that you have locally and all that run the cloud server, the loadbalancers, ... which is much more points of failure you can ever grasp.
The diffrence is, who gets blamed. But if your boss doesnt see you working while noone in the company gets any work done, dont expect to get a raise too soon.
On the other hand
Re: (Score:2)
so in the cloud there is no smtp? no imap? no http? ... do you expect things magically turn into cloud transfer protocol or what?
You've pretty much summed up the understanding of every PHB who was ever sold on a cloud solution.
Re: (Score:2)
Re: (Score:2)
Yup, I know the feeling of using my own setup.
Right now, I'm waiting for a shipment from NewEgg that should see if I can recover everything from my last computer. (They seem to die of jealousy whenever I get a replacement, usually without waiting for me to transfer everything over.)
Naturally, the important stuff is backed up (to a cloud service), and I have that, but there's a few weeks of minor stuff I'd like to have back.
Hit the paid accounts (Score:3)
Re: (Score:2)
Re: (Score:2)
So who at your company gets dinged when 50,000 employees are sitting on the hands while Google fixes their problem? Surely some monetary value can be assigned to a loss of productivity this widespread. Or does management take advantage of Google's outage by calling impromptu staff meetings?
Re: (Score:2)
If things were planned the right way, nobody should get dinged. Of course, things are rarely planned the right way.
But let's say they were.
Then there's a set of SLAs which tells the customer "we ensure 99.99% uptime. If we don't, then we deduct X dollars per minute from what we're billing you" or something like that. The customer has their expectations set and when there's an outage, it's accounted for. Furthermore, an SLA involves a theoretical loss of productivity (which is expected).
This "witch hunt" of
Re: (Score:2)
Or does management take advantage of Google's outage by calling impromptu staff meetings?
And how would you notify your workers of these meetings?
Re: (Score:2)
Re: (Score:2)
Re:Hit the paid accounts (Score:5, Informative)
Confirmed here ... it was down for about an hour including the admin control panel.
One nice thing about multi-tennancy is problems get attention immediately.. they simply cannot be ignored.
Re: (Score:2)
Hey, the FBI needed its live Gmail intercept.
Hmm... (Score:2)
Re: (Score:2)
Re: (Score:2)
Turns out it's unlucky to be superstitious!
Rgds
Damon
Re: (Score:1)
Re: (Score:2)
Apparently 17 is considered unlucky in Italy, which didn't mean "Western countries" last time I looked.
Clint Eastwood [wikipedia.org] would likely disagree with that statement.
Re: (Score:3)
4 it's an unkucky number in Japan. 17 it's also an ulucky number in Western countries.
Amusingly, it is for the same reason.
In japanese, 4 is pronounced "shi" in the On reading, which is an homonym of "death".
In latin numerals 17 = XVII, which is an anagram of VIXI, which is latin for "I lived", implying "I'm dead".
Re: (Score:3)
4 it's an unkucky number in Japan. 17 it's also an ulucky number in Western countries. Coincidence? we at TV-Show-On-Whacky-Therories don't think so.
According to the Wikipedia entry for 17:
Described at MIT as 'the least random number', according to hackers' lore.
In Italian culture, the number 17 is considered unlucky.
When viewed as the Roman numeral, XVII, it is then changed anagrammatically to VIXI, which in the Latin language it translates to "I have lived", the perfect implying "My life is over.
The fear of the number 17 is called 'heptadecaphobia' or 'heptakaidekaphobia'.
Re: (Score:2)
Laffo at 17 being an unlucky number in Italy. You have to rearrange the damn letters first to even make the words that are supposed to be "bad."
Man, superstitions are the dumbest thing in so many ways. At least the "shi" thing has some sort of explanation that makes a bit of sense.
Re: (Score:3)
Coincidence? we at TV-Show-On-Whacky-Therories don't think so.
Oh, you're with the History(tm) Channel?
Re: (Score:1)
Re: (Score:2)
unkucky
ulucky
You can do it! Go on, try again, third time's a charm.
Don't ... that's the first two lines of a summoning spell
Started lastnight with youtube (Score:2)
google gmail down for you (Score:1)
tiniest open source violin (Score:2)
The tiniest open-source violin plays for you. (or at least it would if you had a local copy of tiny_opensource_violin.flac)
Glad my laptop isn't dependant on the cloud (Score:2)
Two clouds with replication! (Score:2)
Sorry for advertising my own product, but pretty much on topic here. :) Buy two (cheap) servers from completely different networks / data center providers, and keep them replicated with http://wiki2.dovecot.org/Replication [dovecot.org]. You can set up MX records to both of them, and use DNS to switch between the replicas for IMAP/POP3 as needed. Either one of the data centers can die and your mail won't stop working. Or keep one of the replicas in local network and your mail keeps working even if your internet connectio
Re: (Score:3)
It's been said before and again. You shouldn't rely on the cloud as your sole point of data access.
It's fine for backup and collaboration, but otherwise... I'm always a fan of in-house.
Re:Oh well (Score:4, Insightful)
Because in-house servers NEVER go down.
The only difference between in house and cloud-based email in this case is who the fingers get pointed at when the fecal matter hits the air conveyance device.
Re: (Score:2)
That's not the only difference.
Re: (Score:3)
Re: (Score:3)
It's better, arguably, to have the servers in house because at least you can be seen working feverishly to fix the problems rather than just sitting on your hands telling your boss to be patient.
Behold the tautology! (Score:3)
Re: (Score:2)
It is better to be competent, than incompetent. It is better to have the servers in house if you are competent. Of course, if you are competent, then you already have the servers in house ;-)
To become very very competent, outsource, then insource the same services every few years.
Usually the outsourcing will move you to a cloud based service.
Re: (Score:2)
Re:Oh well (Score:5, Funny)
This is easily solved by keeping an old tower around, prefereably with lots of fans and blinky lights. When something is down, you drag an impressive amount of gear and supplies around it (Mountain Dews, Cheetos, beer, etc.) and look busy until Google figures it out.
They'll never know. If they ask, you are working on the Google 'preprocessor' or something like that.
Re: (Score:3, Interesting)
Oh, and this is why Slashdot needs to keep this sparse UI - from a distance it looks like every other debugger / testing program. Close up it's incoherent garbage which is exactly what a debugging / testing program would look like to a layman.
Re: (Score:2)
dir
I do that and walk away. Text scrolls up the screen. Everyone thinks it's computer magic. I go outside, smoke and chat.
Renaming takes longer...
Re: (Score:2)
Actually... yes.
You'd be amazed how often glitches such as "router at the hotel the boss is staying at is not working properly" becomes "bloody useless IT department, should have outsourced the lot to somewhere with cheaper labour years ago, at least that way I'd be paying for what I'm getting" when the boss can't get at internally-hosted email while on the road.
This doesn't seem to happen anything like as often when the email is outsourced.
Meaning that even if you are regularly providing five-9's - hell, e
Re:Oh well (Score:5, Informative)
We switched to Google Apps a few years ago. In that time I've seen maybe a dozen full or partial outages. Some were not Google's fault. Internet routing or DNS problems were responsible some of the time. One instance was when a drunk driver hit a telephone pole about a quarter of a mile away and severed our fiber connection. When it is down, I still end up spending half the day dealing with the outage. But In a decade of running our email in house, I had just one outage. We did have a few instances of where our Internet connection was down so outside email did not flow, but at least internal communications worked.
Re: (Score:2)
We switched to Google Apps a few years ago. In that time I've seen maybe a dozen full or partial outages. Some were not Google's fault. Internet routing or DNS problems were responsible some of the time. One instance was when a drunk driver hit a telephone pole about a quarter of a mile away and severed our fiber connection. When it is down, I still end up spending half the day dealing with the outage. But In a decade of running our email in house, I had just one outage. We did have a few instances of where our Internet connection was down so outside email did not flow, but at least internal communications worked.
Ok. So if you are running you Google for Business correctly you should still have access to local copies of important documents and critical email accounts should all be going through a local non web based client that downloads emails locally. ...
So how many times in the decade of running you local email did you have issues that made things the same as if Google was down and you were doing things correctly?
What I see there is you saying
Sometimes when Google was down it was stuff that would have effected us
Re: (Score:2)
I never said Google sucks. I'm just pointing out some of the trade-offs. When you go web-based for your groupware application, the Internet at large becomes much more critical to your internal communications, to the point that any Internet disruptions have a large effect on internal communications. That's just the way it is. As I said, I had ONE complete outage with the internal mail server. Sure, we always had periodic Internet outages, but we we are a school with just one location, so we are prett
Re: (Score:2)
The only person the finger should ever get pointed to is the CEO who couldn't justify a colo site for failover cause s/he are too busy stuffing their pockets w that money in the form of "bonuses". Both in-house and cloud suffer from this. But those that really don't want downtime, find it perfectly achievable, though not free.
Re: (Score:2)
Huh. You know that IT folks can make bad decisions without a CEO's help, right? And really, when was the last time you really wanted the CEO making policy decisions for the IT department.
If you are in the unfortunate position of having to justify more spending to mitigate risks, the best policy is usually just to warn the CEO of the risks in writing. Add dollar signs to make sure they are paying attention: "In the event of a failure, recovery costs are likely to excede $###,### per failure. This risk ap
Re: (Score:2)
I'd imagine most CEOs are involved in business continuity plans, which involve disaster recovery, so there's a very good reason CEO's get blamed often times (they're ultimately responsible).
Re: (Score:2)
Never said that. Nice straw man.
However, there are more potential points of failure with the cloud, so a well maintained in-house solution is generally better, because it eliminates many of the external network points of failure. It also allows for multiple points of on-site failover as well as offsite failover (in the case of onsite catastrophe).
Re: (Score:2)
This is still funny!
Re: (Score:2)
... someone just discovered Chobits it seems.