Why Is Less Than 99.9% Uptime Acceptable? 528
Ian Lamont writes "Telcos, ISPs, mobile phone companies and other communication service providers are known for their complex pricing plans and creative attempts to give less for more. But Larry Borsato asks why we as customers are willing to put up with anything less than 99.999% uptime? That's the gold standard, and one that we are used to thanks to regulated telephone service. When it comes to mobile phone service, cable TV, Internet access, service interruptions are the norm — and everyone seems willing to grin and bear it: 'We're so used cable and satellite television reception problems that we don't even notice them anymore. We know that many of our emails never reach their destination. Mobile phone companies compare who has the fewest dropped calls (after decades of mobile phones, why do we even still have dropped calls?) And the ubiquitous BlackBerry, which is a mission-critical device for millions, has experienced mass outages several times this month. All of these services are unregulated, which means there are no demands on reliability, other than what the marketplace demands.' So here's the question for you: Why does the marketplace demand so little when it comes to these services?"
because they've been conditioned (Score:5, Insightful)
The marketplace has been duped into believing that this is the best technology can provide. People don't have time to know, understand, or research history and find that technology really can be reliable.
I'll get modded troll, but I lay much of this at Microsoft's feet. I laughed them off when I first heard of them and their goal of taking over the industry. After all, I'd been working on systems that ran 24x7 with five-9 reliability for years, and DOS/Windows couldn't touch that.
One time I had an opportunity to visit Microsoft and have lunch with a friend there. I figured while there I'd take the opportunity. I asked them in hushed tones, "Just how do you configure Windows so that you don't have to reboot it all of the time?" They looked at me like I was crazy.
Technology can provide reliability. The general public is no longer even aware that it's possible.
The way it has always been (Score:3, Insightful)
the simple answer - we have more options... (Score:3, Insightful)
Basically, we don't rely so much on a single system that a brief outage can be tolerated because there are alternatives to choose from.
This is also the basis of Clayton Christensen's theories on disruptive innovation - that a consumer of something (technology, etc.) is willing to trade off some of these aspects, like reliability, for cost or performance benefits (however you wish to define those benefits...).
Here's an easy one. (Score:5, Insightful)
Low price or high-quality? (Score:5, Insightful)
We're not talking about software, we're talking about hardware and man-hours. Those will never be free.
Re:because they've been conditioned (Score:5, Insightful)
It's the cost (Score:2, Insightful)
99.999% (5 nines) of reliability is achievable, but it's very expensive and hard to do. Everything has to be redundant, with no single point of failure, everything has to support fail-over seamlessly, the software has to be tested with extreme rigor, and upgrade procedures need to function nearly instantly and support rollback without loss of service.
Not So Simple (Score:5, Insightful)
because 'misson critical' is a myth (Score:5, Insightful)
Re:The cost (Score:3, Insightful)
Simply said, because the equipment isn't/hasn't been able to support it, the only way to build 5 9's or better has been to add more equipment, which increases operations costs, capital costs, etc across the board in an almost linear fashion.
The market has for the most part established the level of service available by establishing the price point the customer is willing to pay for said service.
People love to point towards the big bad telcos and other companies as monopolies and only being concerned about profit margins. They forget that those same profit margins are what drive the company's stock price, in turn causing growth in people's portfolios. It's a vicious cycle and won't end until enough people decide they have enough.
Re:because they've been conditioned (Score:4, Insightful)
Truly, your courage is an inspiration to us all!
In fact, though, I can tell you that in the pre-Windows days, electricity had outages, television had outages, telephone service had outages, gas service had outages... For the same reason we have them today -- people aren't willing to accept the economic and aesthetic costs of providing those services at the level of reliability you and the author are demanding.
Incidentally, is it most people's experience that "We're so used [sic] cable and satellite television reception problems that we don't even notice them anymore"? There were some glitches in a broadcast of Zoolander on TBS last weekend, which I'll admit is cause for complaint. (Especially since one wiped out "I feel like I'm taking crazy pills!") But on the whole, I can't say I've seen substantial problems when there wasn't a blizzard or hurricane, and if I'm forced to to stop watching TV for an hour or two, it's not the end of the world.
Re:because they've been conditioned (Score:3, Insightful)
As is correctly noted above, there are only market pressures involved. When that's the case, customers rarely factor 7 or 8 different metrics (eg. price, quality, reliability etc.) into their decision making. Rather, they identify what they want, then find the cheapest supplier, and provided that there is no compelling reason to avoid the supplier, do the deal.
This means that suppliers concentrate on maintaining enough of a service that they can advertise without being sued, and getting the price down. They have no reason to do any more.
My mobile phone operator gives me a good phone, and cheap calls. But their data charges, and roaming charges are extremely uncompetitive. As data/roaming charges make up a small proportion of my bill, I can't justify prioritizing them when I am shopping around for a contract. I am rewarded with a good old gouging.
At what price? (Score:5, Insightful)
No. They believe it is the best the technology can provide at a given price. Why do people "put up" with cars that only give them X amount of protection in a car crash even though there is technology out there that would make them safer? Because they aren't willing to pay the marginal cost for the extra protection. Arguing about what is possible with technology is pointless. What matters is what a piece of technology can do at a given price.
Everything is a trade-off. The sooner Slashdot learns this the less we will have these stupid "Why don't consumers use the latest, greatest, most expensive technology? We need to force them somehow!" articles.
You don't have to take it anymore (Score:5, Insightful)
When Microsoft decided that I didn't own the rights to my own media and stopped me from being able to copy my own DVDs, I decided to drop them for my media development system and I switched to Linux and Apple. Microsoft doesn't want my business so I went with the people who do. No problem.
When my Long Distance company decided to charge over $1.00 per minute for International calls, I switched to AT&T and their 17 cents a minute program. No problem.
When Frigidaire washers charged extra for the warm water cycle but only give you 5 seconds of hot water and thus, never any, it was no problem to return the unit and buy a different brand. Sure, the salesman wasn't happy but, that is now his problem and not mine. I bought a different brand that did give me what they advertised and promised. No problem.
The list is endless and across all businesses and domains.
The point being is that there are alternatives but, many (or most) people are either too lazy to do anything about it or, like this article, they are too apathetic to do anything about it.
The choice is up to the consumer and, if the consumer would take action, the industry would have to adapt because the market demands it. So far, the market is willing to accept this and thus, the industry sees no reason to change. The less the consumer will accept for their dollar the less they will receive. That, is the problem.
Bingo (Score:5, Insightful)
O RLY? (Score:5, Insightful)
Why? Simple... (Score:2, Insightful)
The same applies to computer engineering. We would easily build a cell phone network that had so many redundancies that it would virtually never go down and would support for thousands of times the expected average load, but we would pay for it in terms of cost. Customers demand reliability. Customers demand affordable cost. What the customer is "willing to accept" is a balance between the two.
Re:because they've been conditioned (Score:5, Insightful)
One time I had an opportunity to visit Microsoft and have lunch with a friend there. I figured while there I'd take the opportunity. I asked them in hushed tones, "Just how do you configure Windows so that you don't have to reboot it all of the time?" They looked at me like I was crazy.
In a certain sense.. you were crazy, at least at Microsoft.
The origins of an OS really show through a lot of the time. Windows started out as a single user OS, so rebooting was OK because the only person you messed up was the guy sitting in front of the screen. It eventually evolved into a multi-user OS, but the "just reboot!" mentality persists to this day.
Linux/Unix on the other hand started out life as a multi-user OS. Rebooting was a big no-no, because you'd affect countless people logged in, and you'd get yelled at for ruining someones work.
It's funny the attitude that comes from the users of each OS. Windows administrators categorically will try rebooting the damn thing first to fix any problem (and it usually works). Linux administrators will only try this as a last resort (and it almost never works).
Anyway, at Microsoft the idea that you can somehow tweak windows just right so rebooting isn't necessary is crazy. They designed the damn thing so "just reboot!" will fix any problem. This of course is an unacceptable solution to a lot of people out their, but for a lot of people it's obviously reality.
Re:Really so common? (Score:3, Insightful)
Hint: Just because you live somewhere without such problems does not mean they don't exist. Ditto for lost e-mail.
Because we are not addicts (Score:-1, Insightful)
But fundamentally, there is another reason, and that is that we are not addicted to connectivity. We have other things to do with our time and we can live with interruptions from time to time. The fact is that such interruptions are usually so short that when we check back with the net after and hour or two of doing something else, everything is back to normal. And occasionally, when an outage is longer, it is fixed the next day unless it is a major earthshaking event like the Asian tsunami a couple of years ago or the Northeast blackout.
In any case, most of the telcos that claim 99.999% uptime don't really achieve this goal. They adjust the figures so that they only measure the best bits of their network and they don't count things like "planned outages". I've worked for telcos for about 10 years so I have seen some of this from the inside.
There is a growing body of evidence that the best way to provide very high levels of uptime, even 99.9999% or better, is to have great diversity and redundancy which most people can achieve by using several different ISPs and types of service. T3 at the office, DSL at home, wifi on the laptop. This doesn't help Crackberry addicts who are hopelessly addicted to the in thing, or iPhone users, but it does help most of us.
Re:because they've been conditioned (Score:5, Insightful)
Re:Here's an easy one. (Score:3, Insightful)
Re:Because we're cheap? (Score:3, Insightful)
Re:The way it has always been (Score:3, Insightful)
No way... (Score:5, Insightful)
I've been in the technology services business for a long time, and with few exceptions, 80%+ customers want their services are delivered as cheaply as possible. Most hospital systems don't even have a 99.999% availability requirement. The 20% the want varying levels of higher than normal availability usually have a government regulation, SLA or other mandate requiring that they do so.
Re:because they've been conditioned (Score:5, Insightful)
Eventually, with all of these little points of failure, you're going to get a good sized chunk of fail. Add in things like the inherent instability of wireless technologies and our nation-wide problem with an aging electrical infrastructure, and you have the sorts of occasionally mildly-inconviencing issues that you see today.
Right now it seems like the things users want to optimize most for are A: speed and B: cost. One day every other month where our home internet is down doesn't seem like the end of the world, especially with the cost of the alternative.
Re:because its ridiculous (Score:3, Insightful)
On the other hand, if google was unavailable for 9.863 seconds per day, every day (which is the equivalent of 1 hour per year), who would care? Just resubmit your query.
What's important about reliability is often not the total downtime but the duration of downtime.
Statutory liability for software defects (Score:3, Insightful)
This reminds me of why Bruce Schneier's dream of legislating liability for software defects is misguided. Sure, statutory liability would make software more reliable, but it would mean that the many who don't need the additional reliability (and currently aren't willing to pay for it) would be forced to subsidize the handful who do. It would also likely claim volunteer-developed software as a casualty.
Re:It's a market-wide problem. (Score:3, Insightful)
It varies by state, but usually it costs 15 dollars to take a company to court, and no lawyers are required. It is generally quick and painless, and people at your local courthouse can fill you in on the details and help you through the process.
Because they're cheap and unobtrusive (Score:3, Insightful)
Re:The way it has always been (Score:3, Insightful)
true for blackberry too (Score:3, Insightful)
Re:At what price? (Score:3, Insightful)
My company offers up to gigabit fiber optic in the city. As you get more into the country areas, you're outside our service coverage, and no ISP will offer that without a HUGE premium. Same goes for cell phone coverage around here, the further you get from the cities, the worse it becomes. You even get less radio stations as you drive further and further out in the country. It's a population density problem, always has been.
Re:What is good enough? (Score:3, Insightful)
Unfortunately for them, I'd be willing to downgrade to 1mbps, but not on the always on, nor the unfettered, and if they do downgrade, I will be readjusting my idea of how much it should cost.
Re:because they've been conditioned (Score:5, Insightful)
This is simply not true.
Yes, it is. People who use Windows, when using Linux, are going to respond exactly the same way to problems - by rebooting.
Anyone that's ever installed software, or run "windows update" knows that rebooting is a very likely part of this process. The dependencies and non-modular approach of Windows are quite apparent. Software vendors say "just reboot" because of all the complexities and dependencies within windows.
No, they do it because it's a simple step for the ignorant end user to understand.
The same simply isn't true for Linux. Replace a critical shared library? No problem, running programs still have a hook to the old version. Any new process that starts will get the new version of the library. Why reload the whole damn OS when restarting a process will do the same thing?
Because for people who don't know that, it's easier to say reboot.
You are conflating knowledgable end users with typical end users. This is at best naive and at worst deliberately deceptive.
You're trying to tell me with a straight face that the BBS market influenced Microsoft? (Which flies in the face of what we've all experienced with Windows).
No, I'm telling you that a random individual's attitude towards rebooting is going to be vastly more influenced by their skill level ad what they're using their computer for than the OS it runs.
No, the reason people have this attitude is because it freaking works.
Exactly. Now, again, why do you think they're going to treat computers any differently ?
I've been administrating Linux machines for 13+ years. I can count on one hand the number of times a reboot solved any problem. The only class of problem this solved is a kernel bug, or the kernel crashing (usually from a hardware problem).
Not done much work with NFS then, I take it ? Or services that have long timeout periods and don't die nicely ?
I struggle to believe anyone has been using Linux for "13+ years" and can only "count on one hand the number of times a reboot solved any problem". Either you've not used Linux anything close to "13+ years" or you've not used it in a very wide range of situations.
Why would anyone reboot without a "good reason"?
The fact that you even need to ask disqualifies you from any useful input to this discussion. Fucking hell. People hit the rest button on their PCs because the monitor power-saving kicked in and for dozens of other reasons that aren't even that good.
The point is that Linux simply has less "good reasons", and requires less reboots. Linux requires FAR less reboots for "patching".
Linux also makes a lot more assumptions about its users (and "users" in this sense reaches from Grandma to software developers).
Wow. Now I know you've really drank the Microsoft kool-aid. Not everyone can afford multiple machine redundancy just to fix the endemic problems of Microsoft who advocate "Just reboot!" to fix so many problems. There's really no reason why I need to reboot just to update what's essentially some new versions of DLLs. The Microsoft architecture is essentially broken if you have to buy another damn machine for the SOLE purpose of maintaining high availability.
Yeah, like I thought. "13+ years" and 12 of those were probably using it on your home PC.
The only meaningful difference between a "reboot" and a hardware failure is the amount of warning. I'll say it again. If your business continuity is vulnerable to individual machine outages (be they from reboots or motherboards going up in smoke), then it's broken. Period. If you can't afford "multiple machine redundancy" then you don't need 24/7 uptime. If you don't need 24/7 uptime, then either scheduled machine reboots (eg: for patching) are irrelevant, or brief outages are acceptable.
Any sysadmin who thinks he can run a high-availability operation without multiple machine redundancy is incompetent. Any sysadmin who is purporting to do so, is grossly negligent. The fact that there's a hell of a lot of people whose Linux (and UNIX in general) bias puts them into these categories, does not make them any less incompetent or negligent.
Re:because they've been conditioned (Score:5, Insightful)
That's three weeks of hard core debugging, tweaking, and hair pulling.
The fact that you were able to wait *three weeks* demonstrates that the problem was, at most, insignificant.
When thousands of dollars (or more) are being lost every minute that a service is unavailable[0], you don't fuck around with idiotic philosophising about how "its UNIX, I shouldn't need to reboot for anything"[1], you just DO IT.
[0] We shall ignore here for a minute the false economy of not just investing in a properly redundant architecture where individual machine outages do not impact availability.
[1] I've been there myself and had arguments with my (at the time) boss about it. It is the difference between how geeks think and how businesspeople think. The geek is interested in figuring out wtf is wrong. The businessman is interested in whether or not his business is still operating.
Even Simpler... (Score:4, Insightful)
99.999% over a year is 31.526 seconds.
No matter how good your staff, no matter how many people you have on site, no matter how robust your systems, no matter how many failsafes you have standing by, ready to be plugged in...
IF something does go down, even the fastest tech on earth is unlikely to identify, pull out, replace and have fired back up whatever the faulty item is in under 30 seconds.
99.999% uptime is essentially fictional. It's simply an impressive sounding number that says, "We'll do everything realistically possible to keep you up 100% of the time. In a typical year, you won't see anything bring you down. You can now tell your investors/clients this and make them feel warm and fuzzy."
It ignores the second part, "But, honestly, if it does go down, we won't have it back within 30 seconds, 100% of the time. Sorry, but welcome to reality. But, for what it's worth, our board's happy to pay you outage fees because it's a small enough risk and the amounts are capped enough, that we're happy to take the risk and costs in exchange for advertising a service we know no one can deliver."
Let's look at regulated phone service, the example in the original post. Can anyone point to a major carrier that hasn't had a major outage at some point? Be it an idiot in a switch room, a power outage affecting a whole side of the country, an anchor ripping up an undersea cable? And how many of them have actually been back within the mandated 30 seconds?
It doesn't happen. That two hour outage is going to take quarter of a millenium of absolutely no more faults to earn back at 30 seconds/year. With luck, it only hit one in 250 customers so you can pretend you're well within your 99.999% uptime but that 1 in 250 isn't really going to agree they got 99.999% after they were down for 1:59:30 more than their contract said they would be.
So, no, 99.999% doesn't exist. It's just a really cool story we tell ourselves whilst being willing to pay whatever the penalties are for missing it, on rare occasions, in exchange for great advertising.
Re:Costs increase geometrically (Score:3, Insightful)
Re:because they've been conditioned (Score:5, Insightful)
This I agree with whole-heartedly. Its a fundamental basis of a market driven economy. Spending effort on things that are too good for the market wastes resources that could be spent elsewhere on items that the market (ie. people) do want. Capitalism does not - and must not - build the best, merely the just barely good enough.
Most people don't give a crap about quality, and if they do then somebody else should pay for it. Its all about the latest and greatest bling and appearing to be better than your neighbours.
So everything we have in our lives - every product, service, and system - is just good enough to work for most of the people most of the time and no more. Our transport largely gets people from A to B (eventually), our health system keeps most people alive a few years longer with not much discomfort, our communications work most of the time for most people in most places, and our politicians mostly look after us OK.
Oh, and most of us do most of our work most of the time when we have to. And no more!
Re:Gas Prices? (Score:2, Insightful)
I wonder how much Exxon and Shell make when we import a barrel of oil from Canada?
http://www.eia.doe.gov/pub/oil_gas/petroleum/data_publications/company_level_imports/current/import.html [doe.gov]
Re:Reality Check (Score:2, Insightful)
Bottom line, things that aren't supposed to happen, do sometimes happen.EXACTLY. That's the part people tend to gloss over. Seeing the latest Southpark episode is not a life or death situation. Likewise, your heart isn't going to explode because you cannot get to Yahoo! immediately.
Trusting one's life to a cell phone is a gamble. While they are a fairly stable technology, there are numerous troubling issues... Batteries don't last forever. Service isn't available everywhere. 911 calls aren't always routed to the most appropriate call center -- although it's much better than in years past. In an accident, your cell phone is just as likely to be damaged as you -- or worse, lost. etc.
Re:because they've been conditioned (Score:2, Insightful)
One other observation is that an individual doesn't directly pay the full cost of his decision to buy crappy software. Sure some buggy software might be "good enough" for him to tolerate in exchange for a cheap price. But suppose he then connects to the public Internet and now his machine gets taken over by a botnet. The rest of society bears the cost of that infected machine spewing spam, DOS attacks, etc., not the guy who bought the crappy software with a security hole the size of Kansas.
Re:because they've been conditioned (Score:3, Insightful)
Thousands of dollars per day? That's all? I work for a web hosting company. When one of our customers' servers goes down for more than 10 minutes, they immediately claim to be losing tens of thousands of dollars per hour.
Re:because they've been conditioned (Score:5, Insightful)
It's not just that the returns are diminishing, they're -NEGATIVE-. It's not just that countries that spend 30-40% less on healthcare compared to USA have similar health and life-expectancy, several of them actually have significantly BETTER results for LESS money.
The reason is basically what you state: Giving EXTREME healthcare to those who already have GOOD healthcare provides little if any benefit, but providing the BASICS to those who are lacking them is cheap and efficient.
So, USA has very very high spendings for those who are "in", but fall quite deeply on the rankings because you fail to provide GOOD healthcare to everyone living in the USA. That's why you're not in the top 40 for any of the most used healthcare-indications despite being undisputed as number one in spendings.
Norway, for example, has similar healthcare to USA, not quite as extreme on the top mainly due to less panic about courts, but still come out way ahead, because healthcare is truly universal.
Costs less, gives more health. What is not to like ?
Re:because they've been conditioned (Score:3, Insightful)
Amen. Hoping for a long, stable uptime on a linux machine that does very intensive and sustained NFS I/O provde to be pipe dream for me. Things did get much better after applying the plethora of nfs.org patches, but you still get some awesome kernel failures.
But I don't care, because I have many machines accessing the NFS mount (mailboxes, btw). I lose one, and keep on ticking. If I lose one machine every 3-4 months for an hour, my service availability stays good - though maybe a bit slow depending on the time of day. I could have mounted the shares on a set of Solaris boxes, but the cost for knowledgeable staff would have been far greater than sticking with Linux. That's right, hardware & software are generally much cheaper than the people to manage it.
So I agree that the original poster's "13+ years" experience with Linux is either a troll, or someone that doesn't have anything but simple use cases for his champion OS.
Individual components of a computing infrastructure will fail. I don't care if it's a $500 compute node, or a $20M disk array. You have to assume that this failure will occur in your design. Availability comes from design with this intent, and that sort of design is expensive.
It always comes down to dollars! As a business, you generally permit the expenditure on highly available design when it has a strong business case. Will you lose $1mil in revenue because of a string of outages? If the probability is very high that the answer is yes, then it would be reasonable to secure $300-400k to remedy the situation. If the highly availabe design costs you $300-400k, but the expected losses from your flaky infrastructure are $50-100k, the money will not be spent, regardless of how many times you whinge on Slashdot.
And so, we accept these average performing, generally there services because we consider it good value for the dollars we pay. We complain that it should be better - sure - but we don't change providers because of the dollars involved.
Re:Partly correct (Score:2, Insightful)
Because it's not worth the enormous cost! (Score:3, Insightful)
99.999% uptime is orders of magnitude more expensive than 99.99%, which in turn is orders of magnitude more expensive than 99.9% uptime, and so on.
The added cost is simply not worth it, in any sense of the word, to the general public.
I, for one, would prefer to deal with a day's worth of power loss in a major storm, than paying 10x as much for my electricity in order to make it bulletproof.
The savings would be better spent elsewhere.
Note that this is not an argument against proper planning and preventative maintenance to REDUCE downtime as much as possible, just an argument against designing everything in the world to survive a nuclear bomb when that level of reliability is simply not worth the cost.
Re:Partly correct (Score:3, Insightful)
Re:because they've been conditioned (Score:3, Insightful)
Plus of course the additional impact upon the environment of the extra energy required to produce and obtain the goods and the waste of failed products. Big profits for corporations and marketers, for which every citizens and future generations pay an extreme price.
Re:because they've been conditioned (Score:5, Insightful)
Why, in general terms, do we redistribute wealth forcibly ?
The short answer is: Because we live in a democracy and the majority of politicians vote in favor of doing that.
The longer answer is; Because living in a stable, healthy population with a safety-net has benefits, even if you're not among the direct recipients of the welfare.
In South-Africa earning $100.000/year means living in a castle surrounded by 10-feet concrete topped with broken glass and barbed wire, surveiled by video-cameras, in a "gated community", driving your kids wherever they need to go for fear of kidnapping and *still* accepting that your odds of being killed by someone desiring your wealth are non-negligible.
In Norway, earning $100.000/year means living wherever the hell you want, surrounded by a garden with strawberries in it, never even having the thougth "kidnapping" cross your mind in relation with your children, posessing no security-camera and indeed unless you live in a major city you'll probably not bother locking the door. Still, even without the precautions, your odds of being killed by someone desiring your wealth is, essentially zero. (more than 2 orders of magnitude lower)
I don't know what that's worth. But it's worth -something-.
I'm much more skeptical of all the corporate welfare, truth be told. If I could directly change what my tax-dollars are used for, my vote would be to cut drastically on subsidizes to dinosaur-industries that are uncompetitive (it's insane that *tobacco*-farmers and coalminers are the two groups receivin the most subsidies in the EU) and to *UP* support of those people who need it the most. Primarily EDUCATION -- I'm the opinion that that is the most sensible support you can give a weak group. It's the only help that can help them with time becoming independent.