Half of All Data Centers Understaffed 211
alphadogg writes "Fifty percent of IT executives say their data centers are understaffed, and companies are still looking for more ways to cut costs, according to Symantec's latest 'State of the Data Center' report. Sixteen percent of survey respondents said their data centers are extremely understaffed, and another 34% called their data centers somewhat understaffed. At the same time, data centers are becoming more complex and harder to manage, with more applications, data and increasingly demanding service-level agreements. 'Data center complexity has led to a lot of staffing challenges,' says Sean Derrington, director of storage management and high availability at Symantec."
12 hour shiths are not the ansaser (Score:5, Insightful)
12 hour shifts are not the answer as well makeing people work every weekend holiday night while the boss / PHB never does any of that.
Would this be a good time for a union? (Score:5, Insightful)
> 50 % understaffed, 16 % seriously.
So how many of you have to answer your blackberries after work?
Is this not the kind of situation that a Union would prevent?
(just an honest question btw, I'm not trying to troll)
Should this be surprising? (Score:5, Insightful)
Does this really surprise anyone?
Many data centers these days are no longer run by engineers or technologists, who have at least some idea regarding the technical aspects of the operation. Rather, many of them are run by people who received their higher education in finance, commerce, accounting, "business" or (perhaps worst of all) even marketing.
Of course, such people have a very hard time seeing beyond the numbers, since they usually have absolutely no understanding of technology, nor what it takes to truly run an effective data center. They insist that the current number of staff are sufficient, even when they clearly aren't, and even when they could easily afford to hire more employees.
I think this just reflects a greater problem of the American corporate society as a whole. People with actual technical knowledge in a specific field get pushed out in favor of people with meaningless MBAs (but all of the right "connections"). So it's no wonder American productivity and competitiveness is grinding to a halt.
Other areas of the world, namely Asia, India and Eastern Europe, realize that it isn't the accountants and financiers who provide productivity, but rather the engineers, scientists and technologists. That's why they can build better cars at a far lower cost than their American competitors can, for example. That's why Korea and Japan have broadband networks that put to complete shame anything in America.
What is "understaffed" (Score:5, Insightful)
I believe understaffed means no one (in the US) is replying to the following ad:
Want to hire data center cat5 cable install tech, mandatory 60 hr week overtime, weekend 2nd 3rd shift and holidays required, require CCIE, MBA, at least masters level degree (prefer phd), minimum ten years experience with "windows server 2008R2" yearly salary $25K/yr no benefits.
Golly, we got us a shortage, best open the H1B floodgates!!!!!!!!!!!!!!!!!!
Re:Should this be surprising? (Score:5, Insightful)
I don't think that it's a big deal that people with diverse backgrounds get into IT. Either they are competent or they are not, and there's no reason someone in finance can't become competent in IT and switch careers.
No problem, but put them at the end of the very long line of folks whom already know what they're doing.
Re:Would this be a good time for a union? (Score:3, Insightful)
You aren't trying to troll and neither am I. It IS the kind of situation a union would prevent, however considering everything else that has been done for union's sake lately (see: destruction of US auto industry) I would suggest you take the unionization decision VERY seriously. How exactly, considering that funding isn't sufficient for staffing at the current expense, do you expect companies to afford to bankroll a union AND get more staff to man the servers? In all likelihood you will end up with lower pay and more work; but hey at least you will have a contract!
In all fairness, (not trying to troll, honest) unions aren't for educated workers who can make rational decisions. Unions were invented to protect unsuspecting workers from manipulative business owners, when the education gap was huge. Now, you probably have a very comparable education to your boss, and probably to his boss and most of the rest of the organization. You are smart, start making your own decisions.
You know what else would prevent you from having to take work calls after hours? Stand up, tell your boss you won't give up your personal time anymore, and let him fix the situation or fire you. Presto, no more late nights!
Re:Should this be surprising? (Score:2, Insightful)
That's why they can build better cars at a far lower cost than their American competitors can, for example.
Ahh I was somehow under the false impression that they were able to make cheaper cars due to lower wages, less environmental regulations, and the lack of labor unions.
Re:Would this be a good time for a union? (Score:5, Insightful)
That's going to be very dependent on the union. (Devil's always in the details.) Many IT folks still have the free-wheeling "just get out of my way & I'll get this fixed" attitude, and in those cases union interference in their work will not be welcomed.
Basically, a collective bargaining agreement is one thing...having someone outside the organization set the bounds of your job (and set limits on how you can be promoted, or which incompetent f-up can be fired) is quite another. I won't say a union is impossible, but it probably wouldn't be one of the big names.
Re:Should this be surprising? (Score:4, Insightful)
All of this being said: a data center is technology, and technology is a mystery. To top it off, it's not getting any easier to understand. "Cloud computing? What's that?" Says the old CO who still uses an AOL account
Bottomline: spending money on tech is always something the big brass knows they have to do, but do so begrudgingly....
What they say VS what they do (Score:4, Insightful)
If you want to get a true picture of life in a data centre look at what the management actually do, what they spend money on and what they produce. If you rely on the answers they give you'll end up broke very quickly. The only way to tell if datacentres really are understaffed is if they start hiring more people: any other action just shows the lie in their responses.
When managers say they need more staff, they generally mean they need more cheap staff (often to replace the expensive staff they already have). They could always fill any critical needs very quickly by offering more financial incentives (the only ones that really mean anything), but this almost never happens. Somehow they manage to bumble on with their "staff shortages" and still meet their targets.
Re:12 hour shiths are not the ansaser (Score:4, Insightful)
12 hour shifts are not so bad if you only work three or four days a week, alternating every other week.
Re:One small part of the study (Score:3, Insightful)
Not really surprising; they fall into the range of companies who tend to have enough money to invest in new tech but lack the corporate clusterfuck that stops them from achieving any kind of change.
Re:Whatever. (Score:4, Insightful)
It's attitudes like that why wages stagnate. Gonna get flamebait for this, but what happened to the yankee spirit? The Founders would puke at the current complacency.
Perception (Score:3, Insightful)
Re:Should this be surprising? (Score:3, Insightful)
If your company never got itself into those, costs are lower. Otherwise you might find that one worker has to be productive enough to pay for 2 retirees, (as well as the CEO's cut
Re:Would this be a good time for a union? (Score:2, Insightful)
The problems with Unions is not that they exist, but that they forgot what their job was. Their job is to protect workers from employer excess. For instance working 80+ hours a week for very little pay and no overtime. And keeping working conditions safe instead of letting safety go to increase profit, because the lawsuit would cost less then the safety measure. The problems with unions is that they forgot that and started focusing on pay and benefits. Essentially robbing Peter to pay Paul. And in the process hurting the employer by creating pay structures and job restrictions that are not sustainable. However, upper management did the same thing by paying top executives way more then they were worth. If executives were not paid 300x more then employees the company would have more money to properly staff. So unions needed to protect employees from the employers, but needed to protect employers from themselves.
Re:In other news... (Score:4, Insightful)
Re:Should this be surprising? (Score:5, Insightful)
Actually it only takes about $2K of labor to build all cars and trucks
That's probably true of most things/services. There is an amazing amount of "friction"(ie: added cost) from all levels of management, marketing, etc. Some of it is necessary, a lot of it is not. It's strange that the people you are 100% sure you need(engineers/builders) are often at the bottom of the salary food chain.
Re:Whatever. (Score:1, Insightful)
I can't believe your were modded up.
He was modded up because he's absolutely right.
America was not built with "outsourcing" in mind.
When the going get's tough, you get off your ass and get tougher!
Re:Would this be a good time for a union? (Score:4, Insightful)
however considering everything else that has been done for union's sake lately (see: destruction of US auto industry) I would suggest you take the unionization decision VERY seriously.
Hahaha. As much as I dislike unions, the destruction of US auto industry was caused by complacent & incompetent US auto industry management.
The US auto industry kept designing & building cars at a price point that few people wanted to buy. Simply put, foreign car companies (on average) made better, more reliable cars.
The union wanted better salary & benefits for their members (entirely understandable, we all want to make more money). But if management agreed to ridiculous levels of compensation, to the point where the business is no longer viable, then that is the fault of management for making stupid decisions.
Re:12 hour shiths are not the ansaser (Score:3, Insightful)
100% of all IT jobs understaffed.
Methinks it is about time we got a professional body (or for those so inclined a union). They would set things like standards, work requirements, exams to work in a data center, and of course we can use it to make sure job stay local as the other professions do. I mean how can you trust your data to a non-professional data center. I mean, do you trust people to manage their own medicines?
I say this only have cynically. If you can't beat em, join em. We have to stop pretending we live in a free-market and use the government like everyone else to protect our turf... all in the name of benefiting society... of course.
Re:Should this be surprising? (Score:3, Insightful)
That's why they can build better cars at a far lower cost than their American competitors can, for example.
Ahh I was somehow under the false impression that they were able to make cheaper cars due to lower wages, less environmental regulations, and the lack of labor unions.
In Japan and South Korea? Are you joking? These countries are the very essence of technology-driven.
Re:The other half (Score:3, Insightful)
Server OS is not the only thing in the datacenter that needs staffing. Facilities work (cabling, power, cooling, etc), SAN, Network infrastructure, and that's without even getting into the middleware or applications themselves.
Even if your base servers administered themselves, it still takes quite a staff to actually do something with those servers.
Re:Should this be surprising? (Score:3, Insightful)
To be honest, Ford is the least incompetent and corrupt US auto manufacturer by a rather long stretch. Of course they're not really entirely a US auto manufacturer anymore either, but that's really beside the point.
Unions have certainly gone too far. Particularly in regards to the ratios of show stewards(I think that's the term) to actual workers. In some places it got as bad as a two to one ratio, so a total of 1/3 of the people who were actually supposed to be doing things were useless, not even counting all the usual dead wood. That said though, management incompetence is still one of the top three reasons companies like this go down.
I don't know if a union is really the answer in IT, or in any professional job for that matter, but that doesn't mean that you have to bend over and take it. IT skills are really something you can learn or something you can't, and to be honest there aren't really all that many of us in the "can" pile and not all of us end up in IT. Just because your job could be filled by some idiot who paid 5 grand for someone to give him a few useless certifications doesn't mean that that idiot can actually do your job. Competent people are actually fairly rare, that's why you end up answering calls at 3 am in the first place. Certainly some degree of out of hours work is part of doing a support job, but if you're working 80 hours a week and getting paid for 40, you're probably making such a low hourly wage that you don't really have anything much to lose, even in this economy.
Re:What is "understaffed" (Score:2, Insightful)
Re:Should this be surprising? (Score:3, Insightful)
You're confusing Sweden with China. In Sweden (and most of Western Europe) environmental regulation is actually tougher than they are in the U.S. And wages are not that far behind ours.
Funny you should make that mistake when all the right wingnuts are making so much noise about the imaginary conspiracy to turn the U.S. into a "European Socialist" economy that can't compete at all:
http://www.nytimes.com/2010/01/11/opinion/11krugman.html?em [nytimes.com]
Re:Should this be surprising? (Score:2, Insightful)
"I don't know if a union is really the answer in IT, or in any professional job for that matter"
I think that's where your making your mistake. Trying to put my substantial ego aside, the business is trying there damnedest to make Datacenter IT folk a commodity, and it's working. We're decidedly not unskilled laborers but in most cases, no matter what we want to beleave, we can be replaced without a big impact to the bottom line. You are not a beutifull and unique snowflake. There is a substantial part of Systems Administration work that CAN be done from halfway across the globe, or by the guy in the cube next to you (or who's resume just hit the boss' desk).
Part of this comiditization is eliminating overtime pay, not respecting personal time and expecting we're available 24x7x365 for whatever whim management has.
A union, or similer group, are the collective bargening leverage that would make my (& your?) personal time a thing the company must value. Forcing the employer to have to take into account when the CIO is playing Veruca Salt on a Saterday "I WANT IT NOW!" for the project he'll end up canceling by Tuesday.
I'm not a fan of unions, don't get me wrong, but I think the pendulum is swinging too far in the directions of a Dickens novel these days & rugged individualism isn't fixing it.
Yeah, blame the unions. (Score:4, Insightful)
Like if producing gas guzzlers that are inefficient and brake easily is the fault of the unions.
I thought that the geniuses commanding those huge bonuses, golden hand shakes and parachutes were the ones dictating corporate policy.
But hey, whatever rocks your boat matey.
Re:In other news... (Score:2, Insightful)
Very, Very few Center Managers actually performed any kind of statistical process control analysis for quality in the datacenters I worked for which were huge and did work for .gov, finance and top 500 and they barely did it. They eventually fired the poor guy as he kept proving management wrong. We had long conversations that helped me understand technology for what it was: "La Technique: L'enjeu du siècle" was an eye opener.
Very few managers understood what project management & change windows were in a datacenter and usually managed to a staffing model which could only break in times of heavy load, inducing a bigger emergency later on.
Really management held the mentality of "the Maytag repair man" is who they need to hire by the business plan but the reality was a team of MacGuyver's were needed for the workload over several clients.
Even educating them on vendor patch cycles and technology refresh could not break them out of scarcity = profit mentality.
The admin team experienced over and over that that next month is the month where Microsoft is going to pound us with new critical patches. Despite explaining this, management also put the work of putting unrelated project X into motion or finishing that same next month.
Extensive studies about patch cycles or changes (failed, back out, succeeded) take a long view - an X bar control chart shows spikes and abnormals. It then takes some analysis to then determine staffing levels that can handle the work on average. It takes a huge business insight to understand why something fell outside the norm and how to handle it when it comes again. Really I don't think the technology today can be managed to wholly eliminate outlying events like traditional manufacturing processes. The now typical 3 year tech cycle prevents such work.
It is true statistical process control can be made to lie but it is better than uneducated guessing.
To me datacenters are huge machines you can walk inside of, really no different than megawatt power generation or pharmaceutical manufacturing and ought to be better managed. I say this in that when they go offline or do not function well an aspect of human safety and productivity is jeopardized.
My favorite manager who I worked for briefly before he retired said, "Cowboy managers and admins have no place in my shop. I want science put back into Computer Science, don't snow me with new technology."
I agree with him.
Shifts and other professionals (Score:3, Insightful)
I agree with the statement but not the reason.
Doctors, Nurses, Medical Staff, Police, Fire/rescue often work 12 hour shifts and holiday.
However those professions realize and have by experience been bitten by the consequences which aid them in helping the professional know their limits and the limits of their peers.
First these professionals make mistakes during the day. More so when overtired, Even more so when out of their normal sleep pattern. Technology professionals somehow ignore this and think they are superhuman and often promote this.
"Oh I stayed up all night to fix your server!" Pat me on the back! While probably true, I don't want to hear that sentimentality from my admin. It meant something went horribly wrong and I don't want it to ever happen again.
Doctors, etc as cited know that they would perform in a diminished capacity the next day and not schedule surgery and/or the hospital management would know to give them a resting day as the liability of mistake be too great. Safety services know that some other station has to possibly cover a crew that just came off a fire/rescue and be very wary to send the same crew back in. Technology companies ignore this to their own embarrassment which is justly earned.
Second doing business changes (minor or major) on weekends or holiday nights is _bad business_ in that it demonstrates the fragility and unreliability to which they do not admit to customers. Why not do the same operation during normal hours?
Would anyone take their business' truck to the car mechanic for an Oil change and accept, "well we have to do it between 3am and 4am so as not to impact your business."
But it's an OIL change, it happens frequently, everyone ought to expect it to happen! This is exactly the same to me as a minor patch, price lists, firewall rules, and application rules for business policy. Such ones are expected, frequent and shouldn't have to be done like as they are now at a forsaken hour in the morning.
The more complex example is "Oh the engine overhaul is going to be b/w 3 and 4am" - I would say give me another truck that does the same thing and I'll be back after you fix my truck during the day when your awake. The analog is the system upgrade. Providers go into fits -" but but your system was so tweaked, We can't simply move it to another CPU", etc... Blah. Its because most centers don't know how to offer a real solution.
IT Professionals ought to advance the profession and figure out why they are working 12 hour shifts and holidays and then systematically eliminate these events as much as possible till only having to do so when a human life or safety systems is jeopardized.
Why IT professionals are not publicly beating up IT vendors for poorly written OS, barely redundant equipment, poorly designed apps, etc, is beyond the scope here.
Who is going to be the next Ralpf Nader, who will write "Unsafe at Any Speed" for the IT industry/Computing Science.
Re:12 hour shiths are not the ansaser (Score:4, Insightful)
Re:Would this be a good time for a union? (Score:2, Insightful)
Seriously, you just copied and pasted that off the UAW web site, right?
Nope. The decline of the big 3 has been going on for decades.
Because the Unions in the Auto Plants forced the automakers to become very inflexible and ossified in how they conducted operations. Any minor change in process required expensive retraining, and lord help them if a new process made it possible for something to be done with fewer employees.
And who agreed to the union demands? Management. Look, if the union can get management to agree to something like "jobs bank" [wikipedia.org] where thousands of employees are paid not to work, good for them. Management didn't have to agree to that kind of stupidity. Management didn't have to structure their operations so that a targeted strike at a small parts plant brings their operations to a standstill.
Maybe the big 3 should hire someone from Walmart to teach them about union relations.
And frankly, the actual labor cost difference between domestic/foreign cars is not that big. The fundamental problem is that foreign automakers make BETTER cars that more people want to buy.
Re:Get Your History Right (Score:2, Insightful)
Re:Get Your History Right (Score:3, Insightful)
I see your point, but there's a flip side to that argument. Just wait until you get older and have to find work.
The classical career arc that was in place from the late 40s until maybe the mid to late 90s went like this for college graduates:
- Graduate high school
- Graduate college
- Get a company to hire you in some kind of traineeship
- Work your entire 40-year career for the same or related companies, with a series of progressively responsible positions that were designed to meet your need for more income and responsibility
- Make progressively more money in a safe job, and not be afraid to do things like buy a house or car.
- Get a gold watch and retire with full benefits and heartfelt thanks from a company who was happy to have the institutional knowledge preserved for that long.
For non-college graduates, you could get a job in a factory, work your butt off for 40 years and still come out OK due to union negotiation of wages and benefits. A bachelors' degree wasn't a mandatory admission ticket to a job, so fewer people were burdened by student loans for degrees they didn't even need.
During this time, there was a huge middle class that was able to spend more money on goods and services. People weren't constantly in fear of layoffs or downsizing. And yes, it came with a price of higher wages and a stronger labor movement.
Today, the pendulum is back near the other extreme. In IT it's worse, because there's no career path for most people above a certain age. Your example of the mainframe guy is valid...but too many employers think that everyone over 40ish is unwilling to learn anything. I know incredibly talented colleagues have had to get lucky and find companies who value experience over the willingness to work 80-hour weeks to compensate for sloppy processes or bad work. On top of that, inflation of education requirements means that more people are being force-fed through college. That's good for the university system, but bad for the economy. To make an economy flow smoothly, you need jobs tailored for different intelligence levels. A secretary doesn't need a BA in Communications to answer the phone. A factory worker doesn't need an engineering degree.
I actually like the model of state employment, which follows the classic model. You may not be able to be as much of a rock star in front of your colleagues, but nothing beats job security and safety. The problem, as you point out, is that people do tend to abuse the system as they progress. However, having a steady stream of consumers who aren't afraid to spend real (non-credit) money is what actually drives business.