The New IT Crisis 354
Matey-O writes "You've succeeded in delivering 5 9's, your server farm is a well oiled machine, the helpdesk lines lie dormant. No? Well then how do you get credit for the work you do, when all that's noticed is the downtime? When the IT budget has to be justified, and you're overworked, undermanned, and you have to apply three patches to 100 servers before Close of Business, what has to change in IT before we melt down? Marc Andreessen has an interesting article on what has to happen to IT next."
I always just "look" busy (Score:5, Insightful)
I feel that if I work hard (and smart enough), then I deserve free time every once in awhile. After all, I earned it.
But, managers don't understand this. So, I relax by reading The Onion or Freshmeat at work, but always make sure my hand can quickly hit ALT+TAB to get back into my work window (usually Emacs).
Re:I always just "look" busy (Score:2, Insightful)
Re:I always just "look" busy (Score:3, Interesting)
I usually just try to always look busy. I have such a reputation for hard work, that in the (rare) slack times I can usually screw off and get away with it.
But you get to that point by ACTUALLY working hard when there is work to do.
Thing is, when you are a network admin for an enterprise, there is ALWAYS work that you can be doing.
Re:I always just "look" busy (Score:4, Interesting)
Seriously, any time I go to work for a new company, the first 2 months are spent automating everything across the board. I'll generally get no sleep quite a few nights writing software, and when I'm done, I just tell management that because of errors in the past, all work orders will be processed through this new interface. I give 'em an internal web addy and then go play Quake/EQ/etc. for the rest of my time there (which has never been less than 1 1/2 years which I get vested options from). The websites handle all the dummy checking, logs all processes to another system to check periodically to make sure all works well, and performs whatever request they want.
I order spares for about every piece of equipment in the building including spare switches and 1-2 spare servers for the occassional *frying* motherboard. If the servers are set up in non-redundant fashion, I make sure load balancers (or happy cisco 6500) are ordered, every server has a backup, all backups are automated and working properly, and put in the DR (Disaster Recovery) report JUST incase they want to go that route.
Frankly, I have MAYBE 1 hour of downtime a year, and that's usually attributed to my tripping over a cable that some numb-nutz (who's gonna get chewed out for an hour) layed outside of the wire maintenance tray. Only reason I move jobs is because I just get WAY too bored. Admining is easy for those who aren't inept.
Sounds like you've picked some cherry companies (Score:5, Insightful)
Seriously, I don't get the money to buy redundent Cisco routers. I don't get to buy extra switches and motherboards.
And the little internal web addy, that has been in place for 2 years? Some tool of a manager who is having a hissy fit, just walked into my office and asked why I haven't fixed his problem that he never told anyone about.
Please......
Whenever you are dealing with people who have almost as many deadlines as you do are involved. Things are never that easy.
Now where I will agree with you are scripts. Learn them. Use them. Love them. But do you have an IDS system? Who verfiys those alerts? Who checks your server logs for descrepencies? Admining is rarely easy.
I'm busy... (Score:4, Insightful)
Combine that with archaic computers, underfunding, etc etc... and it's an exciting job.
Re:I'm busy... (Score:3, Insightful)
Of course, by abilities, I'm referring to them in their destructive and or disruptive potential.
Re:I always just "look" busy (Score:4, Insightful)
What you're really doing is not wasting time, you're sitting in an "on call" position waiting for your phone to ring with the next big crisis. If that call never comes, then you must have been doing something right when you were working hard.
Re:I always just "look" busy (Score:3, Interesting)
If you aren't willing to do it as "research" (for instance as funny as it sounds Slashdot can be a work-related website for many tech firms. It is "putting your ear to the ground" in a sense), then I recommend what another recommended which is that you browse in a manner that is inconspicuous and requires no screen swapping: Lynx, or grahics-less.
Yeah, but I always WAS busy! (Score:4, Insightful)
Forget the fact that for the last two and a half years, I haven't had a real vacation, because I got a call EVERY FREAKIN' DAY about some minor issue.
Forget the fact that I worked a minimum of 2-3 hours every night on company stuff, 'cause I wasn't allowed to make ANY changes during the day. (The night before I was fired, I spent 3 hours writing a script to fix a problem that was affecting only 1% of the users. No real problem, but I didn't want them to have to deal with ANY issues.)
Forget all of that, just get the fuck out. I take some solace in the fact that two days after they fired us they sent out a global e-mail of "Please bear with us, it'll take up to five days for your calls to be answered." And, e-mail was down for almost a week because no one but us knew how it was setup!
IT needs to get the respect it deserves. In this era of decreasing budgets, the only way companies will be able to make any money is to increase efficiency, and that means automation.
We should all follow Marc's example... (Score:3, Interesting)
It's the easiest way to success.
Details here [chrispy.net].
Quote: it's not as if he needs to sit down in front of his screen and busy himself with the notoriously arduous task of hacking out a few lines of software, which, astonishingly, is something he has never done in the short but spectacular history of Netscape Communications Corporation [...] Not a single line of computer code. Never.
Do what everyone else does . . . (Score:3, Funny)
solution for one of the problems.. (Score:5, Informative)
Not flaming, but a fact.. Microsoft SMS will get that done for you with a couple of clicks.. literally..
And we're expecting to receive a beta copy of SMS 2003 (incl. Active Directory, the big feature).. so i'm kind of looking forward to that..
I have to say.. even though we run 2000 on our servers.. SMS is a dam good product and a timesaver!
Re:solution for one of the problems.. (Score:5, Informative)
Red Hat Network allows you to do the same - a couple of clicks, and away you go!
The advantage is that RHN is here now, and works very, very well. I've never seen a Red Hat Network update break anything. Ever.
And they are quick about getting updates ready!
Re: (Score:3, Informative)
Re:solution for one of the problems.. (Score:3, Informative)
I have no clue how much it costs. I assume if you have the need for it, then it isn't that much.
Re:solution for one of the problems.. (Score:5, Interesting)
When users had dumb terminals on their desk, they had the illision of a full power computer, but it was actually a small box with few moving parts that was linked to high-powered computers (or cluster of computers) who were actually doing the heavy lifting. Since all of the functional components were in the Computer Room, there was rarely a need for tech staff to touch the dumb terminals, and the tech people could work in their own distraction-free environment.
What's more, failures could sometimes be abstracted away from users. Hard drive failures happen, that's a fact of the technology. However, if a HD fails on a user's desk, it means that user has lost the use of their computer until it's fixed. If an HD fails in the datacenter, there's usually a backup of the data which can be put into play immediately by mounting the backup on a good drive that's already spinning. To the users, the disk crash can be practically invisible.
There's already tech technologies such as X-Windows and Windows Terminal Service with which to create GUIs on a dumb terminals. Why does the common secretary need a full-powered PC on her desk anyway?
Re: (Score:3, Interesting)
You forgot support costs.. (Score:3, Insightful)
Re:solution for one of the problems.. (Score:2)
This exists in the PC world. Citrix. I have 2 clients of my company that I support who uses this.
One uses Citrix running off two powerful new 2K servers to avoid replacing armies of Pentium I desktops.
Re:solution for one of the problems.. (Score:4, Insightful)
Early nineties: everything was to be decentralized. Get rid of that antique iron! Go Intel/Microsoft! Can the tech Luddites, bring in the Bright Yunguns, who will show us the way.
Mind you, I was one of those calling for the company to modernize on a client-server model.
But I've watched for, what, 20 years since this all started. I watched the expensive mainframes go into the landfill, and the new servers take their place.
I've watched those servers get more expensive. I've seen the communications hardware become massively expensive. I've watched the new Data Priesthood come into being. I've watched companies being wiped out converting to, and maintaining, the new paradigm.
Not that it would have been good to stay on the old iron. BUT...
We've come full circle. The hounds are baying for simpler solutions, and most importantly, for the elimination of the new mainframers, the admins.
I predicted it over ten years ago. The PC's will cluster into data centers, pseudo-dumb clients will spread onto everyone's desks and laps, the services will automate, and the present IT industry will convulse. A shakeout is coming. Time to get that teaching career started.
As a sidebar: in my experience, far more money was spent on executive games and perks than was ever lost on IT spending. Cost accounting is applied where management wants application.
Re: laptop users left out? (Score:3, Informative)
Then, it's just a matter of training for people to understand that they need to copy their completed work up to their shared disk space on the server when they get a chance. If you keep the versions of software on the server and on the laptops the same, all should go pretty smoothly.
(Of course, this also undermines some of the supposed cost-benefit of Citrix, because you're still buying full software licenses for each laptop, plus all your user licenses for everyone potentially connecting to the Citrix servers.)
My 2 cents (Score:4, Insightful)
systems with a lot of 3d and 2d diagrams
and histograms.
Bosses love that.
Re:My 2 cents (Score:3, Funny)
Re:My 2 cents (Score:2)
Re:My 2 cents (Score:2)
Re:My 2 cents (Score:3, Informative)
Am I missing something? (Score:5, Insightful)
Some businesses demand complex solutions, and I fail to see how these complex solutions are going to be met by turnkey solutions -- where a manager can go out, purchase a server, turn it on, and have it run his business for a year without any kind of customization whatsoever.
If you are, so am I. (Score:5, Interesting)
Because phone service is a relatively well-defined, consistent, limited problem domain. Internet servers, dynamic web sites, and local security are loosely-defined, constantly shifting, open-ended problem domains. They're very different, and you can't compare one to the other.
However, for certain applications, there are well-defined standards, and well-defined practices. Still, for a lot of IT, it's a matter of custom engineering and architecture. For example, online content management: you can buy one of the management engines off-the-shelf, which will probably do most of what you need in a structured manner. For CRM, well, there's about a dozen of those. These packages are well-behaved in that it provides a well-defined interface, but that's not always an option (i know, i used to do data migration for small- to medium-sized businesses. at the low end, when you change systems, you'd better damned well know perl or some other text processing language to massage the data--that is, you need to be good with your bailing wire).
In the future, this situation will hopefully be better with standardization (mostly using XML it seems, even though the actual encoding doesn't really matter.. we could have standardized years ago, but nobody saw any benefit then). Having done data migration in the past, i'm all for keeping things disparate and non-standard, but that's because the work pays well and is fun
A better analogy might be a pool of corporate autos. Except that you don't have to interconnect the cars to get them to share load dynamically, or access content generated on one to form a report on the other, etc. A lot of IT is like trying to drop a big old hemi into a metro, or getting a suburban to go anywhere with just metros to provide power (two in front and one in back, it might go up a hill!).
Overall, I was not impressed with this article, but I'm afraid it's going to carry more clout than it should. oh well.
Re:If you are, so am I. (Score:4, Insightful)
I'd even take this a step further. Let's just say, for the sake of argument, that we reach Marc's promised land of "it just works" hands off IT administration. Then what?
Marc is missing the point. IT is not a necessary evil, it's a competitive advantage. I've sometimes told people who spend too much time attributing their problems to their computer that they should stop using it then. If you can do your job better without the computer, then by all means do.
I have yet to see anyone take me up on this challenge. They know, we all know, that despite their occasionally infuriating peccadillos, that computers make us more productive.
You cannot remain competitive by sitting on your hands. Marc's world of "it's all better so we can rest now" is pure fantasy. The problem with this article is that it portrays IT as a problem, rather than a solution and a competitive advantage. Who wouldn't like to compete against the company that decides it's achieved all essential IT objectives? You want to get your ass kicked, then stop trying to figure out better ways of doing things.
Re:Am I missing something? (Score:2)
Re:Am I missing something? (Score:4, Insightful)
It's not just telephone systems.
Think about almost any complex engineering system. Many are computer-controlled nowadays.
But almost all of them, even those that aren't computer-controlled, are taken for granted by everyone. Transport, power generation and distribution, warehousing, running industrial plants, you name it.
When was the last time you praised the people that keep all these services running ?
But when they go wrong - the power goes off, there's no produce on the shelves to buy, roads are closed or flights are badly delayed - there's millions of people jumping up and down and demanding the situation be sought out straight away.
When IT isn't a hobby or an "entertainment medium" it's nothing more than a tool to get a job done. I know how difficult it can be to work in IT, but I've got to look at some of the responses here, compare them to the jobs of people working in some other industries, and think: "what a bunch of whingers".
Meatless drivel (Score:5, Insightful)
This is an editorial, not an article.
How about talking about Grid computing. Or Organic IT. Developing systems and monitoring capabilities that go beyond telling you things are down, or when they're too busy. They actually add capacity on the fly through virtualization, taking from inactive systems to cover for the active system.
I think Marc once had a vision, but I'm not sure how strong of a visionary he is these days...
Re:Meatless drivel (Score:5, Informative)
You mean Marc Andressen of Loudcloud, who sell server and datacentre management services, and who are desperate for revenue? That Marc Andressen?
It isn't even an editorial, it's free advertising disguised as a story. I hope they're paying whoever owns Slashdot this week for it.
Useless Drivel (Score:3, Informative)
Re:Meatless drivel (Score:4, Interesting)
Re:Meatless drivel (Score:3, Insightful)
Aside from this essentially being an advertorial, there remains the fact that he presents a problem, (kindof) and then presents no solution.
Auto Mechanics, tired of only being appreciated when that car breaks?? Try fixing it before it breaks! Don't ask how.
Firemen, tired of only being appreciated when someone rams a plane into a building?? Try just driving around town hosing everything down.
Now, lets talk "DataCentre Management tools". My experience is this: You need to hire two extra geeks just to get it running, It never really works that well, and spending that money of some clustering and a good perl guru would have a more drastic effect on your uptime. I personally don't have a Unix box that has not maintained 5 9's in availability over the last 4 years. I have a few NT servers that can boast the same.
Products like these are only supposed to appeal to the shirts in a business. You know, the same guys who get freaky about "free" software.
And to think MS once feared this guy....
~Hammy
Who is Marc Andreessen? (Score:3, Insightful)
And if you use software that is written to ``do it right", won't uptimes of 5 nines happen by itself?
I'm just one guy who spent a year trying to get Andreesen's software to work on hundreds of computers. I really don't have any opinion about this multi-millionaire.
Geoff
Opsware? (Score:5, Interesting)
Moderated Articles (Score:5, Interesting)
I am not typically supportive of
Re:Opsware? (Score:2)
However, as I see it, there are two categories of features: those that are fairly generic (like all the features the Opsware guys have) seem naturally amenable to integration into the Operating System - patch deployment, app deployment, app tracking, and so on. Making these products for Windows seems like a losing proposition since MS will just assimilate the features into the next server OS release. But I would say there's a robust market for such products for the Unices.
And then there are features of software itself that needs to be self-managing and self-monitoring. This is substantially harder - we tried to make an application at my last company that would control its own clustering, assigning workload to application instances in an app cluster appropriately (it was the kind of thing not amenable to some kludgy EJB-type solution - it needed real workload partitioning at the application level). Better software toolkits are needed to support this, J2EE just doesn't come close. But I don't know if anybody will make money making such a toolkit. Perhaps.
Re:Opsware? (Score:3, Insightful)
You can't abstract away problems. And software is designed to be customized more than plug-in and exchange like lego blocks. Further, the 'leaky metaphor' problem hits these kind of systems big. (An example from a previous article, "UDP does not guarantee data delivery, and certainly not in the correct order, but TCP/IP does." So where is that guarantee if I cut a 1 foot break in the ethernet cable?)
Software is going to have to get a lot more dumb for this sort of thing to work.
Re:Opsware? (Score:2)
And why shouldn't it?
Having the ability to view VBA-enabled e-mail in Outlook is a feature that few office workers appreciate, but enables so many headache-causing virus situations.
Having a 20 GB HD on everybody's desk allows them to save their data locally intead of on network shares where it is easier to back up. HD failures are much easier to abstract when their is a RAID system running in the back room instead of a HD on every desk.
In fact, how many clock cycles is the average company wasting by having a gigahertz chip on every desk? Wouldn't it be easier if all the high-powered chips were in the data center, timeshared among those who need them at the moment?
Re:Opsware? (Score:3, Interesting)
> shares where it is easier to back up. HD failures are much easier to abstract when their is a RAID
> system running in the back room instead of a HD on every desk.
Great. Now you've got to abstract the network reliability and performance problems. Abstract the lack of privacy, too, while you're at it.
I think these 'magical solutions' try to avoid the appearance of trade-offs. They give the false impression that whatever is the method used is going to be perfect. And local storage vs remote storage is a good example. Assume a magical datacenter management tool gave me an option between the two. How would it abstract the weakness of one vs the weakness of the other?
Re:Opsware? (Score:2)
You are exactly right. I believe Grady Booch said it best: The task of the software development team is to engineer the illusion of simplicity. The corollary of that is that you can never eliminate complexity, you can only shuffle it from one part of a system to another.
Credit for WHAT work? (Score:3, Insightful)
If your servers aren't up with 5 9's and people are always calling in with problems, why do you assume that work as been done that needs credit?
My servers are always up. I proactively manage them to keep them up (rather than "bring them back" up). My users are contented because a) I've trained them to bring me projects before they need them and b) I give them what they need to do their jobs.
If you aren't doing the same, you suck as a sysadmin, there's no two ways about it.
Re:Credit for WHAT work? (Score:5, Insightful)
I agree. In every sysadmin job I've had, I made sure from the day I was hired that people understood that the less they noticed me - or my work - the better I was doing my job.
Sysadmin fix problems. Good sysadmins make sure problems don't happen.
Re:Credit for WHAT work? (Score:4, Insightful)
And what happens at the end of the year? We go on a spending spree. Our boss goes crazy spending.... last year, they bought iPaqs up the wazooo.... not because we needed them, but because they had tons of money left.
I have written memos, proposals.... and it all falls on deaf ears. So I ask you, how am I supposed to give everyone else the tools to do their jobs, if they don't give me the tools to do mine?
Re:Credit for WHAT work? (Score:2)
1. migrated to a firewalled, fiber-gigabit, switched network for 1200 machines and servers from a flat-unprotected-hub based one covering multiple locations
2. migrated said machines from win95/W2k to Windows XP
3. Brought up Active Directory to manage the enterprise.
4. grown from a dozen servers to more than 100
5. brought up a 2TB NAS system with backup, migrating from Novell.
6. Gone from a single static webserver to multiple clustered web servers, with data served up by clustered SQL servers.
7. Migrated away from Lotus Notes to an Exchange based email system.
(on a pro-linux note, we ARE investigating how/if it could assist. Starting in Intrustion Detection and Monitorin)
Now that we've done this, most servers have a great record for uptime, but the maintenance is starting to get hecktic, and it doesn't look like it's going to slow down any time soon.
Re:Credit for WHAT work? (Score:5, Informative)
I can honestly say that none of the servers I look after (running Oracle databases) are anywhere near to 5 9's... and no-one cares. Why? Because no-one who doesn't explicitly have to cares about servers, they care about application availability. If you're on Unix or NT, aiming for 5 9's at a server level is counterproductive, because it means you fall behind on regular maintenance like applying patches that require reboots (yes, Unix has to be rebooted a lot too). Aiming for 5 9's application availability through clustering should be your goal, and if your clustering solution is any good it will happily tolerate slight version mismatches across its nodes so you can smoothly roll out upgrades without compromising service.
enterprise systems mgmt: reboot = 4-letter word (Score:3, Informative)
As a former UNIX Systems Manager, I can tell you that when I started in the position, *yes*, the damn UNIX boxes were rebooted alot. To "clean out the system", to "reset the database", to "run a scheduled job weekly at Friday 5pm when the server restarts".
One by one, I eliminated these.
Packaged software (read: binary only, proprietary) that leaks memory gets fixed - or gets tossed out. Same for those wonderful "zombie creators".
Ridiculous measures (read: rebooting) to correct *symptoms*, rather than performing root-cause analysis/identification, go the way of the dinosaur - those were sometimes tough sells, but if it takes a seminar on enterprise systems management of a uniVerse database server, then so be it! Every one of those budding acolytes can be trusted to carry forth the message in the future - you do not need to reboot enterprise class servers to perform software or application "maintenance"!.
Taking the time to identify root cause pays off, when you are able to stop rebooting the server after you've fixed the problem (and, amazingly enough, the symptoms disappear, too :-/ ).
In some cases, it was easy: no one working there remembered why some servers were rebooted. Asinine - but easily corrected.
You still need regular maintenance downtime - aside from urgent patches (security, availability etc) and hardware problems. These are scheduled, off-peak, and redundancy (where used) means no hit to "availability" (which applies to the service offered, not to each node).
To sum up: what my gray hairs and the bags under the shadows under my eyes have taught me is that 99.999% of reboots that contribute to "lack of availability" are in reality lack of skill/experience, talent, or both. With a mild dose of business people making the decision to reboot servers daily or weekly, like their PCs...
Just my experience, and my $0.02.
Re:enterprise systems mgmt: reboot = 4-letter word (Score:3, Informative)
OK, so you're running an Oracle database server on Solaris, in dedicated connection mode. You need to connect another application server node to it, which will push it over its permitted number of processes. To do this, you need to add more semphores in the kernel and reboot, because Solaris can't allocate them dynamically.
So you cleanly fail over onto your second node, reconfigure and reboot node 1, rejoin the cluster and fail node 2, reconfigure it and reboot, rejoin the cluster and you're good to go without compromising application availability. If you know a way to do it without rebooting, you'd better go tell Sun, because they don't know it.
To sum up: what my gray hairs and the bags under the shadows under my eyes have taught me is that 99.999% of reboots that contribute to "lack of availability" are in reality lack of skill/experience, talent, or both.
A lot of people think Unix is the be-all and end-all of operating systems. But it's a long way from that.
I don't know where you work (Score:2)
Re:I don't know where you work (Score:3, Funny)
Contrast that with the Socialist Way: You pretend to work and they pretend to pay you.
It sounds like the solution... (Score:5, Insightful)
I'm sure that Bill Gates could write an article about the crisis facing the Internet, and how
Nothing (Score:5, Insightful)
What WILL change? nothing
for every burnt out admin thats going to quit theres 5 more waiting to take his place
good companies keep good employees
eather your not a good employee
or your not working at a good company
Re:Nothing (Score:3, Insightful)
I've seen that comment several times but it finally occurred to me what's wrong with it: KM. (That's "Knowlege Management".)
When an admin leaves, he takes a tremendous amount of knowlege away with him or her, which must be painstakingly re-aquired by the next admin. This can easily take in excess of a year, depending on how well the original admin did his job. (It takes less if everything's broken; well-oiled parts of the machine may not require attention for a long time, so those take longer to learn.)
Changing admins is far, far from free, and business will eventually notice that. (In fact, they are starting to, but my impression is that KM is still a "kooky" field, not yet mainstream. Corrections from those closer to that community welcome.)
It's not the work you do (Score:4, Interesting)
I've worked in a few large outfits and in my own small business, and I can testify there is almost zero correlation in a large office between the work that you do and people's perception of what you do.
The people who have the most problems, the ones who have a terrible catastrophe which just always seems to happen to them, are seen as the problem solvers. Despite the fact that their own lack of organization, incompetence, or laziness often brings these things upon them. No matter, they can proudly trumpet how they once again "saved the company" and worked 30 hours straight. The ones like me who prevent the problems, who organize their day so that nothing exciting happens if it can be avoided, and quietly solve problems on their own without assistance before people notice them, are seen as either invisible or lazy.
And no, I don't work a 30 hour day. Ever. I don't need to. I'm not bitter... gak!
Quit (Score:4, Insightful)
The very reason I left. When something crashes, who gets blamed? When users forget their passwords (which are usually something as simple like their friggin' username), who gets hasseled? When admin lays down an impossible time table with ridiculous performance expectations, who gets told "make this work or else?" When the company starts loosing money due to poor business decisions and/or the economy being in the toilet, who's the first dept to get cut?
Not as appealing as those tech school commercials make it out to be, huh?
"New crisis"? (Score:2, Interesting)
Automate system administration (Score:3)
I recommend the paper, "Bootstrapping an infrastructure [infrastructures.org]."
This ignores two problems... (Score:5, Insightful)
b) A significant amount of trouble with maintaining systems comes from having to figure out lots of different pieces of hardware. Lots of random equipment makes IT support a great deal more difficult. There are two solutions:
1. Standardize all company hardware on a small number of systems/components, say one type of desktop, one type of server, and a few special purpose machines, and then only support those. Tools like VASystemImager then can make tasks like upgrading and bug fixing vastly simpler.
2. Use inexpensive thin clients interfacing to some powerful central server, ala Largo, and only have to maintain that central machine and swap out cheap, dumb clients. Also simplifies things tremendously.
People will no doubt point out that you have to run different types of OSs for different jobs and so on, but you can still use the central server/thin client approach and just make the connection to whichever OS you need transparent. It takes thought to set up, but once it is working you don't have 4000 individual support headaches to deal with. Only a few machines to upgrade, support, etc.
Unfortunately, this won't happen. First, you would have to have a truly MASSIVE infastructure upgrade, which replaces a working system. Riiight. Second, you need to have management willing to try something new and be patient to wait for the long term results. That's not how they think - they think next quarter profits. There is also sheer mental inertia to contend with.
It would be much easier for new companies to adopt this idea from the get go, than for older companies to adopt it. That may be where new, useful IT principles get applied.
The only way current companies will do something is if the system BREAKS, and I mean just totally stops functioning. Thats when they will wake up to the fact that significant changes are needed.
This is a huge problem where I work (Score:3, Informative)
We are NEVER allowed enough time to do more than barely keep things running. Which always leads to things breaking that could easily have been prevented had someone been there to do routine maitenance.
New deployments (like a Linux server that does remote access for terminal emulation) that I do are compromised by the fact that I'm never given more time than is needed to barely get it working. I never truly get to finish a job to my own high standards.
My employer views this service contract as "nonbillable hours" despite the fact that they pay us $9K a month for it. The boss wants myself and our other engineers working on other "billable" projects that bring in far less than that $9K. That money doesn't get this company a SINGLE dedicated contract employee (despite the fact that our whole tech staff's monthly salaries don't equal $9K)
Not only that, our company recently fired our most talented engineer without cause (the week before Thanksgiving) becuase he could hire a "paper tiger" (ie: MCSE) for $5K a year less, and a H1-B who he brought in and treats like a slave.
It's definately the dark side of IT oursourcing, and something that companies considering doing this should think about.
If I were going to outsource an IT department for a company with multiple locations and servers, I'd keep at least ONE in-house guy and use the outsource company soley for the "it's broken" crises which need more manpower.
You Americans need to fuck people back. (Score:3, Insightful)
The boss would be left with an empty department to explain to his VP and justify his bad decisions. Not that it would happen - I have the total faith in my boss and the morale is good, people help each other out.
You Americans need to learn to fuck people back and not take shit from nobody. The SINGLE reason managers (in IT and elsewhere) can treat people like this, is that the people being screwed react with bending over and asking for more.
Re:@#$%! H-1B's (Score:4, Insightful)
"Don't get me started about this H-1B thing. You will get my britches in knots again."
The "paper tiger" MCSE/CNE, I object to. Not the H1-B. He was hired 3 months before the boss let my friend go (who was my mentor, and largely respobsible for why I'm as good as I am). The H1B in question is bloody brilliant. One of the hardest working and smartest people I've ever met. He remembers anything I teach him after showing it to him ONCE... I can only WISH I were that smart.
I object to H1-B more because it VICTIMIZES those who come here on that kind of visa more than I object to skilled immigration. The USA is better off letting people like him into this country, especially when it deprives an opressive country (Iran) of a brilliant mind. But scumbags like my employer shouldn't be allowed to bring him in on a H1-B "slave" visa...
Did he say anything? (Score:5, Insightful)
Well, here's where you expect an innovator like Andreesen to come up with a brilliant idea that's going to begin the next IT paradigm shift, but all he says is that we need to find some revolutionary way to automate our own stuff -- basically, to automate the act of automating things. And how? Well, he doesn't really know. He makes some vague reference to sending out automatic updates to hundreds of servers at a time, and that's it.
Real bright there, Marc. Automatic patches and updates. As if that's the answer. In the real world, you don't have a huge farm of servers that all run the same patchlevel of the same operating system. I've got a few hundred boxen behind the glass, for example, that are a mix of Linux, Solaris, FreeBSD, Windows 2000, and Windows NT. And I'd guess that at least 50 percent of them would experience some sort of problem if we were to just push updates out to them unattended -- different applications require different patchlevels and break on others.
Let's not forget the fact that there's more than just servers. There's infrastructure such as routers, firewalls, and switches. And of course there is the dreaded desktop, which is probably the source of 90+ percent of IT headaches. Until the IT world wakes up and gets the hell off local desktops, the maintenance nightmare will continue. Seen what Microsoft is doing lately? Their vision of the future is one in which applications are loaded through a browser and executed in a local
The other trend you're going to start to see is outsourcing. People are realizing that it's expensive to build and run a data center. Fortunately, you don't have to. All you have to do is run your servers at a hosting center [xand.com] that knows how to do outsourced IT (as opposed to just hosting web sites, like the first generation of centers like Exodus did).
There are ways of streamlining IT after all. Unfortunately, Marc Andreesen didn't touch on any of them. I give this article a "C minus."
Preventative maintenance (Score:2)
The financial rewards of having a good IT department cannot be expresed in real numbers. The cost of a downtime that didn't happen is impossible to accurately measure, while it's easy to measure the cost of having a quality IT team working to prevent downtimes rather than to fix them.
Managers need to understand that cutting their IT department will cause lost productivity in other departments in small units. Those 10 minutes of work time Sally loses while trying to find an admin to remind her what her password is, the lost sales that result when e-mail goes down and customers instead turn to your competitor, the custom product that has to be thrown out because a wrong quanity made it on the paperwork that went down the production floor, etc.
In the end, the business that "don't get it" will slowly become victims of natural selection and close their doors.
Have you tried: (Score:5, Funny)
Yes, going Postal. It's hot, it's new, it's... Satisfying.
"Cool and smooth, I don't know how I dealt with those bastards that kept asking 'this mouse?' without going Postal" -- Satisfied customer.
And now you too can try it, for the low low price of a single aluminum hardball bat. Spun aluminum with a non-slip rubber grip means never having to say "For all that is good and holy it does not optimize your connection!"
Re:Have you tried: (Score:2)
> bat. Spun aluminum with a non-slip rubber grip means never having to say "For
> all that is good and holy it does not optimize your connection!"
You can't properly go postal without automatic weapons. If you're going to do something, so it right!
dave
Re:Have you tried: (Score:2)
"so it right" should be "do it right".
dave
What a pointless article... (Score:5, Insightful)
DUH! I mean c'mon, the reason why IT is such a mess is because all the IT staff are being expected to do more with less and do it faster. They are overworked fighting fires which means they aren't given the resources to do advanced planning and put together systems that would really get things right. He makes it sound like this is the responsibility of the IT people, but it's really the responsibility fo the business as a whole to have some foresight and help these IT people out.
Overworked sysadmins do what has to be done to keep something working RIGHT NOW. Sysadmins with some free time will spend their efforts writing scripts, automating tedious tasks, and making sure fires don't happen. When fires do happen, they have the time to deal with them effectively because they've had a chance to automate a lot of the other tasks.
I have known many a sysadmin and I have never met a one who wasn't constantly pissed off because he lacked resources. A friend of mine was sysadmin for a company that wanted to have 24 hour uptime for their systems. He was the ONLY sysadmin. That sort of crap happens constantly in the IT world because the other members of the company have the wrong atttitude about IT, that it's an expense. If they looked at it like they look at factories and buildings, as an infrastructure investment, then you'd probably see a lot more happy IT managers out there.
Without reading the article.. (Score:3, Informative)
Marc Andreessen has an interesting article on what has to happen to IT next.
I'm going to hazard a guess that this will forecast the overworked, underpaid endgame of IT gruntwork and usher in a new era where companies with CUSTOMIZED SERVICES and NEXT GENERATION TECHNOLOGY come in and automate IT and drastically reduce every admin's workload. He happens to run just a company that provides these services.
I will now read the article and be amazed if that isn't exactly what he says.
Fluff (Score:2)
Vendor applications rarely work without issue and most companies want to find a way to link every system to every system. They then pay large sums of money out to have someone integrate the applications for them. In the process they create something which is very hard to manage.
Saying that the bubble is over doesn't mean anything in IT. Your competition is still alive and you still need to be doing that one extra thing that makes you the better buy. If anything this means at the moment that you need to be pushing more not less.
PR schemes as news. (Score:2, Insightful)
So how do you soften people up to the idea? Wait a couple of months, release a low key but suggestive "article" to get the concept back in peoples' heads, then launch the offending software/hardware/schema again about a year later onto a public, now resigned to seeing the new 'feature' as inevitable. It's a standard pr tactic.
We saw it with Intel's P3 PSN fiasco, numerous webmail service privacy policy changes, and the XP activation scheme. And I'm not saying that all this is the work of evildoers, just that this is what it is.
Why IT is not (yet) telecoms (Score:3, Interesting)
Telephony is a mature technology that doesn't completely change the way it works every five years. For 80 or so years the way the signals got routed didn't change much at all. Then exchanges went digital, and, for the transition period, it was all bailing wire like the article says, except that the telephone companies had - dare I say it - telephone number budgets to pay for the changeover.
By comparison, the rate of change in IT is still very high. We've gone from mainframe to micro, from thin client through peer networking and back to thin client, from standalone to the Internet, we've done dial-up, ADSL, wireless...
... and one of the main reasons for all the bailing wire is that no company can afford to throw away all it's infrastructure every 24 months. If telephone systems stored data, and if handsets ran bespoke software, there would still be a few manual exchanges in use for backward compatibility.
Management... (Score:4, Interesting)
Take networking for example - it used to be in our place BNC and the occasional run of UTP cable - all attached to relatively unintelligent devices. Now its all Cisco switches, fiber and cat5e - and it really is a full time job in itself managing a large network with so many 'intelligent' devices.
Also taking into account the addition of so many more servers (SQL, Mail, Finance stuff, Student Records, DNS, Proxy..) - the list is endless. Again, these systems have really bloomed in the past 4 or so years, at least for where I work.
I guess they dont see how much goes on behind doors when it comes to this business..
remind them of your accomplishments (Score:2, Interesting)
How do I justify my existence? Easy, the colo machines that I don't administer get hacked or broken almost once a month. Each time, I send a report and I also tell my boss "by the way, I already took care of this, we won't have this problem".
(Maybe I should send a report each month: "security holes that DIDN'T affect us this month".)
After a while he realized that I'm like an insurance policy (as well as my Windows counterpart who does basically the same thing).
Now I just hope that all these colo customers don't sign up for our monitoring service, that will really make me invisible.
Another less ethical option is to leave a few unsolved but safe problems in the machines so that you have a small fire to put out each week, to make yourself look busy.
Sound design wins again.. (Score:4, Insightful)
More Evident (Score:2)
Credibility (Score:3, Insightful)
My experience.... (Score:2)
When we brought in a new CEO, she began looking for ways to cut costs. I was slated to be on the chopping block because she "never saw me fixing anything". Luckily, I was saved by the management who realized that I was the only one who held things together (of course, they still didn't pay me squat since I was the only person who didn't have a degree and they were mostly PhDs and Masters of whatever). Once I ended up moved on to a higher paying job, they pretty much fell apart.
The lesson I suppose is that even if you've prevented all of the problems and the organization runs smoothly because of your efforts, you need to make your efforts visible. Sometimes, bragging isn't a bad thing.
What happens? (Score:2)
You point out that, though perhaps your IT guys are paid a relatively high salary compared to the rest of the company, your time between failures, and your overall spending are much less than that of similar companies. THAT is what the suits understand.
If your IT guy is some kind of stinky zen monk who does nothing all day but medidate or work in the zen garden he build in his office, the suits will be happy if they are spending 1/10th as much as the competition, and everything just plain works.
Believe me... every manager out there, ever CEO, gets to hear from every company he deals with, in the news, and his mother in law about how computers don't work, the network screwed up, etcetera. When he sits back and thinks "Man, mine works fine..." he will have better faith in his IT guys.
Big words, little substance (Score:3, Informative)
In two paragraphs he pronounced to the world the basically all IT infrastructure and paradigms are broken. He later suggests that in order to keep up and succeed, things will have to change.
Duh.
The article is so short on details of the failure and possible solutions, I don't know why he wasted the space and bandwidth to deliver this most elegant piece of fluff.
Fortune 2000 and enterprise in general has been raping its IT payroll for well over a year, probably close to two years. In that space, technology has changed, service delivery ramped up to top up the cuts-enhanced bottom line, media reinvented itself 4 or 5 times, and customers got a whole lot pickier and smarter.
So the one piece of non-fluff in the article was the mention that a lot of data centers are being held together with spit and string. Well, this is what happens when you whip 5 people to do the work of 20. Seems like thats ok to do as long as the victoms have ballpoint pens in their shirt pockets and hornrimmed glasses perched on the noses.
The biggest problem in the IT field these days is entrenched in the problems Tim Perdue experienced at SF. Every time an achievement is approached, 45000 know-it-alls with 6 digit incomes glom on, take credit, and micromanage at the DNA level. The suits, blissful in their ignorance and trusting of middlemanagement, believe the stuff that spews out of these ninnies mouths. The solution is for upper management and grassroots IT people, the folks in the trenches, to get together.
Upper management, in order to be able to do this, needs to be sensitized to the machinations of IT people. They need to know what makes us tick, or they risk finding out what makes us ticked off. They best be doing this quick too. The downturn will end, business will pick up, and a lot of these companies will be up a creek with their infrastructure decay and miniscule overmanaged IT budgets.
Revenge of the nerds indeed.
The problem with the hacker ethos (Score:5, Insightful)
This is what feeds Andreeson's IT crisis today: the fact that technology professionals took their apparent suffering and feelings of being not understood, and used it to isolate themselves. They refused to act like businesspeople in an organization that lives and dies by its profit and loss statement. They complain about how management doesn't appreciate them, but how many learn to do a cost analysis that will show the business reasons for buying software X or hardware Y? In other words, the glorification of the geek in the 90s gave a lot of geeks the idea that they didn't have to learn the language of business to survive there. That's why they're underfunded, underappreciated, and harassed.
I've had the benefit of a boss who demanded a business analysis for any significant technology initiative at the company. He doesn't get computers, but he understands ROI. He understands a well-presented business case for anti-virus software. We have a wireless network in our new facility in Texas for a real-time inventory management system for one reason: my cost analysis showed that the implementation costs would be recovered within a year because of labor saved from eliminating batch-mode downloading, and that the cost over five years of our wireless system was ~15% of the batch-mode system.
When geeks figure out that they have to speak the language of the business, then the IT department gets properly funded, properly respected, and properly treated.
Re:The problem with the hacker ethos (Score:3, Informative)
once sided.
"If these damn smelly hippy UNIX admins could learn
to act more like good corporate lapdogs, we'd make
a bundle"
Not to say there isn't some validity in there. But
the other side of that coin is:
"If these retarded, non-tech savvy idiots gave me
a boss that understands technology, didn't let
the morons in marketing make promises they didn't
check if we could keep. If HR didn't have so much
power they effectively ruin the company..." and
so on adnauseum.
The real problem stems from the "us" versus
"them" philosophy. "Gotta have a strong manager
watch over them there technogeeks or they'll walk
all over us". And the "You don't need a technical
person to manage technical people" mentality also.
It has to do with the level of fear a
non-technical person has about technical people.
Because technical people "know things". And they
are probably "trying to trick you". So the ball
starts in their court. Most of the time upper
managment fumbles that ball all the hell, and
deserves the treatment they get from their
overworked, underpaid wage slaves that they don't
listen too or appreciate. That's just my opinion
after 14 years in the business from both sides
of the fence.
Where to start (Score:3, Insightful)
- Ask your non-geek boss how to make a business presentation on the business merits of a technology initiative. Ask her about cost analysis. Ask her about justifying new software to her bosses (what they're looking for).
- Pick up a textbook on managerial accounting (which is different than what accoutants do): it'll go into great detail on how businesses analyse costs and plan with them.
- Look at your company's profit and loss statement and figure it out. I started at the company as a cost analyst, and now I know exactly how the business is financially structured.
- Try doing it yourself: start a spreadsheet. List all the costs of something, say, upgrading to the latest version of Office. Include every cost you can think of: retraining, troubleshooting, labor spent installing, licence fees, costs to convert all the documents on the server. Don't just list dollar figures, list your calculation: 30 hours at $20.00/hr (but don't forget to include the cost of benefits and other expenses: an employee paid $10/hr costs the business $12-14/hr). Then list all the monetary benefits: time saved with zippity feature X, etc. Compare the bottom line: will the business benefit financially from the upgrade?
At it's heart, understanding the internals of a business means understanding it's programming language: finance. You don't understand how a business works until you understand the flow of money through it. Figure that out, and you'll be a respected geek who can talk to managers on their own terms.Nagios (Score:4, Interesting)
I moved out of a group running a lot of big Sun machines (I set up an E10K for them) because of managerial issues. Before I left, we had a budget item for about $250,000 to set up a monitoring and job-scheduling application. It was going to take *another* Sun box to run, and we were being told that it would take 3 months to get it all set up and configured.
With Nagios, I can do everything we they were talking about implementing. I spent 3 weeks, and it cost me nothing. I employed a dual PII 266 that was collecting dust. (I also used an old P166 as a dedicated kiosk for showing the web page.) My boss and my co-workers think it's great. I'm dying to show it to my old group...
machine? (Score:3, Funny)
Pretty weak-ass server farm, if you ask me..
Automation nation (Score:4, Interesting)
I predict that we'll have software that can write major software at just the same time as we have software that can write convincing novels. In both cases you have the task of putting together language that respresents a broad swath of messy reality.
Now, systems administration may be more like writing a good technical manual than writing a good novel. Ever notice how many good tech handbooks there are out there? You haven't? Maybe it's because novels are easier. Good systems administration is about leveraging people strengths with machine strengths, and vice versa. Automation without the human element is as uncompetitive as, well, the human element without automation these days.
Re:Automation nation (Score:3, Insightful)
Actually, we do already have software that can write software, it's called CASE, Comupter Aided Software Engineering.
Unfortunately, it's takes about as much effort to explain to CASE what you want your application to do as it would to write the application yourself anyway!
Automation isnt possible today. (Score:3, Interesting)
Well because the software we use today doesnt lend itself volontarily to extencive automation. You can automate patch installations, users added to all relevant systems at one click of a button, backup and all such things. The problem that some people seems to have a hard time grasping is that software sucks mostly. It is ridden with faults that make any automation fail randomly no matter how well it is implemented. Thats where most IT staff is doing their job, straighten out faults in the software and installing it. Support is also very hard to automate.
Before any automation can be used on a daily basis software must get much better and have much less bugs at shipping date than today. Its a very wrong approach to go backwards and automate fixing of faults related to bugs. Fix the bugs instead.
Stop trying to sell us something Marc (Score:3, Interesting)
Why did it fail?
1 No one really knows how to do it
2 The infrastructure is too expensive
3 Customer requirements are too dissimilar from one another
4 No one has the balls to tell customers their requirements are crazy and impossible
5 Transition costs are poorly understood
6 Exeutives are measured by overhead and customer satisfaction and not doing the right job the best way
7 People are not a resource they are an overhead item
supporting open source: radmind (Score:4, Informative)
Incidentally, the CTO of loudcloud ( a.k.a. opsware ) is Tim Howes, of LDAP fame and formerly of the UMich RSUG ( the same group that has since developed radmind ). small world.
The best sys admin (Score:5, Funny)
Worship the SA, do not replace the SA with a small shell script.
There's something in what he says (Score:3, Interesting)
The idea that impressed me then was the thought that nobody would seriously consider generating their own electricity now that it's a utility. But back in the early days companies and communities did just that. Same thing today with computing, but tomorrow...
It strikes me there's a shade more to IBM's vision than there is to Andreessen's, though. Check out the IBM version [ibm.com] here, with links to some more in-depth material.
Comment removed (Score:3, Insightful)
Ah yes, automation. Nice plan. (Score:3, Insightful)
There's just a few gotchas. These revolve around consistency.
Are your servers consistent? Are your OS's consistent? Are your applications deployed the same way in each of your remote offices? Do your users have the same applications on their computers?
Or is your IT infrastructure a giant ad hoc hairball resulting from IT decisions being made by non-IT personnel? Do you use anything because "that's what everyone else uses, so it must be good enough, therefore opinions to the contrary are wrong"?
Have you implemented things like network management and application/workstation management only to find that the investment is worthless because your organization cannot adhere to a set of standards long enough to make such systems effective, resulting in such an expanse of policy variance that the "management" systems themselves become a net drain on your management resources?
If so, then you've come to realize that the IT problem does not have a technical solution, rendering the entire premise of the article false.
The problem is cultural. IT people are not trusted to make IT decisions, such as meaningful policies with regard to how technology is implemented and what resources are required to deliver a given level of service.
What all of this means is that your Great Answer may not come from the vendor/consultant of the month, but by simply asking your people what is going on, listening to them, and giving them the leeway to make things right. And yes, this was written by a consultant.
Re:I'm stupid (Score:3, Informative)
Re:I'm stupid (Score:3, Informative)
99.999% reliable.
Re:I'm stupid (Score:5, Informative)
Technobabble for 316 seconds (about 5 minutes) of unavailability (downtime) per year (99.999%). The figure usually refers to overall application availability, not individual machine availability (e.g., SlashDot's individual servers may go down periodically for tens or hundreds of minutes per year for software / hardware upgrades, fixes, etc., but we as users don't necessarily notice because of redundancy, fail-over, etc.).