SF Not an Exception In Giving IT Too Much Control 245
CWmike writes "The city of San Francisco's IT department is certainly not the exception when it comes to allowing just one person to have unfettered rights to make password and configuration changes to networks and enterprise systems. In fact, it's a situation fairly common in many organizations — especially small to medium-size ones, IT managers and others cautioned in the wake of the recent Terry Childs incident."
God complex (Score:5, Funny)
Re:God complex (Score:5, Funny)
-Spaceballs, 1987.
Re:God complex (Score:5, Funny)
Re:God complex (Score:5, Insightful)
The subject of the article is about one central admin having too much control over too many machines, and the risks that entails when they go bad.
Which makes a person wonder... how much worse when billions of consumers are giving total control over all their machines to a centralized authority through Trusted Computing and Vista?
I mean, what happens when Microsoft goes bad?
Re: (Score:2)
When microsoft goes bad? (Score:2)
Re: (Score:2)
Re:God complex (Score:5, Insightful)
It's not just when they go bad. What happens if they get run over by a bus or a stampede of wildebeests? If they are the only person to know the admin passwords, commands, etc, they are the single point of failure, regardless if they go bad or not.
Just as we harp on backing up our files (um, yeah), we also need to harp on a backup for the admin. There should always be someone else, even if it's the mayor, who also has the list of admin passwords.
Re:God complex (Score:5, Insightful)
As far as I'm concerned... passwords are just the beginning. Configurations and such can also be a nightmare to replicate when they're undocumented. Ever stepped into a project where they only guy working on it is gone and you have to figure out how to setup your machine / development environment just to get it to run? It's awful. All of the "don't install that patch, it ruins everything" or "you have to install these components in this order so that they don't interfere with one another" is gone and you have a horrific puzzle before you.
Re:God complex (Score:4, Insightful)
I came into the same philosophy as you a few years ago when I was in the position where I took over a network that was completely undocumented. Now I have Visio diagrams and written explanations of almost everything including a complete inventory of what I have on what network at each site.
I started it with the idea of the bus principle but I've come to rely on it myself as I'm the only admin and so I often have parts of the network I don't touch for a year at a time. This means I forget how things are put together so I refer back to my own documentation. Works every time.
Re:God complex (Score:4, Interesting)
Ya know, I would kill to have another person around with the same skillset that I have but it just ain't gonna happen. Periodically I print out a report of all my passwords and lock them in the safe of the CFO. That way if another admin comes in because I got run over by a bus or more likely in my case, got in a horrid car wreck going well into the triple digits he or she can read my documentation and gain access to the system.
Not the best solution but it works since they refuse to hire me help even though I am way overworked increasing the likelihood I will kill myself traveling to and from work at all hours.
Re: (Score:3, Interesting)
This is exactly what I do too - only, in addition to passwords the document is about 4 pages long and lays out everything someone coming in from the outside would need to know to run our network and servers. It is kept in the safe of our Managing Director.
I think a lot of people just don't understand this Terry Childs story. I know a lot of situations like this where one person in IT has all of the administrative control.
I feel for the guy, and think that, possibly, there may even be more to the story. I am
Re: (Score:3, Insightful)
Yeah, I imagine he was aware of a lot more than most other people as admins usually are. I know that I have much more information about the company and how it operates along with its goals than I necessarily need to do my job but it's the nature of trust.
You have to be able to trust your admin so you should treat them accordingly. That is the first mistake of most employers these days. They treat everyone like dirt including the people that can burn them really badly because they don't understand how much
Failure is not an option (Score:3, Funny)
I'm getting to the point in my network that I'm the only single point of failure.
I'm sorry, Dave, I can't let you do that.
--Your Cisco HAL 9000 Router
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
Re: (Score:3, Interesting)
Now - it is also important to understand that the IT department isn't some fringe function of a company that can be handled and accessed at will, it is today the backbone of many organizations and as important as the accounting division but much more complex.
This means that you must have a reasonable way of handling the IT department. But it is also necessary to check that a single person can block the whole solution. The latter is virtually impossible to resolve since physical access to servers will allow
It will happen again, and continue to happen. (Score:5, Insightful)
Sadly I think the only thing one can do with things this size, is appoint someone and pray he isn't chaotic evil.
Re:It will happen again, and continue to happen. (Score:5, Insightful)
No, that doesn't work. What if, instead of just refusing to divulge the password, Childs had shot himself in the head or gotten hit by a bus or something. He locked down his network so well that only through a password that was only in his head could anyone have admin access.
Re:It will happen again, and continue to happen. (Score:4, Interesting)
While more people should have had access to the network were it ever really needed, sometimes the only really efficient way to take care of a really intricate and dedicated task is to have one person do it all.
He certainly could have been more responsible about it though and prepared assistants to understand exactly how it worked, but who knows, maybe he really was trying to document his system for others but management got in the way of anything productive. That's what management's for, right?
Re: (Score:2)
Well, at my job management stays the fuck out of the way.
This may be less productive in the long run. For example, I'm posting on Slashdot right now.
But the fact is, when you have a complicated system, you need to make sure that more than one person understands it. Sure, in this case it was done with malice aforethought, but a situation where Childs got hit by a bus the day before he would have otherwise locked everyone else out is not hard to imagine.
Re:It will happen again, and continue to happen. (Score:4, Interesting)
This whole "I'm unique and a genius and only through my incredible mental powers does this network keep running" schtick was idiotic long before the lunatic out SF decided that he was God of the Network and beyond any of the Powers that Be. Yes, it's true that complex networks can be tough to explain, and yes, I can well understand why the architect of a network might not want someone else screwing with the configs, but come on, at least a few of us have been faced with having to untangle a complex network config. For the most part, I find the really complex ones I've had to deal with were more due to a distinct lack of ability rather than because the guy was some supergenius. Make some decent network diagrams with good descriptions of what various routers, servers, etc. do, and a reasonably well-trained and/or experienced network guy will likely be able to figure it out. It might be painful at points, and if the old guy is truly gone (rotting in prison because he's a narcissistic wanker or because he got hit by a bus) it might take some work, but providing the configurations aren't some sort of spaghetti routing tables, it should be reasonably possible to pick it all up.
I'm sometimes wonder whether guys like Childs are more hiding their own inadequacies than trying to protect the network from incompetents. I've done a few configs that I've been a little embarassed about, but because of time constraints I went with the flow and hoped either it would stay working or that I'd get a chance further down the road to clean things up.
At any rate, I think it's the head of any IT department's job, implicit in that very position, that the network architecture have some documentation, and that things not just be stored in one's cranium.
Re: (Score:3, Informative)
Luckily these people are becoming less and less common. Why?
Bosses are getting smarter. Some of the bosses actually come from an IT background and know what is going on.
Computers are common. People even Non-IT people are use to using computers, and have a general high level idea what is going on.
SOX and ISO documentation is part of the job now not just a nice to have.
Global Competition, Big fishes in small ponds have been tossed into the ocean. Are you sure you are smarter then everyone else.
Saying it can't
Re: (Score:2)
Re:It will happen again, and continue to happen. (Score:4, Insightful)
Re: (Score:3, Informative)
If something can't be done (by you) and it needs to be done, then what's wrong with bringing in a consultant?
You're not in competition with the guy, he's an expert at whatever you're hiring him for, not to do your job.
When you hire the consultant, just make sure he's not the kind who works behind a closed door. You're paying him to share information with you too, so that you can do general administration on the subject later
Re: (Score:3, Interesting)
I mean think about it, do you think that there was just one person hired in all of SF to manage the network? Exactly, there were people getting paid and not producing. People giving up their freedom in return for promises of stuff without effort. (AKA socialists, communists, freeloaders, hippies,but not all hippies, some of my hippie friends are cool, etc.)
Those are
Re: (Score:3, Interesting)
Exactly!
I learned a long time ago that there is more value in producing a simple, robust, reliable, and reproducible environment than spending all my time and energy milking 10% more out of a configuration that no one else will be able to understand or maintain.
If your system is so complex that someone of half your ability couldn't be trained to maintain or operate it it, you are incompetent. Experience is knowing the best way to support the long term goals of your environment. Experience is not about being
Re: (Score:2)
And by SOX separation of duties, only certain people are allowed to know/do certain things.
Childs having the only password protects against someone else screwing with logs or data.
Hmmm.. 1 bad incident of a bad admin that was minorly costly and annoying vs numerous incidents that have cost billions of dollars where executives had the data changed.
Re:It will happen again, and continue to happen. (Score:4, Insightful)
I do not agree, but even if what you say is true, that just goes to show that efficiency isn't everything!
In the enterprise space, it simply isn't possible to have one person design, implement and operate a non-trivial environment. There aren't enough hours in the day to do all that is required, and I (for one) would like to have a bit of free time - even if all I do with it is sit in front of my playstation.
If the work is so complicated and the deadlines so tight that only one person can pull it off, the project is a disaster waiting to happen. Truly competent technical staff would be the first to escalate that situation to management, rather than indulge their inner megalomaniac and try to do it all solo.
I think we nerds tend to focus on the really cool technology so much that we fail to see the big picture. When you step back a moment, and put it all into perspective, it does not matter if I work 7x24x365 to complete a really complex project on my own. It _really_ does not matter if the design is incredibly elegant, the implementation flawless, and the cut-over into production goes as smooth as silk if one mis-step in front of a speeding bus renders the whole thing an unsupported mess the first time it breaks.
While some might mourn my passing, the lack of documentation and shared knowledge and experience will have reduced all of my heroic efforts to a complete and utter waste of time.
My obligation to my employer isn't to hoard knowledge and information to myself - it is to share that knowledge and information with the other members of my department. If I bring everyone else up to speed, I can have a few week-ends to myself because even the most junior member of the team can step up and help resolve problems if the knowledge base and procedures are thorough and well-documented.
Coming from that perspective, I am unable to find much sympathy for Childs or his former employer. Both have demonstrated extraordinarily poor judgment and are paying the consequences for their lack of professionalism.
(I have to say though, I'm not sure Childs deserves to be in jail, or face such an absurdly high bail amount.)
We nerds tend to interpret "productive" differently than management does. I'm sure most would agree that sitting in front of the keyboard actually doing UNIX-related work qualifies as productive. Management might place a higher priority on documentation, or training other team members as equally if not more productive tasks.
I work in a large enterprise environment and most of my time is not spent at the command line. Most of it is spend communicating with other departments, with my own department, with project teams I have been assigned to, with various levels of management, and with vendors. While I would not have thought so at the beginning of my career, I now see that effective communication skills (which includes listening to others) plays a greater role in being successful at my job than what I do at the command line.
Re: (Score:3, Insightful)
Heheh... heh... it's kind of funny... you can't network people to work on a network.
Re: (Score:3, Insightful)
On the other hand, all these access and relevant documentation was sealed and under lock with the instruction only to be re
Re: (Score:3, Interesting)
Let me speak up for a fellow in my own situation.
I'm not responsible for all IT, but I'm responsible for each and every bit of electricity that traverses a wire in my company. I report to the CTO. I'm the only one of "me"'s that we have.
I've got a pretty spectacular bus factor at the moment, because we can't hire anyone else. The money is tight as is, so I'm doing the only responsible thing. Document everything. Make sure that the passwords are stored somewhere besides my brain, and that someone else can ge
Re: (Score:2)
There needs to be a recovery plan in place, but it's often not practical to have complete redundancy of the "the sysadmin is dead long live the sysadmin" variety.
As long as the system is stable and there is a process for succession of admins, thats about as good as it gets. And there will be "gotcha!" moments, and crap like that, but that is true with all IT gigs.
Re: (Score:3)
Have two vaults. Different locations. In vault is a file cabinet. Each system which requires unique passwords has it's own file. The file is secured so that it has to be opened and proof of that occurring must be evident. Access to the vaults are recorded.
We always joked about someone getting hit by a bus and suddenly dieing with the admin passwords to their system in their head. Well, it wasn't a bus, but it happened. We had such vaults and we had admin access which allowed us to reset other admin
Re: (Score:2, Funny)
Singularity ftw.
Re: (Score:2)
"...and pray he isn't chaotic evil."
well, at least there's an 8 out of 9 chance that's the case.
Re: (Score:2)
well, at least there's an 8 out of 9 chance that's the case.
You really think alignment is evenly distributed?
Re: (Score:2)
lol, our little company doesn't even have anyone else that write basic reports in our sucky report writer much less anything more advanced :(
On the other hand there are no complicated networking stuff (not handled by contract on the main system) or anything so complicated someone couldn't fix or replace it all.
At least being the boss' son and set to inherit it fairly soon there is no threat of me going rogue :)
IT Best Practices... (Score:2)
...you're doing it wrong.
Just waterboard the guy... (Score:4, Funny)
I mean, really. What do we have now? The guy loses control, flips out, locks everyone out of the system, they are down for who knows how long as they bring in crackers and consultants and what not, and the guy goes to jail.
But...
If you just waterboard the guy, until he coughs up the password, the system's not down for really any longer than it takes a Windows Update to screw everything up, so you can just let the guy who locked you out walk, instead of putting him in jail or prison for who knows how long.
Waterboard in this case would be simpler, safer, and better for everyone.
Re: (Score:2)
One reason for not using torture to gather information is that the information gathered is typically unreliable. That is people will say anything to make the pain stop, regardless of whether it is true or false.
For password retrieval, where it is simple to verify the truthfulness of the response, that reason doesn't apply. So, I think your proposal has merit.
Remind me never to take a sysadmin job. :)
... and it isn't even illegal. (Score:2)
The Moment of Truth: Guantanamo (Score:2)
They should televise it, too!
http://en.wikipedia.org/wiki/The_Moment_of_Truth_(US_game_show) [wikipedia.org]
Re: (Score:2)
Re: (Score:2)
Actually he did not do any damage and left the system in a safe state.
A system that is one power-blink away from catastrophic failure with recovery specifically disabled is not in a "safe state." Though I quite accept that Childs was not malicious, he still screwed the pooch.
Re: (Score:2)
Re: (Score:2)
Making it impossible to manage should something go wrong *is* a sort of damage, albeit intangible.
Re: (Score:2)
I mean, really. What do we have now? The guy loses control, flips out, locks everyone out of the system, they are down for who knows how long as they bring in crackers and consultants and what not, and the guy goes to jail.
You should read the articles. It sounds like the Terry Childs's security policy, which his superiors and co-workers all knew about for a while, was to lock everyone except himself out of the routers. The city later hired a security officer, who pointed out the obvious risks in having one guy with all the passwords, so management pressed Terry for the passwords. Terry, thinking that management (or other incompetent fools) planned to do stupid things that would break the network, refused, and management th
Re: (Score:2)
Isn't there a small but significant number of torturees who will choose to allow themselves to be drowned rather than give up the password?
Then, you cap the time and extent of the waterboarding so that the guy doesn't die, and he's looking at a year in jail.
Someone who is already conditioned not to panic when they are upside down in a sea kayak while the gash on their head from hitting the lava rock is gushing out so much blood they can't see. (I've had that).
See, here's the thing, in the case of waterboard
Re: (Score:2)
You really seem to be missing the point of waterboarding. In waterboarding, you do not drown the person. In waterboarding, you keep pushing that person to the point where they feel like they are drowning, and then bring them back up for air. You do not get the option to die. Your only option is to continuously revisit the sensation of dying.
So you have to be more like "professional almost dying person" if you want to be able to defeat water boarding or other torture like tactics that keep you in the "dying"
Here come the elephants. (Score:5, Insightful)
I forget who said that "an elephant is a mouse designed by a committee." Sure, you can get paranoid about network design and control, and give the job to a committee. But that is going to be really clumsy.
The issue here really is not about size of the design team, it is about vetting the guy who does it. ( The guy who is in charge of the network for my business is someone who I really know and trust. He was best man at my wedding. )
Re:Here come the elephants. (Score:5, Funny)
So he's going to change all your passwords *and* run off with your wife?
Re: (Score:2)
The issue here really is not about size of the design team, it is about vetting the guy who does it. ( The guy who is in charge of the network for my business is someone who I really know and trust. He was best man at my wedding. )
What happens when he is hit by a big red bus?
Re: (Score:2)
I do think that may be a valid crossover point. IE my company provides life insurance of 2.5x salary. The IT I take care of (not my main job, but I am the only IT guy) could be completely re-done in a week for significantly less than 1x my annual salary, plus downtime of 0.5X my salary.
Therefore if the death of the friend, and associated costs, is significantly higher than the cost of replacing the current un-documented work. Then that scenario shouldn't weig
Re: (Score:2)
That's hardly a qualifier. It's not like your best man is holding the rope to keep you from falling into molten lava. My brother was my best man, but that doesn't mean I'll trust him with my money or lively hood.
Re: (Score:2)
The guy got married! Clearly, the best man wasn't holding the rope right. :)
Re: (Score:2)
Now I have to get married to find an employee worth trusting?
Re: (Score:2)
"an elephant is a mouse designed by a committee."
The actual quote is: "A camel is a horse designed by a committee." And http://en.wikipedia.org/wiki/Design_by_committee [wikipedia.org] Wiki attributes it to Vogue.
Re: (Score:2)
It depends on how the committee is set up. If the committee only has to sign off that they understand what the person is doing, could take over if necessary, and the person is not capable of being really evil (technically restricted from it), but not that the solution is well-designed or state of the art,
Re: (Score:3, Insightful)
It's about having the guy who knows everything to document it all.
I used to be that guy who knew the entire project. I thought it was crucial for me to know everything, so I remembered everything. All the minutiae, I knew back and forth. No single person on the team had that breadth of knowledge. Thus, I was working all the time.
Then I started to document everything in memos. I sent them around. I recorded everything. It took a whole bunch of time, but it was the best investment ever. I could delegate my w
The familiarity in this story isn't just the IT (Score:2)
"Childs, an employee working for San Francisco's IT department, used his privileged access to lock everyone out of a crucial network for days."
I wonder if it wasn't an intentional lockout, instead someone realized all of a sudden that Childs was numero uno and saying "GIVE ME THE CODES NOW!" and when he didn't someone had a hissy fit and took things very far very quickly instead of competently sitting down and talking with Childs fairly.
Re: (Score:2)
Did you not read any of the articles? Oh, wait, I'm on /. Basically the guy sees the writing on the wall and includes the password equivalent of a dead-man's switch......switch goes off and he's the only one with a key.
Layne
There is no evidence of that... (Score:2)
Basically the guy sees the writing on the wall and includes the password equivalent of a dead-man's switch.
Not according to insiders. He had *always* had the routers configured to clear when someone tried to guess the password, long before any of this started. Why he did this, I don't know... it seems extreme to me but for some networks it's probably appropriate... it IS a standard configuration in the routers. It sounds like someone or something convinced him that this was "best practices" for security, so
Re: (Score:2)
Maybe just almost-rational paranoia to keep someone from breaking into the network at some location to get confidential information... even though if someone has physical access to the router they've already gotten close enough.
Re:The familiarity in this story isn't just the IT (Score:4, Insightful)
Supposedly that's it, according to some of the articles. He thought a lot of the others were screw-ups, so he kept access to himself. Everyone seemed to know it, as well, right up to the top of the IT organization. A new security person was hired, and that person didn't like the situation (may have come up during some sort of review). They made a point of asking him for the passwords, which he interpreted as "hey, we want to screw up the network - you know, the one you feel really possessive about" and refused. Didn't seem to recognize the authority of whoever delivered the message (don't know if it was the new security person or not). They then sent the police after the apparent master criminal.
Also, while they couldn't make configuration changes (that's what "locked out" meant apparently), the network continued to run, even without his intervention. So he might've been a doofus about this issue, and for all I know a total jerk with no people skills, but it sounds like (crazy access issue aside) he knew his job pretty well.
I suspect the new security person (who for all we know is more of a policy person than a technical person) handled it badly on their end as well, and may have gone for a club (formal meetings, demands) when a lunch conversation might've done the trick. The guy shouldn't have held onto exclusive access, but it sounds like the security person didn't handle it well. Apparently, that individual now fears for their safety, which I suspect is either an overreaction or a further attempt to demonize Childs to make it seem like whatever actions taken are justified.
Not news to nerds (Score:5, Informative)
The configuration? Well I am not real sure what they mean? Basic configs such as IP addreses and such have been documented at even the shoddiest implementations I have seen. Plus, if you know how to run that server, you probably know or can find and make changes to the "configuration". But if there is only one person at that company that knows that server/technology, well then there is probably only one person that knows the configuation! What should the accounting manager know how to run our servers?
But the bigger issue is that in a SMB, and in my current positions, I could CHANGE THE PASSWORD!!! Doh, they forgot that you can do that!
TFA goes on to say things about hiring an administrator and then an auditor for the admin. WTF? Never heard of this happening in my career. I do know the military uses these methods, but that makes sense for them. The average sign printing company (even a 200 employee company) can't do that.
TFA highlights a situation that we all knew existed... and didn't even give a (reasonable) proposed solution.
Re: (Score:2, Informative)
It happens. I was called in to try to rescue a small web shop's hosting business. The hosting business was a side business of the web design shop, with two web servers, a database server, and a mail server. All the hosting stuff was run by one guy, he was the only one who knew the passwords, and they unfortunately went with him when he died on the operating table. Five months later, when the increasingly unpatched servers started falling victim to attacks, they called me to try to fix the mess. Of course th
You asked for it, you got it. (Score:5, Insightful)
That's not all they're asking for (Score:5, Insightful)
"You asked for it, you got it." and you are spot on because if they don't correctly assess this current situation, and assign blame to the deserving names, then they are only 'asking for it' to happen again and again.
Re: (Score:2)
Indeed.
And... you wanna' know the best way to lose all your data and not no how it disappeared:
Be careless with who you give you passwords to.
Reminds me of a conversation I had with one of my ex-bosses:
ME: I have complete access to everything on our network.
BOSS: Then what keeps you from reading my e-mai?
ME: Ethics.
Unethical people do unethical things... this isn't an I.T. issue, it's an H.R. issue.
I've always been the one or one of the very few people with omnipotent access of every system in the companies
Opportunity for router vendors (Score:4, Funny)
Cisco should start selling Childs-proof routers! *rimshot*
Opportunities abound! (Score:2)
I guess they could use those annoying screws to secure the lids
Business Mad Libs (Score:3, Insightful)
Yes, this is prevalent. Unfortunately, no, it has precious little to do with IT.
This quote from TFA is quite true, but universally so. Let's play Business Mad Libs:
"Single points of failure are always bad," said John Pescatore,
an analyst at Gartner Inc. "There should never be one person who is
the only person who knows ____ MISSION CRITICAL INFORMATION ____."
Companies need to make sure there are at least two if not three people
who share the knowledge of ____ BUSINESS PROCESS______. "As a minimum,
require it to be documented and stored somewhere if personnel
limitations say you can't have personnel with overlap," Pescatore said.
Have fun playing the accounting, regulatory, legal, and R&D versions, just for warm-up.
Now, if the business managers weren't smart enough to either know this applied to IT as well as their other divisions, or not smart enough to not recognize that that they needed outside advice on how to apply business rules to IT - well, you have to wonder how well the other parts of their businesses are running.
This is silly (Score:5, Insightful)
Of course there will be people in IT who have power, and of course that power can be abused.
Somebody at a television network has the power to broadcast rocking horse porn if they want to as well and there is no time machine to unrock that horse.
The articles hypes up one person being able to abuse power as if it were unique to IT and suggests a remedy that more than one person should have this power, as if this had any bearing on anything, e.g. the ability for the abuser to simply revoke access to others. What, somebody else should be assigned the exclusive ability to revoke? Then that person is the potential abuser. This is silly.
What "incident"?? (Score:5, Insightful)
So where is the "incident"?? What did he do wrong?
By law he might have done "wrong" by not relinquishing the passwords immediately. But by the people of San Francisco, he may have saved them a lot of trouble and headaches. So, he was faced with a dilemma: obey the law, or do the right thing.
Sad.
Not qualified to comment. (Score:5, Funny)
Whenever I register for a site where my email address is my username, the password I use happens to be the same password that I use for my email account.
With that in mind, I'm going to go ahead and not express any opinions on security.
Re: (Score:2)
You say potato... (Score:3, Insightful)
Here's a simple solution... (Score:4, Insightful)
Re: (Score:3, Insightful)
a) that doesn't work in ANY small and probably most medium sized businesses
Small business,no, but then again most small business's, if they do have a network, is well small and not a big deal. I used to setup networks for small companies, most are ad hoc, no dedicated server types, where everyone has admin privileges. A medium size company should be able to do it. As long as you another IT person you can separate the duties amongst them. Hell, I'm one of just four InfoSec people and we share all responsibilities and admin rights.
HA! (Score:5, Interesting)
As if it's ITs fault. Most companies I've worked at I have pointed this very situation out and usually get overruled based on the cost of doing it "right".
(It isn't enough to have several people with the password, you need to know how to recover if you lose total communication with the guy responsible - ig. died.)
Also it isn't just IT. Last months pay got delayed at my company, which really shouldn't happen since KPMG is responsible for taking care of payments for our company. The reason? The lady responsible for authorizing the transfer was the only one with the passwords to do so, and she was in labor.
Replacements? (Score:3, Interesting)
I know people in various industries who consider obscure hacks, lack of documentation, etc "job security."
To me, being the guy who can do it all is great for job security, but the flip-side is that if you're the *only* guy that can handle things... sure, you're semi-irreplacable, but that applies equally to being fired as when you want to take a day off or holiday. Personally, I prefer work-competence as a reason for not being fired, and documentation/standardization as a way to ensure that somebody else ca
A Lesson from Star Wars (Score:5, Insightful)
Others think of the bus rule, what happens if the guy who knows everything about mission critical infrastructure components gets hit by a bus?
That is why I have taken a page from the Sith Lord Darth Bane and apply the rule of two. When I build a network I teach and train one apprentice. Then if they suck I fire them and hire a replacement, but if they are good, when I get bored and decided to move on, I feel confident they can take on a apprentice themselves.
It is neat, clean and simple, better still it doesn't have the rules and complexity of Jedi type systems requiring me to check in docs to a source control system, report changes to managers what don't understand, have managers that don't understand sign-off on things they don't understand and avoid dumb rules like not being able to train techs that appear to old, etc.
Yeh, if you ask me the Republic, I mean Network as a whole is best off with Sith types in charge versus bureaucratic Jedi types.
Re: (Score:2)
I have given out our "master" passoword, and I don't understand what the problem is with other people doing it. The President has 1/3 of the password, the HR director has 1/3, and the lawyer has 1/3, all locked in seperate fireproof safes, in seperate buildings. If things really go bad, or I get hit by a truck, the three of them can get together, and get the password, and give it to my replacement...
The Childs story stinks like five day old fish (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Why can't the people in charge... (Score:2)
What is it with government IT management? (Score:2)
Has anybody else noticed that these reports of gross IT mis-management are almost always government related?
I think there was another story on slashdot, a while back about some guy who accidentally deleted one billion dollars worth of records, and there was no backup. When I was in Florida, there was some scandal about the state spending millions on this new welfare computer system, and the entire thing was borked, so they hired the same company to fix it, and the company borked it again.
Sure, we laugh at t
Re: (Score:3, Insightful)
Thats because only the government related ones concern the public. This stuff happens all the time in the private sector. However, private companies can die, the government cannot (as much as some people around here would like it to)
Re: (Score:2)
What about the companies the government won't let die?
*cough* airlines *cough*
IT is at its core arcane and requires *trust* (Score:2)
The administrators *need* access to the highest level of security. Maybe software and operating systems as a whole need to be rebuilt in the shape of a military complex where sensitive access does not have to be granted to the builders.
But, hey, even the builders see the vaults before they are used.
Hire enough people to do the job right (Score:2)
This is a question of management not hiring enough people to do things right. What happens if the one guy who knows everything goes on vacation? If he never went on vacation, no one would say boo because, in our warped culture, having a desire to do anything but work around the clock is somehow abnormal.
Less control... how about more staff? (Score:5, Insightful)
Seems to me that in many cases, the IT department may be rather grossly understaffed (either in terms of # of staff, or # of experienced staff).
Many places I've worked end up with a Lord-of-all-IT situation simply because they haven't got anyone who can replace him* or back him up, or weren't willing to pay for backup/additional/experienced staff.
* male gender used for convenience purposes.
Banks deal with this (Score:5, Insightful)
One of my first jobs was a bank teller. Our passwords were sealed in an envelop, which we initialed, and locked in a vault which needed two keys to open.
If the two officers needed my password, they'd open the vault, open the envelope, breaking my seal (letting me off the hook of responsibility).
IT has to learn from banks.
Re: (Score:3, Insightful)
The problem was not that he was the only one with access, although that is an issue in small IT departments. No, the problem was that he had enough access to change all the other administrators' passwords. Lots of people had access to the systems, and there were probably procedures in place to name a successor in the event that Childs was fired or hit by a bus. Instead, Childs changed everyone else's password and locked them out.
The only way to protect against that type of an attack is to make the Administr
It depends on who the "one person" is (Score:3, Insightful)
It really depends on who the "one person" is. Committees rarely design good crypto algorithms or protocols, for example. On the other hand, if you just pick the "one person" at random, you risk picking the wrong person.
I guess it's sort of like picking a dictator. If you pick the right person, and hold that person accountable, they will get things done more efficiently than a committee. If you pick the wrong person, they will get the wrong things done more efficiently than a committee.
Duh (Score:3, Interesting)
I've written this one before.
When you have IT people, they're going to have control of your IT infrastructure. Sorry, but there's not much you can do about that. They need access to your data and your equipment to do the job that you want them to do. You'd better find trustworthy people.
This is kind of like complaining, "I have a chaffeur, but I'm nervous that he might go crazy some day and drive me off a bridge, or head-on into a semi." Yes, that is a risk that you'd face by having a driver. And I'm sorry, but no amount of technology gobbledy-gook is going to prevent disaster if your driver does, indeed go crazy.
You face risks whenever you have someone do something for you -- that they might do it wrong, or that they might try to screw you. You're giving them control of some portion of your life. If you're not okay with that, or you don't trust the person that you've hired, you'd better rethink whether you're in the right business...
How about backdoor that can't be turned off? That (Score:2)
How about backdoor that can't be turned off? That was put in by the guy who build the system.