SF Not an Exception In Giving IT Too Much Control 245
CWmike writes "The city of San Francisco's IT department is certainly not the exception when it comes to allowing just one person to have unfettered rights to make password and configuration changes to networks and enterprise systems. In fact, it's a situation fairly common in many organizations — especially small to medium-size ones, IT managers and others cautioned in the wake of the recent Terry Childs incident."
Re:It will happen again, and continue to happen. (Score:4, Interesting)
While more people should have had access to the network were it ever really needed, sometimes the only really efficient way to take care of a really intricate and dedicated task is to have one person do it all.
He certainly could have been more responsible about it though and prepared assistants to understand exactly how it worked, but who knows, maybe he really was trying to document his system for others but management got in the way of anything productive. That's what management's for, right?
HA! (Score:5, Interesting)
As if it's ITs fault. Most companies I've worked at I have pointed this very situation out and usually get overruled based on the cost of doing it "right".
(It isn't enough to have several people with the password, you need to know how to recover if you lose total communication with the guy responsible - ig. died.)
Also it isn't just IT. Last months pay got delayed at my company, which really shouldn't happen since KPMG is responsible for taking care of payments for our company. The reason? The lady responsible for authorizing the transfer was the only one with the passwords to do so, and she was in labor.
Can't sell trust. (Score:1, Interesting)
I am apart of a SMALL IT firm. We run into this ALL the time.
We have run into clients who's own domain name is not owned by them but their support staff that purchased it. When the service provider is fired due to breach of contract or SLA, they often take the name down until the final invoice is received. This is often in dispute because the last month of work has many extras. Their domain name is held hostage!
We both hold to the same worldview which allows us to have full trust of each other and our clients trust us. We have access to each other's email and passwords for work related stuff.
Whenever we get a new client, we examine all their records and make sure we have passwords to everything. We give the client everything and alert them to any changes.
You can't sell trust, but clients know it or learn it.
Re:It will happen again, and continue to happen. (Score:4, Interesting)
This whole "I'm unique and a genius and only through my incredible mental powers does this network keep running" schtick was idiotic long before the lunatic out SF decided that he was God of the Network and beyond any of the Powers that Be. Yes, it's true that complex networks can be tough to explain, and yes, I can well understand why the architect of a network might not want someone else screwing with the configs, but come on, at least a few of us have been faced with having to untangle a complex network config. For the most part, I find the really complex ones I've had to deal with were more due to a distinct lack of ability rather than because the guy was some supergenius. Make some decent network diagrams with good descriptions of what various routers, servers, etc. do, and a reasonably well-trained and/or experienced network guy will likely be able to figure it out. It might be painful at points, and if the old guy is truly gone (rotting in prison because he's a narcissistic wanker or because he got hit by a bus) it might take some work, but providing the configurations aren't some sort of spaghetti routing tables, it should be reasonably possible to pick it all up.
I'm sometimes wonder whether guys like Childs are more hiding their own inadequacies than trying to protect the network from incompetents. I've done a few configs that I've been a little embarassed about, but because of time constraints I went with the flow and hoped either it would stay working or that I'd get a chance further down the road to clean things up.
At any rate, I think it's the head of any IT department's job, implicit in that very position, that the network architecture have some documentation, and that things not just be stored in one's cranium.
Replacements? (Score:3, Interesting)
I know people in various industries who consider obscure hacks, lack of documentation, etc "job security."
To me, being the guy who can do it all is great for job security, but the flip-side is that if you're the *only* guy that can handle things... sure, you're semi-irreplacable, but that applies equally to being fired as when you want to take a day off or holiday. Personally, I prefer work-competence as a reason for not being fired, and documentation/standardization as a way to ensure that somebody else can back me up when I want to take a few weeks off (real time off, as in not near a computer and not "on call" with a pager/cellphone going off in my pants pocket next to the pool).
Re:Banks deal with this (Score:1, Interesting)
What would happen if you changed the passwords to be different from those in the envelope and didn't tell anyone ?
I've never seen any password control interface which requires two people to hit the enter key.
No surprise (Score:2, Interesting)
You also tend to see a lot of multi-hat positions (Chief Security Engineer/Firewall SME/Lead Network Admin), and mentioning security best practices such as Duty Rotation and Separation of Duties is usually met with a "yeah, right..." smirk and chuckle.
Unfortunately, it's all usually a function of budget + quality of applicants + total inability to communicate effectively with City Council/County Board/etc. to explain why what the PHBs want needs to be properly funded and staffed.
Inevitably, the powers that be decide they need something, and all heads in the room turn to the resident nerd-genius, who immediately geeks out about how he could accomplish it technically using spit and duct tape. The managers unclench when they realize they aren't going to actually have to do their job; what little money there is money gets blown on hardware and software, and the whole thing gets wired up in a perfect example of 'just barely good enough engineering' or a hobbyist project.
It's not really how you expect your local gov't to operate, but they do it all the time. It's kind of like knowing where sausage comes from. Just don't ask.
Re:Banks deal with this (Score:2, Interesting)
Re:It will happen again, and continue to happen. (Score:3, Interesting)
Let me speak up for a fellow in my own situation.
I'm not responsible for all IT, but I'm responsible for each and every bit of electricity that traverses a wire in my company. I report to the CTO. I'm the only one of "me"'s that we have.
I've got a pretty spectacular bus factor at the moment, because we can't hire anyone else. The money is tight as is, so I'm doing the only responsible thing. Document everything. Make sure that the passwords are stored somewhere besides my brain, and that someone else can get to them.
Treat yourself like any other piece of networking equipment. If you can only afford one, make sure it's settings are backed up and that you know how to recreate it if need be.
Re:God complex (Score:4, Interesting)
Ya know, I would kill to have another person around with the same skillset that I have but it just ain't gonna happen. Periodically I print out a report of all my passwords and lock them in the safe of the CFO. That way if another admin comes in because I got run over by a bus or more likely in my case, got in a horrid car wreck going well into the triple digits he or she can read my documentation and gain access to the system.
Not the best solution but it works since they refuse to hire me help even though I am way overworked increasing the likelihood I will kill myself traveling to and from work at all hours.
Re:God complex (Score:3, Interesting)
Now - it is also important to understand that the IT department isn't some fringe function of a company that can be handled and accessed at will, it is today the backbone of many organizations and as important as the accounting division but much more complex.
This means that you must have a reasonable way of handling the IT department. But it is also necessary to check that a single person can block the whole solution. The latter is virtually impossible to resolve since physical access to servers will allow any individual to obtain full control over that server.
And don't forget that it doesn't help to reassign functionality to a security department, that will only move the problem.
The best solution is to keep the IT department content and be in tight cooperation with them. Dictating orders and hard central management will result in less than happy IT personnel.
Central administration of a company may on the paper look like it's efficient, but unfortunately this also means that instead of disturbances at a single office the whole company will be at risk of total standstill.
Re:It will happen again, and continue to happen. (Score:3, Interesting)
I mean think about it, do you think that there was just one person hired in all of SF to manage the network? Exactly, there were people getting paid and not producing. People giving up their freedom in return for promises of stuff without effort. (AKA socialists, communists, freeloaders, hippies,but not all hippies, some of my hippie friends are cool, etc.)
Those are the people who should be in jail. While their laziness or unwillingness to learn/question did not produce Childs, it allowed him to get out of control.
Re:It will happen again, and continue to happen. (Score:3, Interesting)
Exactly!
I learned a long time ago that there is more value in producing a simple, robust, reliable, and reproducible environment than spending all my time and energy milking 10% more out of a configuration that no one else will be able to understand or maintain.
If your system is so complex that someone of half your ability couldn't be trained to maintain or operate it it, you are incompetent. Experience is knowing the best way to support the long term goals of your environment. Experience is not about being able to make an environment that you will be stuck maintaining for the rest of your life.
Re:God complex (Score:3, Interesting)
This is exactly what I do too - only, in addition to passwords the document is about 4 pages long and lays out everything someone coming in from the outside would need to know to run our network and servers. It is kept in the safe of our Managing Director.
I think a lot of people just don't understand this Terry Childs story. I know a lot of situations like this where one person in IT has all of the administrative control.
I feel for the guy, and think that, possibly, there may even be more to the story. I am glad we heard more about what was really going on from that person who knew the situation well - but I would like to hear Terry's side really.
Duh (Score:3, Interesting)
I've written this one before.
When you have IT people, they're going to have control of your IT infrastructure. Sorry, but there's not much you can do about that. They need access to your data and your equipment to do the job that you want them to do. You'd better find trustworthy people.
This is kind of like complaining, "I have a chaffeur, but I'm nervous that he might go crazy some day and drive me off a bridge, or head-on into a semi." Yes, that is a risk that you'd face by having a driver. And I'm sorry, but no amount of technology gobbledy-gook is going to prevent disaster if your driver does, indeed go crazy.
You face risks whenever you have someone do something for you -- that they might do it wrong, or that they might try to screw you. You're giving them control of some portion of your life. If you're not okay with that, or you don't trust the person that you've hired, you'd better rethink whether you're in the right business...