SF Not an Exception In Giving IT Too Much Control 245
CWmike writes "The city of San Francisco's IT department is certainly not the exception when it comes to allowing just one person to have unfettered rights to make password and configuration changes to networks and enterprise systems. In fact, it's a situation fairly common in many organizations — especially small to medium-size ones, IT managers and others cautioned in the wake of the recent Terry Childs incident."
It will happen again, and continue to happen. (Score:5, Insightful)
Sadly I think the only thing one can do with things this size, is appoint someone and pray he isn't chaotic evil.
Here come the elephants. (Score:5, Insightful)
I forget who said that "an elephant is a mouse designed by a committee." Sure, you can get paranoid about network design and control, and give the job to a committee. But that is going to be really clumsy.
The issue here really is not about size of the design team, it is about vetting the guy who does it. ( The guy who is in charge of the network for my business is someone who I really know and trust. He was best man at my wedding. )
Re:It will happen again, and continue to happen. (Score:5, Insightful)
No, that doesn't work. What if, instead of just refusing to divulge the password, Childs had shot himself in the head or gotten hit by a bus or something. He locked down his network so well that only through a password that was only in his head could anyone have admin access.
You asked for it, you got it. (Score:5, Insightful)
Business Mad Libs (Score:3, Insightful)
Yes, this is prevalent. Unfortunately, no, it has precious little to do with IT.
This quote from TFA is quite true, but universally so. Let's play Business Mad Libs:
"Single points of failure are always bad," said John Pescatore,
an analyst at Gartner Inc. "There should never be one person who is
the only person who knows ____ MISSION CRITICAL INFORMATION ____."
Companies need to make sure there are at least two if not three people
who share the knowledge of ____ BUSINESS PROCESS______. "As a minimum,
require it to be documented and stored somewhere if personnel
limitations say you can't have personnel with overlap," Pescatore said.
Have fun playing the accounting, regulatory, legal, and R&D versions, just for warm-up.
Now, if the business managers weren't smart enough to either know this applied to IT as well as their other divisions, or not smart enough to not recognize that that they needed outside advice on how to apply business rules to IT - well, you have to wonder how well the other parts of their businesses are running.
This is silly (Score:5, Insightful)
Of course there will be people in IT who have power, and of course that power can be abused.
Somebody at a television network has the power to broadcast rocking horse porn if they want to as well and there is no time machine to unrock that horse.
The articles hypes up one person being able to abuse power as if it were unique to IT and suggests a remedy that more than one person should have this power, as if this had any bearing on anything, e.g. the ability for the abuser to simply revoke access to others. What, somebody else should be assigned the exclusive ability to revoke? Then that person is the potential abuser. This is silly.
What "incident"?? (Score:5, Insightful)
So where is the "incident"?? What did he do wrong?
By law he might have done "wrong" by not relinquishing the passwords immediately. But by the people of San Francisco, he may have saved them a lot of trouble and headaches. So, he was faced with a dilemma: obey the law, or do the right thing.
Sad.
Re:God complex (Score:5, Insightful)
The subject of the article is about one central admin having too much control over too many machines, and the risks that entails when they go bad.
Which makes a person wonder... how much worse when billions of consumers are giving total control over all their machines to a centralized authority through Trusted Computing and Vista?
I mean, what happens when Microsoft goes bad?
You say potato... (Score:3, Insightful)
Here's a simple solution... (Score:4, Insightful)
Re:What "incident"?? (Score:1, Insightful)
A Lesson from Star Wars (Score:5, Insightful)
Others think of the bus rule, what happens if the guy who knows everything about mission critical infrastructure components gets hit by a bus?
That is why I have taken a page from the Sith Lord Darth Bane and apply the rule of two. When I build a network I teach and train one apprentice. Then if they suck I fire them and hire a replacement, but if they are good, when I get bored and decided to move on, I feel confident they can take on a apprentice themselves.
It is neat, clean and simple, better still it doesn't have the rules and complexity of Jedi type systems requiring me to check in docs to a source control system, report changes to managers what don't understand, have managers that don't understand sign-off on things they don't understand and avoid dumb rules like not being able to train techs that appear to old, etc.
Yeh, if you ask me the Republic, I mean Network as a whole is best off with Sith types in charge versus bureaucratic Jedi types.
The Childs story stinks like five day old fish (Score:5, Insightful)
Re:This is the best way, anyways.. (Score:3, Insightful)
Heheh... heh... it's kind of funny... you can't network people to work on a network.
Re:It will happen again, and continue to happen. (Score:3, Insightful)
On the other hand, all these access and relevant documentation was sealed and under lock with the instruction only to be retrieved in the event something happened to me (accident, incapacitation, death, etc.).
Not wanting to give out critical information to anyone is something (most of the time at least) responsible to do. Not assuring continuity or failing protecting the critical information to be lost through unforeseen circumstances, shows a serious lack of professionalism.
Re:Here's a simple solution... (Score:3, Insightful)
a) that doesn't work in ANY small and probably most medium sized businesses
Small business,no, but then again most small business's, if they do have a network, is well small and not a big deal. I used to setup networks for small companies, most are ad hoc, no dedicated server types, where everyone has admin privileges. A medium size company should be able to do it. As long as you another IT person you can separate the duties amongst them. Hell, I'm one of just four InfoSec people and we share all responsibilities and admin rights.
That's not all they're asking for (Score:5, Insightful)
"You asked for it, you got it." and you are spot on because if they don't correctly assess this current situation, and assign blame to the deserving names, then they are only 'asking for it' to happen again and again.
Re:What is it with government IT management? (Score:3, Insightful)
Thats because only the government related ones concern the public. This stuff happens all the time in the private sector. However, private companies can die, the government cannot (as much as some people around here would like it to)
Less control... how about more staff? (Score:5, Insightful)
Seems to me that in many cases, the IT department may be rather grossly understaffed (either in terms of # of staff, or # of experienced staff).
Many places I've worked end up with a Lord-of-all-IT situation simply because they haven't got anyone who can replace him* or back him up, or weren't willing to pay for backup/additional/experienced staff.
* male gender used for convenience purposes.
Banks deal with this (Score:5, Insightful)
One of my first jobs was a bank teller. Our passwords were sealed in an envelop, which we initialed, and locked in a vault which needed two keys to open.
If the two officers needed my password, they'd open the vault, open the envelope, breaking my seal (letting me off the hook of responsibility).
IT has to learn from banks.
Re:God complex (Score:5, Insightful)
It's not just when they go bad. What happens if they get run over by a bus or a stampede of wildebeests? If they are the only person to know the admin passwords, commands, etc, they are the single point of failure, regardless if they go bad or not.
Just as we harp on backing up our files (um, yeah), we also need to harp on a backup for the admin. There should always be someone else, even if it's the mayor, who also has the list of admin passwords.
It depends on who the "one person" is (Score:3, Insightful)
It really depends on who the "one person" is. Committees rarely design good crypto algorithms or protocols, for example. On the other hand, if you just pick the "one person" at random, you risk picking the wrong person.
I guess it's sort of like picking a dictator. If you pick the right person, and hold that person accountable, they will get things done more efficiently than a committee. If you pick the wrong person, they will get the wrong things done more efficiently than a committee.
Re:God complex (Score:3, Insightful)
Re:Here's a simple solution... (Score:1, Insightful)
Yes, we have the same thing here. But anyone of you (or us) can lock down the entire system the same way this guy did. There is no perfect solution. Anyone person who can change passwords can screw up either the entire company or their section of the company they have control over.
In the end it comes down to requiring complete trust to the people running your network. Even if you had "audit" people like the military supposedly does from what I've read, you're still at least one step behind the person able to make the changes. Sure you'd get caught, but if you wanted to screw something up, you still could.
Unless your system has in place a mechanism so that Jim creates accounts and Joe has to push OK to enable the account or Joe can change a password but Jim has to push OK to enable the actual change there is no getting around the ability of a network admin being able to screw the company, city, county, etc over. I have yet to see a system like that. Routers, Windows, Linux, et al don't support anything like that that I've seen. If the person can change passwords the change is immediate and isn't forced through some sort of red-tape.
Re:The familiarity in this story isn't just the IT (Score:4, Insightful)
Supposedly that's it, according to some of the articles. He thought a lot of the others were screw-ups, so he kept access to himself. Everyone seemed to know it, as well, right up to the top of the IT organization. A new security person was hired, and that person didn't like the situation (may have come up during some sort of review). They made a point of asking him for the passwords, which he interpreted as "hey, we want to screw up the network - you know, the one you feel really possessive about" and refused. Didn't seem to recognize the authority of whoever delivered the message (don't know if it was the new security person or not). They then sent the police after the apparent master criminal.
Also, while they couldn't make configuration changes (that's what "locked out" meant apparently), the network continued to run, even without his intervention. So he might've been a doofus about this issue, and for all I know a total jerk with no people skills, but it sounds like (crazy access issue aside) he knew his job pretty well.
I suspect the new security person (who for all we know is more of a policy person than a technical person) handled it badly on their end as well, and may have gone for a club (formal meetings, demands) when a lunch conversation might've done the trick. The guy shouldn't have held onto exclusive access, but it sounds like the security person didn't handle it well. Apparently, that individual now fears for their safety, which I suspect is either an overreaction or a further attempt to demonize Childs to make it seem like whatever actions taken are justified.
Re:God complex (Score:5, Insightful)
As far as I'm concerned... passwords are just the beginning. Configurations and such can also be a nightmare to replicate when they're undocumented. Ever stepped into a project where they only guy working on it is gone and you have to figure out how to setup your machine / development environment just to get it to run? It's awful. All of the "don't install that patch, it ruins everything" or "you have to install these components in this order so that they don't interfere with one another" is gone and you have a horrific puzzle before you.
Re:It will happen again, and continue to happen. (Score:4, Insightful)
Re:Banks deal with this (Score:3, Insightful)
The problem was not that he was the only one with access, although that is an issue in small IT departments. No, the problem was that he had enough access to change all the other administrators' passwords. Lots of people had access to the systems, and there were probably procedures in place to name a successor in the event that Childs was fired or hit by a bus. Instead, Childs changed everyone else's password and locked them out.
The only way to protect against that type of an attack is to make the Administrator-level access much more fine-grained. One admin should not be able to change other admins' passwords. In practice, that might take the form of a global login server (with appropriate backups) that is not under the control of the admins, or maybe a good SELinux-style setup. The idea is to never have a "global root" role, but instead break it down into "config root" (for admins) and "access root" (for people who administer user accounts and such).
This is obviously much harder in smaller setups who might not have the personnel to split things up that way.
Re:It will happen again, and continue to happen. (Score:1, Insightful)
What you're forgetting is that he *offered* the passwords, since the day he was arrested - just to somebody he trusted. I'll bet if he were dealing with you, I, or most of the low numbers on /., Childs would have felt confident turning it over to us after a few minutes of talk to assess our sanity. But since you've done a few configs, I'm sure you can think of one or two people you've met in the workplace who shouldn't be trusted with car keys, let alone passwords.
It isn't just him being overprotective, either. He is still a citizen of S.F., and if it were me, I'd avoid having the whole thing crash and incite riots and anarchy until I got someplace farther away where it couldn't bother me. Like, say, the Bahamas.
Re:Here come the elephants. (Score:3, Insightful)
It's about having the guy who knows everything to document it all.
I used to be that guy who knew the entire project. I thought it was crucial for me to know everything, so I remembered everything. All the minutiae, I knew back and forth. No single person on the team had that breadth of knowledge. Thus, I was working all the time.
Then I started to document everything in memos. I sent them around. I recorded everything. It took a whole bunch of time, but it was the best investment ever. I could delegate my work more effectively. There was a paper trail of everything, great records.
Don't be that guy!
Re:God complex (Score:3, Insightful)
Re:God complex (Score:4, Insightful)
I came into the same philosophy as you a few years ago when I was in the position where I took over a network that was completely undocumented. Now I have Visio diagrams and written explanations of almost everything including a complete inventory of what I have on what network at each site.
I started it with the idea of the bus principle but I've come to rely on it myself as I'm the only admin and so I often have parts of the network I don't touch for a year at a time. This means I forget how things are put together so I refer back to my own documentation. Works every time.
Re:It will happen again, and continue to happen. (Score:4, Insightful)
I do not agree, but even if what you say is true, that just goes to show that efficiency isn't everything!
In the enterprise space, it simply isn't possible to have one person design, implement and operate a non-trivial environment. There aren't enough hours in the day to do all that is required, and I (for one) would like to have a bit of free time - even if all I do with it is sit in front of my playstation.
If the work is so complicated and the deadlines so tight that only one person can pull it off, the project is a disaster waiting to happen. Truly competent technical staff would be the first to escalate that situation to management, rather than indulge their inner megalomaniac and try to do it all solo.
I think we nerds tend to focus on the really cool technology so much that we fail to see the big picture. When you step back a moment, and put it all into perspective, it does not matter if I work 7x24x365 to complete a really complex project on my own. It _really_ does not matter if the design is incredibly elegant, the implementation flawless, and the cut-over into production goes as smooth as silk if one mis-step in front of a speeding bus renders the whole thing an unsupported mess the first time it breaks.
While some might mourn my passing, the lack of documentation and shared knowledge and experience will have reduced all of my heroic efforts to a complete and utter waste of time.
My obligation to my employer isn't to hoard knowledge and information to myself - it is to share that knowledge and information with the other members of my department. If I bring everyone else up to speed, I can have a few week-ends to myself because even the most junior member of the team can step up and help resolve problems if the knowledge base and procedures are thorough and well-documented.
Coming from that perspective, I am unable to find much sympathy for Childs or his former employer. Both have demonstrated extraordinarily poor judgment and are paying the consequences for their lack of professionalism.
(I have to say though, I'm not sure Childs deserves to be in jail, or face such an absurdly high bail amount.)
We nerds tend to interpret "productive" differently than management does. I'm sure most would agree that sitting in front of the keyboard actually doing UNIX-related work qualifies as productive. Management might place a higher priority on documentation, or training other team members as equally if not more productive tasks.
I work in a large enterprise environment and most of my time is not spent at the command line. Most of it is spend communicating with other departments, with my own department, with project teams I have been assigned to, with various levels of management, and with vendors. While I would not have thought so at the beginning of my career, I now see that effective communication skills (which includes listening to others) plays a greater role in being successful at my job than what I do at the command line.
Re:It will happen again, and continue to happen. (Score:1, Insightful)
This is a ridiculous defense. If his superiors demand that he release the passwords, and unless there's some protocol explicitely in place requiring him to only release the passwords to specific individuals, he is obliged to do as he is told. Whatever his justifications, he is not permitted to deny his superiors this data. It isn't his call to make.
Not if he believed that he would be putting the public welfare at risk. It's a legitimate defense. In addition, he's not in the military so all the hoohah about protocols is just bunk. Once he was fired, his obligation to the city was terminated. Only if there were some stipulation in his employment contract spelling out a requirement to return all city owned property upon the end his employment would there be legal standing to take action. Passwords could fall under that requirement, however, it would be a civil matter and not a criminal one. The clown of a prosecutor in this case, has, like many other prosecutors in this country, taken an overly broad interpretation of a vaguely written law and misapplied it as a state sanctioned method of forceful coercion.
He'ss a prima donna with a god complex. He's no hero. He may be the best network admin in history, but I wouldn't hire him in a million years, and I have a feeling there are a lot of guys out there that won't. I expect that his career is probably over, because his name's all over the place, and who the fuck would want some lunatic who won't release passwords to superiors, save for the one he trusts?
You'd be an idiot not hire him. He fucking designed a top-notch network that is so secure that even engineers from Cisco couldn't get access into it. That's pretty damn impressive. I think he'd really shine if you stick him with some other engineers that match his skill-level and competence and where he wouldn't have sole responsibility for running the network. Working with municipal morons was probably depressing as hell. Hell, I bet he could win on an insanity defense.
Re:God complex (Score:3, Insightful)
Yeah, I imagine he was aware of a lot more than most other people as admins usually are. I know that I have much more information about the company and how it operates along with its goals than I necessarily need to do my job but it's the nature of trust.
You have to be able to trust your admin so you should treat them accordingly. That is the first mistake of most employers these days. They treat everyone like dirt including the people that can burn them really badly because they don't understand how much their company relies on IT. I know the company I work used to come to an abrupt halt when there was an outage. Since then I've removed the single points of failure, the only thing left is me. They forget that redundant systems get kind of complex though and they assume anyone out of college can do it for 30k so they fight me for 70k.
Re:God complex (Score:1, Insightful)
Doing some intelligent CYA (Score:1, Insightful)
I ran into exactly the same situation as Terry Childs in my short time (about two years) working for a municiple organization.
The difference, however, was being more aware of how stupid people are. For one thing... never lock your boss out of the system. Since there were so few IT policies in place prior to me getting there, it gave me quite a bit of leverage (at least early on) toward getting ones in place.
The first thing I did was change the top-level account password. The password I changed it to was completely meaningless gibberish, which was written down onto a piece of paper and placed into a sealed envelope, which was entrusted into the care of the CIO-equivalent position. I told him it was for emergency use only, and it needed to be treated as the most important piece of information he had... which it was, in the practical point of view. In my time there, it was never used.
Afterward, there was a lot of whining and moaning about people who wanted access... so I got to work on logging. All changes were logged, so accountability was in place (at least, as good as it could be. I kinda made it seem like far more than it was), and all specified people were given special administrator accounts (I detest elevating access on a person's everyday use account). From what I recall, none of those people ever used the accounts they had whined so hard to get, because they knew their activities would be logged (although honestly, not logged as much as I explained to them, but that was for everyone's good).
The problem with many of these people was that they viewed the network as a toy which they could play around with to learn... whereas myself and the qualified staff viewed it as a crucially important business asset which needed to work no matter what. So scaring the tinkers by making them know they would be held accountable for any stupidity on their part made them content to only mess up their own work PCs, rather than the network.
It's amazing what a great deterant accountability is!
After reading the REAL story of Terry Childs, it was hard not to feel sympathy for him. Municiple organizations don't really take many things seriously, and don't have many people who have worked in "real", private sector, IT jobs. Many are either right out of college, transfers from other (non-technical) departments, etc, people who don't really view IT as their career, or do but have no experience working in an enterprise IT environment.
The things he was doing are typically managed by an entire department... and that's often the case in public sector IT. I would LIKE their departments to be run the same as a normal enterprise IT shop... but when you have to deal with politics, where's just no political will to do so. Governmental IT is viewed as an expense rather than an asset, and generally an expense which they try to spend as little on as possible. The idiotic conservative "SMALLER GUBMENT!!!" lunacy doesn't help either, since all it does is guarantee nothing can ever be done in a proper way.
So while I can sympathize with him... he could have been more politically aware. The people who were asking for access, had they thought they could get fired for screwing something up, likely would have never used that access. They only wanted it because they didn't have it.