Documenting a Network? 528
Philip writes "Three years ago I was appointed as a network manager to a barely functioning MS-based network. Since then I've managed to get it up and running — even thriving — but have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come. It looks like I'm going to have to document the network with all sorts of details that outside consultants could understand too (no, I don't want to be the outside consultant), especially since it's likely that my replacement will have less technical expertise (read 'cheaper'). Are there any good templates out there for documenting networks? Is anyone who has done it before willing to share some experiences? What did you wish your predecessor had written down about a network that you inherited?"
Good News (Score:5, Interesting)
Your successor will never find any documentation that you leave behind (or if you show it to them they won't bother with reading it) and by the time they notice it they'll have already screwed things up to the point where the documentation will be obsolete. This means you can save yourself the trouble of doing the documentation unless that documentation is going to make you more effective while you're there.
Documenting teamwork (Score:3, Interesting)
Alternatively... (Score:2, Interesting)
Professionalism be damned, we're in the middle of a recession and you're wanting to make yourself dispensable? Go ahead by all means, but pass your boss my number :)
Let's be real (Score:5, Interesting)
Short answer: don't worry about it too much. Put together enough that it looks like you've done something then go have a beer.
You could have the most amazing docs the world has ever known - with passwords and clear instructions - ad the odds are about 20% that the next guy will even read them.
The next guy will figure that he/she knows much more than you as evidenced by the fact that they are there and you are not. And, the cheaper they are (read: inexperienced) the more likely this is to be the case. When things go wrong, they will blame you anyway.
So document away, but for YOUR sake so that if/when you are called in after the new guy horkens everything, you can have an easy time putting it all back together. But don't wait for the call... people will put up with almost anything when pride is on the line.
And go have a beer.
setup a wiki (Score:5, Interesting)
At my last few companies and my current one that I work out, one of the first things I do is setup an internal only Wiki server.
Not only does this let me document everything I can about the network but I also try an train my co-workers in using it to document information they feel is important for their job too.
The effectiveness though seems to be related to the level of computer literacy of the user, i.e. my last company the software developers went crazy with it and documented everything they could think of. But other than them or us in the I.T. dept, no one use would hardly touch it at all.
Either way it's still a simple method for your to store notes, diagrams or any information about your network in an easy to find single location that you feel would be important to the company should you leave for any reason.
Re:I know... (Score:5, Interesting)
This may sound funny, but I recently had the same experience. I took over the position of CTO of an electronic payment company, and after one week, I figured a lot of critical systems are missing root password, including Linux, AIX, HP/UX and SCO Unixware. No one knows the password, it's been changing hands so many times, and the people who were responsible for those machines have left, without leaving the passwords behind.
Those are critical systems that must run 24x7. We had to rebuild the system on new machines, re-route transactions to the new machines, and shutdown the old ones to recover (single user mode).
And that's a platform handling over 400 billion in transaction per year. Scary. But that's the easiest problem I have inherited, mind you.
The Basics (Score:0, Interesting)
Passwords!
Vendor support numbers
Circuit ID's
Customer / Client numbers that support asks for
Links to KB articles that have resolved problems in the past.
When where and what the backup system runs. (You do have a backup right?)
Re:I know... (Score:3, Interesting)
Re:Windows != SPAM (Score:1, Interesting)
If every copy of Windows on the Internet somehow magically disappeared, the SPAM problem would not abate. Bot herders and spammers would simply shift their efforts to other platforms.
Yes, and even though no one has ever done the experiment, if you got rid of religion the world would be a better place. Man, I could go on all day theorizing about hypothetical situations.
Okay, enough foolin' around. Here's what we do. Everyone in the world can shut down their windows machines for one week and then we'll measure the spam. That should settle the issue.
The overlooked parts (Score:4, Interesting)
There are a few things that are often overlooked and outright forgotten when documenting networks. I had to take over a few networks, let's see what I usually miss:
Every admin remembers to hand over passwords. Except for the routers.
Routers and other "managed black boxes" are notoriously being left out from the list of passwords. Fortunately, more often than not it's the standard password because "nobody has to touch them but me anyway" (ignoring that, if people only touched what they should, passwords would be moot...)
Every admin remembers to draw you a network layout. They don't tell you WHERE those switches physically are, though.
In large companies (read: Lots of room to cover, independent of the number of people working there), this can indeed be a problem. Especially when there's not one single server room where everything is collected, when you have switches and routers hidden in cupboards and other "innocent looking" furniture, cables that appear out of nowhere and disappear into walls, without an indicator where they surface again. Or what purpose they serve, first of all.
What HAS to be documented is the reuse of resources
That's the worst of the "undocumented changes". When you find a switch that shouldn't be there, you know you have to investigate, you know something wasn't documented. When you find a certain box sitting where it is supposed to be, you don't investigate. You expect it to do what it allegedly does. If it does not and has been "recycled" to fulfill another role, the whole documentation goes out the window. Because now you start questioning EVERY piece of hardware.
Re:I know... (Score:3, Interesting)
So what happens if said predecessor gets hit by a bus, has a heart attack or a stroke and can no longer tell you the passwords?
Boot single user and reset them?
Yes, I realize that doesn't scale, but it works fine in a lot of small environments.
start with nmap, then wiki (Score:3, Interesting)
2. set up a wiki
3. paste the results of the portscan into the wiki
4. start writing about everything that showed up
i've actually done this before with a pretty high degree of success, pm me if you want some help setting it up
Re:Lots of flowcharts! (Score:3, Interesting)
I find the more long words in someones job title, the less useful the function they actually perform.
Good title you have there.
Documentation of Networks (Score:3, Interesting)
My favourite technique for making sure documentation is done and updated, get the new guy to do it. Then he/she has to go all around the campus, locating servers and getting serial numbers form all sorts of odd equipment and making sure all of the support aggreements are current and the contact details for the vendors are accurate.
The other favourite is if I find new equpment that has been installed and is not labelled or documented, I get the installer responsible to audit all similar equipment to make sure there are no other ones missed out. After haivng to crawl around dozens of risers and labelling or confirming all switches etc are correct and documented, they don't often make the same mistake twice.
We also have a password management system which also allows details like how to install the management console or the URL to access a system for management to be stored.
My answer to any question about "What is the password for X", or "what the hell is the name of the server for X applicaiton" is "Its in the store" Then if it isn't, we add it 8) Only takes a few times for the newbies to start looking up the information themselves first.
The other key file is a massive Visio document with a summary page with a managment style overview, and then a document with everyhting in it in layers like an electircal diagram or building plan.
Lay in the workstaitons VLAN, the switch management VLAN, the Servers VLAN, link to things that are self contained like all of the Firewalls and DMZ configurations.
etc.
Re:Lots of flowcharts! (Score:1, Interesting)
Docs don't help sometimes. I'm also a consultant. The present client is a govt department with approx 9000 users. They have 6 eDir trees connected by IDM with different trees being authoritive for different things. They have 5 CAs in production alone. They have two seperate AD forests, one used to manage servers, one to provide users and groups for a sharepoint implementation. Users are not synchronised to the ADs. Apps are distributed by Novell, except for the ones distributed by a citrix farm tied to the 2nd AD implementation. They use notes for mail, and have approx 4000 notes apps in existence. They have NO single sign on at all. It's all fully documented. It just doesn't fucking work.
Re:I know... (Score:4, Interesting)
Or how about a new CTO pissing both admins off and one walking out as the other was fired for no good reason.
I've had to walk in behind something like that several times and reset the passwords or load the password hashes into some cracker in order to find the passwords. A lot of times, you can pull them from workstations the old admins used and they are easier to crack then the newer MS servers. The funniest thing is that the CTO usually wants me to stay on full time and gives me a dirty look when I won't do it because he made life so miserable that two other people walk off the job under his supervision.
ITIL...! (Score:1, Interesting)
ITIL...ITIL...ITIL...
Why has no one mentioned this standard yet? Is it because it's European (British) and the US won't use anything not invented there?
The Europeans have already got a complete set of standards for deigning, buildng and operating computer systems properly. USE THEM!!!
Re:I know... (Score:5, Interesting)
Yes, it's easy to open, but you'd know whether someone tried to tamper with it.
Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)
Learned this one from a history of the U.S. Black Chamber [wikipedia.org].
Re:I know... (Score:3, Interesting)
Re:I know... (Score:2, Interesting)
If the password can be worked around, what does it matter if it gets written down in the first place?
Because if the password is written down, then in (most) windows shops you can login remotely as administrator (or get administrative rights while logged in as a non-priviledged user).
GP did say Boot single user, which entails physical access to the system.
While effectively any system to which there is network access can be compromised (eventually), every system to which you have physical access can be compromised (and that, rather trivially).
Re:I know... (Score:3, Interesting)
If you are primarily a Linux shop you might use Coda or NFSv4, but if you still have Windows clients you might also run Samba. Actually, Samba can serve CIFS and not just SMB; CIFS has extensions for Unix features. So it's not a bad choice for serving files to Unix, but it's no NFSv4. (In some ways, that is a good thing.)
Re:I know... (Score:1, Interesting)
Promotion is not always a good thing.
Re:The MACK(TM) Truck Rule (Score:3, Interesting)
We used to have a similar 'BUS' rule (ie, what if so-and-so got hit by a bus) until someone we all knew got hit by a bus. That sucked, he was a good guy and we had just referenced that joke a week earlier.
Now we have the 'LOTTO' rule (ie, what if so-and-so hit the lottery and left the company to be independently wealthy.)
We all miss Dave. And most of us secretly wish he had documented his fucking code before getting hit by that bus.
Re:too busy to document the network (Score:3, Interesting)
And exactly how long ago was that?