Slashdot Log In
Documenting a Network?
Posted by
kdawson
on Tue May 26, 2009 12:15 AM
from the what-matters dept.
from the what-matters dept.
Philip writes "Three years ago I was appointed as a network manager to a barely functioning MS-based network. Since then I've managed to get it up and running — even thriving — but have been guilty of being too busy with the doing of it to document the changes and systems that were put in place. Now as I look back, I'm worried that I am the only one who will ever know how this network works. If I get hit by a bus or throw in the towel for any reason, I'd be leaving behind a network that requires some significant expertise to run. Ultimately, this won't be a good reference for me if they are trying to work out technical details for years to come. It looks like I'm going to have to document the network with all sorts of details that outside consultants could understand too (no, I don't want to be the outside consultant), especially since it's likely that my replacement will have less technical expertise (read 'cheaper'). Are there any good templates out there for documenting networks? Is anyone who has done it before willing to share some experiences? What did you wish your predecessor had written down about a network that you inherited?"
Related Stories
Submission: Documenting a network by Anonymous Coward
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
I know... (Score:5, Funny)
The Passwords.
Re:I know... (Score:5, Insightful)
Whether the comment was intended to be funny, I find this to actually be a serious issue...
Parent
Re:I know... (Score:5, Insightful)
Whenever an important password was changed, I would hand over the new envelope
Parent
Re:I know... (Score:5, Insightful)
If the predecessor does write the passwords down, he deserves to be fired.
That's knee-jerk stupidity, and you should be ashamed of your non-thinking fundamentalism.
Passwords do need to be written down, and stored in "escrow". I put the list of passwords in an envelope, lick seal it, sign and date the seam, and then seal it again with clear packing tape. Give it to the boss to put in his safe.
Yes, it's easy to open, but you'd know whether someone tried to tamper with it.
Parent
Re:I know... (Score:5, Funny)
That's knee-jerk stupidity, and you should be ashamed of your non-thinking fundamentalism.
I just assume that any event capable of destroying my ability to transmit said passwords to my successor also destroys any ability to give a damn about my job. Problem solved.
Parent
Re:I know... (Score:5, Funny)
So what was it like working for the City of San Francisco, anyway?
Parent
Re:I know... (Score:5, Interesting)
Yes, it's easy to open, but you'd know whether someone tried to tamper with it.
Try spraying the envelope with refrigerant. The paper becomes translucent when wetted and you can sometimes read what's inside, and then it dries without a trace (unlike wetting it with water, which swells up the paper fibers leaving the telltale signs of tampering.)
Learned this one from a history of the U.S. Black Chamber [wikipedia.org].
Parent
Yes... I use dust-off (Score:5, Informative)
the cans of compressed air in every office supply store? inverted they throw out a very cold liquid that does exactly what you describe.
Parent
Re:I know... (Score:5, Insightful)
Parent
Re:I know... (Score:5, Insightful)
Either he writes the passwords down, or he uses weak passwords that a human mind can remember.
Besides, a password is a security token. A piece of paper or a little plastic card with the password printed on it or a USB stick with SSH key or whatever saved on it are also security tokens. They aren't inherently less secure than memorized passwords; you simply have to secure the physical object by, for example, locking it in a safe.
"Passwords should never be written down" is an idiotic rule, right there with "never use goto".
Parent
Re:I know... (Score:5, Funny)
If you think that is ironic, then note that the story poster is worried his ability to get a job if he dies in a bus accident.
Parent
Re:I know... (Score:5, Funny)
If you think that is ironic, then note that the story poster is worried his ability to get a job if he dies in a bus accident.
maybe he could work as a voter in Florida?
Parent
Re:I know... (Score:5, Interesting)
This may sound funny, but I recently had the same experience. I took over the position of CTO of an electronic payment company, and after one week, I figured a lot of critical systems are missing root password, including Linux, AIX, HP/UX and SCO Unixware. No one knows the password, it's been changing hands so many times, and the people who were responsible for those machines have left, without leaving the passwords behind.
Those are critical systems that must run 24x7. We had to rebuild the system on new machines, re-route transactions to the new machines, and shutdown the old ones to recover (single user mode).
And that's a platform handling over 400 billion in transaction per year. Scary. But that's the easiest problem I have inherited, mind you.
Parent
Re:I know... (Score:4, Funny)
So that's what happened during the bank collapse!
Parent
Re:I know... (Score:5, Funny)
Check the top drawer of the your desk where the paper clips are lying normally.
Oh? It is locked and you dont know where the key is?
Parent
Do what the guy before me did (Score:5, Funny)
Re:Do what the guy before me did (Score:5, Insightful)
The next step up from a Post-It, though, is a snazzy label-maker. My portion of the company uses these extensively to document our development lab (we do some NMSy stuff). Of course, it's not a production network, and standards are a little different.
Parent
Good News (Score:5, Interesting)
Your successor will never find any documentation that you leave behind (or if you show it to them they won't bother with reading it) and by the time they notice it they'll have already screwed things up to the point where the documentation will be obsolete. This means you can save yourself the trouble of doing the documentation unless that documentation is going to make you more effective while you're there.
Re:Good News (Score:5, Insightful)
This is more to the point. Most network admins have the attention span of a flea and won't read past the first sentence of anything you write; actually, I could probably expand that statement to include most people generally. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Parent
Re:Good News (Score:5, Funny)
You just thought I wouldn't catch a reference to Cicero's De Finibus Bonorum et Malorum.
Original Lation from which it was derived: ...neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet, consectetur, adipisci[ng] velit, sed quia non numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?
At vero eos et accusamus et iusto odio dignissimos ducimus, qui blanditiis praesentium voluptatum deleniti atque corrupti, quos dolores et quas molestias excepturi sint, obcaecati cupiditate non provident, similique sunt in culpa, qui officia deserunt mollitia animi, id est laborum et dolorum fuga.
English:
Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find fault with a man who chooses to enjoy a pleasure that has no annoying consequences, or one who avoids a pain that produces no resultant pleasure?
On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain.
What was that about my attention span?
Parent
Re:Good News (Score:5, Funny)
...fellatio uber alles...
WTF???
Parent
Re:Better News (Score:5, Insightful)
Not really, because as most high level executives know, IT doesn't really do anything!
Parent
Re:Better News (Score:5, Insightful)
Parent
Re:Better News (Score:5, Insightful)
Yeah, I think that's more or less true. At one of my previous jobs, I had a guy try to imply that I didn't deserve my pay because I "wasn't doing much". When I asked him what I should be doing, he said, "It's just that you have a really easy job. The IT guy at my last job had it much harder. He was always running around, fixing things. You just sit at your desk because nothing ever breaks."
I can't remember now, but I think I might have done a literal facepalm right then. I said something like, "Has it occurred to you that, if you think none of our IT stuff ever breaks, I must be doing a good job? If the IT stuff at your last job kept breaking all the time, he was doing a worse job than me?"
Parent
get some help (Score:5, Insightful)
What about? (Score:5, Informative)
Re:What about? (Score:5, Insightful)
A summary diagram (dia/visio) is still pretty key, but nmap (or specifically the latest versions of zenmap (the gui frontent to nmap) provides a good detailed view of network topology and if you include the capture file, you can annotate specific computer details in the notes section (ofc you have to assume your replacement will be familiar with zenmap for that to be useful though)
Parent
schematics (Score:5, Informative)
Basic network documentation:
I've found that starting out with the very basic physical layout and working your way up in complexity is greatly beneficial.
i.e. start out documenting network cable runs including cable type. follow it by switch layout. follow that by routers and vlan setups. follow that by the servers that provide basic network functionality(e.g. DHCP, etc...). If this is a windows network, that would likely mean detailing the domain controller setups. From their systematically document the systems in order of importance to the business, etc...
Also, visual diagrams are extremely helpful.
Let's be real (Score:5, Interesting)
Short answer: don't worry about it too much. Put together enough that it looks like you've done something then go have a beer.
You could have the most amazing docs the world has ever known - with passwords and clear instructions - ad the odds are about 20% that the next guy will even read them.
The next guy will figure that he/she knows much more than you as evidenced by the fact that they are there and you are not. And, the cheaper they are (read: inexperienced) the more likely this is to be the case. When things go wrong, they will blame you anyway.
So document away, but for YOUR sake so that if/when you are called in after the new guy horkens everything, you can have an easy time putting it all back together. But don't wait for the call... people will put up with almost anything when pride is on the line.
And go have a beer.
That is not real, is cynical and unprofessional. (Score:5, Insightful)
No wonder our field and many of our professions have such a bad reputation.
I have read only a few posts and two (moded up 5) say pretty much to ignore the issue.
In several networks I have worked with fundamental information was non existent. This translated in lost time, down time and actually losing money (if you lost your job in one of those companies recently, the indolent SAs or Network administrators may be partly to blame).
You never know who the next guy will be, if he is less experienced or capable then the documentation will be very valuable, if he is more experienced or capable then you would have saved their time to do some real work, after all they (and you) have not being hired to do forensics.
How a professional can hide behind the "let's be real" nonsense is beyond the pale.
Parent
Start with disaster scenarios (Score:5, Informative)
For me, visio's are great and everything, passwords too, but really the most valuable thing you can do is document single points of failure, outdated software/hardware, etc., license keys/expiration dates, cert expiration dates, personal support contacts you have and all vendor relationship details as well are essential. Do you use change control? If you do, go back and comment your changes, if not, do the best you can at explaining why things are the way they are. Get some open source software that is good at indexing data and create a searchable knowledge base from the information above. Don't concentrate on docs that can be found on the web at first because any admin worth their salt will know where to look for how to's, etc. Focus on the why's, the where's and disaster recovery.
My two cents...
Re:Start with disaster scenarios (Score:4, Informative)
Get some open source software that is good at indexing data and create a searchable knowledge base from the information above.
A Wiki?
Parent
Here is what I would get (Score:5, Informative)
1. Viseo overview of the network drawing with complex areas drawn out specific detailed viseo's (even a scanned sketch or paint drawing is better than nothing)
2. A spreadsheet with circuit ID's mapped to router and interfaces.
3. Document the trunk interfaces as well as the LAG's (Link Agrregation Groups, port channels, whatever you want to call it)
4. TACACS passwords / domain logins in a secure location (or radius or diameter or whatever you use)
5. Data center capacity as a function of 1. Rack Space, Cooling Capacity, Electrical Load.
6. Write brief knowledge articles describing any problem areas and explaining a history of anything you think would be hard to figure out easily. No need to go hog wild, just re-brand the RCA documentation you have. You do have Root Cause Analysis right?
7. Network protocol hierarchy map. Where are your major redistribution points, what is your routing strategy etc.
8. If you have a voice network document all your DID's, PRI trunks, Gatekeepers, Dial Plan, and any translations you use on h.323 gateways or how MGCP or SIP is configured. If you have a complex call center you should probably pay to have it professionally documented in down to the minute detail.
9. SSID's and BSSID's for any wireless you may have as well as passwords, 802.1x authentication methods, along with linking documentation.
10. Make the documentation part of your CMR process (Change Management Review) and incorporate it into the time allotted for a change.
I know these are just rough ideas and you should get many more ideas from all the smarter people on here than myself, but whatever advice you get I would say you would need to have the documentation update able via subversion, or some document control system and have some kind of review process for it, even if it means getting together over pizza with some of the other groups and asking them about their environment and getting pointers and possibly help on documenting it all. Documentation is a full time effort and IMHO there is no such thing as too much documentation. You would be surprised how good documentation can aid you in problem resolution down the road or aid vendor support in helping you resolve a major outage. The three basic principles of network care are document document document. :)
Cheers,
Anonymous Coward.
Re:Here is what I would get (Score:5, Insightful)
circuit ID ? in that a farmer's synonym for "MAC address" ?
Not everything is ethernet.
Parent
Re:Here is what I would get (Score:5, Funny)
Actually, it's a simple test designed to separate the people who should be working with the corporate wide area network from the people who should be forcefully prevented from coming anywhere within fifteen metres of the comms cabinet, using a taser if necessary.
If you have to ask which group your answer has placed you in then I'm not going to tell you.
Parent
TiddlyWiki (Score:5, Informative)
On the side, I manage a small network, and I've also wondered the same sort of thing: if someone else needed to find their way around, where would they start.
A Wiki makes for a really nice way to document things, not least because you can include all sorts of cross references. For example, a list of servers, with links to the services they provide - and a list of services, with links to the servers. But Wiki's normally run on servers, which leaves your successor with a chicken-and-egg problem.
A bit of random surfing turned up TiddlyWiki [tiddlywiki.com], which is a Wiki in a single HTML file. A really elegant bit of engineering, and very handy for self-contained documentation. Since the entire Wiki is just a single file, it's easy to protect. I wound up with two: one with "public" information describing the general architecture and one with private information (including passwords). The private one you can put on a USB-stick in a safe, hand to your boss, or whatever seems appropriate...
setup a wiki (Score:5, Interesting)
At my last few companies and my current one that I work out, one of the first things I do is setup an internal only Wiki server.
Not only does this let me document everything I can about the network but I also try an train my co-workers in using it to document information they feel is important for their job too.
The effectiveness though seems to be related to the level of computer literacy of the user, i.e. my last company the software developers went crazy with it and documented everything they could think of. But other than them or us in the I.T. dept, no one use would hardly touch it at all.
Either way it's still a simple method for your to store notes, diagrams or any information about your network in an easy to find single location that you feel would be important to the company should you leave for any reason.
Why do people make things hard for themselves? (Score:5, Informative)
Why not use an automated too?
www.open-audit.org
Contact Numbers (Score:5, Insightful)
You likely don't run *everything* and the new person needs to know who to contact when the interaction between inside and outside fails.
I've been doing this for a while (Score:5, Informative)
My job requires me to do exactly what you're looking to do but for multiple companies/networks. Then, as soon as I'm done, I usually pack up and go or get hired in and fix the network.
Since I'm writing the Network Overview for managers AND potential future network managers I tend to write mine in the following format:
1) Synopsis of what the network does for the company, what general technologies they use (Windows AD vs *nix OD, thin clients vs Windows boxes, Cisco vs Brand X), and what the LOB software is.
2) Points of contact for the ISP and other providers (anti-spam, anti-virus, hardware, etc). Passwords for various accounts and services.
3) Logical network overview map (visio), containing firewalls, routers, switches, other devices, open/forwarded ports, IPs, what the servers do, what vlans are in place, Quick explainations for why (such as why vlan vs a seperate subnet).
4) Physical map of devices if the complexity of the network calls for it.
5) Software notes, what apps are critical for the business and which systems they rely on.
Then, for my specific job I have to do the following:
6) Licensing issues.
7) Network weaknesses/points of failure.
9) Other rec's.
Do what everyone else does. (Score:5, Insightful)
Draw a horrible diagram in Visio (or similar) of what's connected where without any indication how it actually works!
Lots of flowcharts! (Score:5, Insightful)
Present client I am at I inherited a network of about 15,000 clients that was previously managed my a very incompetent IT department. Started by looking at the existing flowcharts and discovered that almost everything that was documented was wrong... Long story short I have been spending a fair bit of time reverse engineering their production environment so that we could accurately document it. Unfortunately we had come to the conclusion that we can use /nothing/ that the previous administrators left behind for documentation. You don't want someone like me coming in and looking at your documentation and declaring you incompetent, it can cost you your job.
You haven't detailed the size of your organization to know if you will need sign off from other departments or not. If possible try to get sign off so that they have a reference and you can create a standard that can be used to fix things and to ensure your designs don't get trampled by a new admin in another department. You really need to provide more detail on your environment for people to answer you.
I do most work in Visio, starting at 50,000 feet and working my way down. At this level I need to document network topology, server distribution and database server distribution. I work my way down from there using a zoom in style that has served me well for 30 some clients. Depending on the size, complexity and your area of responsibility you may need to flowchart anywhere from a 2-3 levels to potentially dozens of disparate processes. You haven't mentioned much about process development, I assume you want people to know how to do at least critical portions. Never write a process without flowcharting it, this will save you grief by getting people to focus on the process instead of a step by step set of directions. It takes someone fairly good to document the complex and make it look simple, that is your job at this point in time.
The bottom line is that your documentation should show dataflow for each critical system. As long as you can do this someone else can step in and work with what you have, even if they may not understand a given piece. One of the big advantages of flowcharting everything (especially processes!) is that this will readily show you weakness and holes that may have been previously overlooked. When flowcharting complex processes don't be afraid to have a single point represent an entire additional complex process that can be distictly referenced of it's own accord (as an car repair manual of mine once described the process to replace a crankshaft "Step 1. Remove Engine".) If you try to put to much detail in a given process you lose your audience and the value of the documentation.
Bottom line when I am done with a design document it covers server, network, database and client topology in varying levels of detail with dataflow. A typical design document I would turn over would be 150 pages with most of that broken down into different sections describing what was done, why it was done, the best practices followed for build, and best practices for lifecycle. The document typically does not get read by any one person, instead it would be a reference for a number of different departments that will each reference it according to their own needs.
Documentation at your hands, and timestamped (Score:4, Insightful)
If you are in the server room, and you have:
A: a spreadsheet that your predecessor made.
B: a post-it note on the switch saying it what it does.
Which one do you trust?
For the physical/low-level network the documentation should be in the network. Just like source code should contain comments about this particular piece of code, a similar approach works reasonably for the physical network. I see no point in a having an outdated spreadsheet. It is more useful that the cables and ports are labelled and numbered, that there is a post-it note on a switch say where the links go, etc.
The grand overview should be in electronic form, though. A scanned hand drawing is fine. A photo of a whiteboard drawing is fine too.
For the logical network put comments whereever possible. On settings, VLAN configurations, server connections, account setups, ...
Again, the grand overview should be in electronic form.
I have found it useful that the information is timestamped, so you know when it was last valid.
Re:Documentation at your hands, and timestamped (Score:5, Funny)
> Trust spreadsheet
You pull the wrong cable.
It's suddenly dark.
You get eaten by a grue.
Parent
Wiki Wiki Wiki (Score:5, Informative)
I have a nicely formatted template page with all those categories set out. I also maintain a page of IP address assignments and an inventory of harware specs of all the machines in the office (which is helpful in the cases of "We need to reproduce a bug that only happens on ____ processor with ____ video card" and of "We're getting new machines. Who is in most dire need of an upgrade?").
I write down everything in these, and find myself referring to them very often. My predecessor gave me a Word document with all his notes in it, which has been very useful, and I used that as a starting point for my pages. The wiki has saved me a ton of time, kept me organized, and serves as a great reference for me and for the inevitable next admin.
The only caveat is if the wiki (or the server it's on) goes down. This has happened once, and my instructions for fixing the wiki were... on the wiki, so extra troubleshooting for me. Thus, I find it good practice to maintain a hard copy of the wiki pages, especially the page that tells how to fix the wiki.
I'm running this on Redmine, which has proven to be bleedingly simple to use and administer, and much easier than trac, which we used before. It's especially nice having it on the intranet, as I'll just have a browser open to the wiki as I work on systems and refer to and update it as appropriate. It's very handy to document exactly how I performed a strange or experimental installation of some software that I'll want to replicate later without making myself crazy, and I'll take the extra few seconds to retype the commands I just used into the wiki from anywhere in the building, though I probably wouldn't do the same into a Word doc.
It's not so much the mundane day-to-day that I find that important to document. It's the weird fixes, the trouble spots, the command line parameters, the installation procedures, the changes that shouldn't have fixed it but did, and the horrific chain reaction situations that make one piece of software crash because a seemingly unrelated piece of software has the wrong version of the 64-bit library. Things that take 4 hours to figure out and 3 seconds to implement... those are the ones to document, and those are the ones that I'd be kicking myself 20 months later for neglecting to write down. In an afternoon, any schmuck could walk into the building and figure out which network cable goes where. Documenting the strange bits (and the frustrations), though, can get a malfunctioning mail server back up and running in 3 minutes instead of 3 hours (which, of course, is secondary to good administration keeping the server from going down in the first place).
Re:Department of Redundancy Department (Score:5, Funny)
Now I see this Microsoft bashing on Slashdot regularly and it's completely unfounded. I use a Windows-based PC and it has provided me with years of worry, virus and spyware free operation.
In fact, our PC's at work are WeOffer popular brand names drugs such as ViagraFrom $1.87, CialiFrom $2.38, SomaFrom $1.07, TramadolFrom $1.38, LevitrvFrom $2.52, Celebre, Zocor, Fosamax, Effexor, Zyrtec, Plavix, Premarin, Flomax, Paxil, Zoloft, Prevacid, and Evista. Now it's easy to get your needed drugs 0nline.
Parent
Windows != SPAM (Score:5, Insightful)
Attempting (even facetiously) to blame SPAM on Windows is wrong. If every copy of Windows on the Internet somehow magically disappeared, the SPAM problem would not abate. Bot herders and spammers would simply shift their efforts to other platforms.
If your doubt this, consider what the winner of this year's PWN2OWN [tippingpoint.com] contest had to say about why it's easier to target Mac OS X [zdnet.com].
BTW, this is not a troll, and I'm not a (Windows|Mac|Linux) evangelist of any kind. I just find kneejerk Windows bashing rather tiresome
Parent
Re:Department of Redundancy Department (Score:5, Funny)
An old girlfriend tells you you need viagra, and you still dont't get it.
Parent
Re:Alternatively... (Score:5, Funny)
English only please... I don't understand this word "vacation" you have mixed in with the others.
Parent
Re:False Info (Score:4, Insightful)
"The network started horrible, here is how I cleaned it up" is a GOOD reference. I have killer references from two jobs I automated myself out of this way. Each time I got a more interesting more challenging better paying job by doing so.
Parent