Institutional Memory and Reverse Smuggling 312
An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."
It's common (Score:2, Interesting)
A unnamed corp where I use to work would hire retired employees for contract work at high per hour wages for months due to issues like this.
Absolutely true (Score:5, Interesting)
Software Evolution (Score:3, Interesting)
One of the courses of my university study was called 'Software Evolution' in which tutors showcased all kinds of problems with legacy software. Software using languages or platforms no one knows about anymore or lost source code of mission critical systems. Especially older banks and insurance companies suffer from this problem, often choosing the 'wrap around the bit that we know that works' instead of a complete overhaul of a system no one really knows what it does but does it really well. I fear everything we build today will cause the same problems, but worse. Perhaps the old idea that readable code should help people understand its function saves us from this scenario...?
Re:I see this in code I work on all the time (Score:4, Interesting)
Somebody writes the code, doesn't bother to comment it at all and then you come in years after the fact. You look at the code and wonder, "why did he do it that way instead of this way?" Then the big gotcha, you think I could ask him but he left the company 5 years ago(At this point slap your forehead and hope you don't break anything working on the code.)
One way to mitigate this is "extreme programming" where developers pair and constantly switch off projects and tasks. You end up with a larger body of people that understand the code and comments and code that is much more understandable because if you write it in a way that isn't, someone will notice the next week, complain and you have to go back and fix it.
Also, if you're hoping you don't break anything, hopefully you have a battery of unit tests and functional regression tests that will catch the lion's share of anything you might break and let you know in short order.
Opportunity for more pay (Score:5, Interesting)
I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works.
I certainly hope that you negotiated a much higher salary than before.
I have known people who were fired in a "chain saw" maneuver who proved to have core corporate competencies in their heads. They negotiated for a much higher rate the second time around...
Re:Company rules against removing documents (Score:5, Interesting)
Yes, that's what makes it tricky. I wasn't supposed to have these documents, but some important internal drawings have been lost, and I have a copy. The safest solution would be to avoid bringing it up, I agree. Though I'm not blackmailing them for the documents or anything.
I've heard it happen several other times. I've had colleagues who consulted for a company, then a few years later get a call or email asking if they by any chance still have a copy of some old document. Of course they weren't supposed to have kept a copy, but if they accidentally did, it would be really nice if they could send a copy because the company seems to have lost it...
It mostly happens at lower levels of the org chart on a don't-ask/don't-tell basis I think. Lots of people keep personal archives, and sometimes the official archives have to get patched out of them.
I work there (Score:2, Interesting)
I work in a library at a large organization and I see this every single day. The short answer to the problem is that it needs to be someone's responsibility to ensure access to KNOWLEDGE (NOT data, documents, etc, but knowledge) in the long term. I get requests from our employees every single day asking for information on project x that was shut down in the 90's. It is my job to track down any information that I can. Unfortunately, I have neither the directive or funding to start gathering and organizing the knowledge in such as way that it is useful a few decades from now. So instead I tell our employees "sorry, you will have to spend a few hundred thousand to recreate the information."
There are lots of people out there that can fix this type of problem, but they are never given the tools to do so.
assertions are better than comments (Score:5, Interesting)
now one does need to document the expected inputs and results of subroutines. better than commenting is to use assertions as well as unit tests. so for that contrived example, what I would actually do:
int foo()
{
int result = 5;
assert( 4==result);
return result;
}
if you don't maintain the assertions and unit tests with the payload code you'll find out about it right away.
Old Story (Score:1, Interesting)
This guy worked at a company for 30 years. He retired and his company called him back in when an important machine started to break down. He placed an X where they had to fix the machine and then left. He sent the company a bill for $10,000. The company asked him to detail the bill. He sent: Placing a mark where to fix the machine $1. Knowing where to place the mark, $9,999.00. Old story and these days I make a few bucks because people improperly documented their work OR had no clue what they were doing. For example using a 1/4" industrial air disconnect to feed 2 air cylinders which hold a maximum of 10CF each at 90psi and then trying to fill them in about 10 seconds with 60psi of air from a 90psi source. Ain't guys with Phd's really smart :-) If it weren't for those really smart and educated people I might never have made any money.
Re:I don't comment my code anymore (Score:5, Interesting)
> that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
IMO, _proper__ commentting discusses WHY you did something. If someone wants to understand HOW it was done, read the code, as you correctly reason. Any half-decent coder should be able to figure out what the code is doing.
EXCEPTION: If you are doing something tricky, then document that. i.e. http://www.codemaestro.com/reviews/9 [codemaestro.com]
NASA (Score:2, Interesting)
Re:Sometimes IT has to help the business (Score:5, Interesting)
An important task for Historians is to regularly remind the bosses what Historians are for so they don't get sacked
salvage and surplus (Score:5, Interesting)
At my institution we routinely buy back equipment from salvage yards it was sold to. But this actually makes sense. We clear out lots of old stuff rather than pack ratting it. It's a small price to pay for the convenience of getting stuff we need back.
Ask intel (Score:3, Interesting)
One of the things they do very well is... no, not designing chips, they suck at that, but duplicating fab plants. Down to getting everyone that's gonna work in there up to speed before the plant is finished.
The key is of course not to just build a plant. It's to build at least two plants, at least one in reality and one in the documentation. Fred Brooks (yes, exactly in his book _the mythical man-month_) explains how this works for software and computer hardware. It won't be different for processing plants, except of course that your production numbers are lower. But even with just one physical plant, you need full documentation on how to replicate it, with every task and every job described. And then, of course, you update the documentation along with any updates to the plant, bring it back up to the level where you can xerox the paper into reality again.
The test is exactly that: Can we give this to another engineer and have him duplicate the thing from that? Yes, most software documentation fails that test and is utterly useless as api docs or user manual to boot-- as if those code monkeys don't even read what they wrote, the illiterate bums. And yes, you probably should check every five or ten years if it's still understandable. If that's expensive, well, you get to pay more thirty years down the line. Or you can start from scratch, if you prefer.
Notwithstanding that this documentation archaeology an interesting excercise, but as TFA illustrates, it makes for rather schizophrenic situations, is way out of the usual corporate flow of how things normally go, cost a bundle, takes a lot of time, and is fraught with uncertainty. Care and feeding of the corporate library is future-proofing the IP assets, far more so than filling a war-chest with patents. It's about as sexy as caring for the backup tapes, but you have those too for a very good reason.
Re:I see this in code I work on all the time (Score:5, Interesting)
Or just plain boobytrap the code all over the place with sanity checks and known gotcha conditions, spiked with good comments and explanations.
I ran across that just last week when developing an importer for iTunes music library - reusing an xml class I wrote 3 yrs ago, it blew out on me with a "you're being STUPID" sort of error message just under a dozen times during development. Opening up said code block revealed comments I made when I wrote the sanity check, describing WHY it had coughed up the message, and more importantly, what was causing the problem.
I of course had little to no memory of ever writing them, but there they were, providing me an instant catch-me-up in the why's of my code, and saved me from having to relearn all the internals and subtleties that went into writing that bug-free, beautiful, and time-saving class of code.
That's one of the great things about having a poor long-term memory - it forces you to comment (and liberally sanity check) your own code well, not because someone else will need it, but because you are going to need it. You get to be the direct beneficiary of your own good practices.
labour relations (Score:4, Interesting)
Many companies have a tiny number of people who manage things, and then a sea of consultants who do things. Often the management of the consultant groups are consultants themselves.And even the managers of the managers are often consultants. The result? Corporate Memento syndrome. No long term memories get formed because the people who do the remembering are gone. Stuff is rar'd or stuf'd or tar balled into a series of docs that sit on a server than is run by an bunch of consultants in IT who have no idea what is on the drive.
Then the drive either fails or is dumped into a bigger storage drive and labeled in a folder "management data 2009" or whatever, and there it rots because no one has any idea what's inside, and the people with access are consultants who don't/can't know.
and why? Because it's cheaper to hire a river of consultants for 6 months than it is to hire people and keep them around for years. The result? Permanent amnesia. All of it will be lost, and no one will care. the essence of 21st century culture - digital business - will be erased and forgotten. Forever. Some would say "Good riddance", but I'm enough of an armchair anthropologist to feel the loss.
Re:And so, civilisation ends. (Score:4, Interesting)
This is actually why the space program is in the state it's in. They can't build more heavy lift rockets because there is nobody left who actually knows how they were built in the first place.
Re:So what's the problem then? (Score:4, Interesting)
Sounds familiar. I work in accountancy practice. The first year someone does a good job, gains an understanding of the business, thinks about it, documents the prep file well and puts important long-term stuff into the permanent file. Takes say 100 units of time.
Second year, someone does a good job, mostly gets to follow last year's files but keep up the documentation, updates it and the permanent file. Takes maybe 60 units of time, which sets a realistic bar.
Third year, someone follows the previous year file but doesn't bother documenting their own and fuck the permanent file. Takes 40 units time and a pat on the back. Resets the bar at 40 units.
Year 4, see year 1.
History vs. Archaeology (Score:5, Interesting)
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem. By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
All too true. The sooner you learn not to care, the sooner you will stop feeling the pain. Surrender to mediocrity; obey all rules and your managers without question. Above all, do not let your work interfere with your life. You will be a happier person, and probably get promoted. Listen to me, oh young ones...heed the words of the Ancient One: Despair!
Once upon a time, I worked in a relatively small organization within a Very Large Company. The VLC was renowned for its inventiveness, until a new manager came who felt that inventing things was much too expensive; it would be more profitable to just to put the word "Invent" on the corporate logo, and then re-sell stuff from countries who make things cheaply. Because our corporate logo was on these re-branded products, we must obviously have invented them. Thus, marketing ceased to be a partner of R&D, and instead replaced it. Clever, eh?
One of my contributions to the VLC was a simple document management system (DMS). My boss mentioned that all the engineers in our lab kept complaining that "they can't find anything"—specifically, nobody could find old documentation for code that was written by vanished civilizations of programmers that had previously labored at the lab. In fact, it turned out that this was a general problem among the labs of the VLC, and that they all wished mightily for a solution. Apart from really ancient projects (from, say 10 years ago), it was pretty difficult to find documentation for even recent projects, as each lab had a different way of organizing its documentation, different document formats, and no channels for distributing the documentation to other groups who might have a legitimate interest. So I brought in an easy-to-use and easily maintained DMS, pitched it to the engineers at our lab, and got them to use it. My boss then made me contact a bunch of other labs, and offer them the use of the same DMS. A lot of their engineers liked it, and there was much happiness. I got lots of pats on the back for what was really a pretty simple idea, and my boss loved me for it. A year later, I was laid off.
As I was the only one maintaining this system, and no one ever asked me to train a replacement, I doubt that this DMS—and its contents—survived, save perhaps as digital ghosts on some unlabeled backup media in a faraway storage cave. In a sense, I'm one of the perpetrators of information decay; I caused all the knowledge deposited in "my" DMS to get black-holed. Come to think of it, this is probably not a problem; I'm pretty sure they've laid off all the engineers by now.
I understood very well that simply setting up a DMS was not the cure for the I-can't-find-anything problem. It was a beginning, at best. What was needed was a comittment by the organization as a whole to make sure the information continued to be available for as long as the company exists. Because engineering practices change, so do documentation practices; documentation technologies themselves change—not to mention physical storage media and data encoding. My DMS probably should have been replaced with something better by now (if it had continued in use), and that means someone would have had to plan the transition, and move the old docs to the new system. Maybe the best thing to do is print everything as hard copy from time to time, and hire a librarian to keep track of this backup.
Considering how much has been said and written (and blathered) about "knowledge management", it's amazing how little attention is ever given to the temporal aspect of institutional knowledge. Every engineering company should have an Office of Knowledge that has as part of its responsibility not
Re:Know-how and know-why... (Score:5, Interesting)
Re:And so, civilisation ends. (Score:4, Interesting)
I didn't say anything about "reinventing." I said building more of what they have.
It costs a lot more to invent, test, and deliver something than it does to build another of something which already works. There's likely to be a lot of efficiency improvements with newer heavy lift rockets because of advances in technology. That has nothing to do with the point I made, which apparently went whizzing over your head.
Re:Know-how and know-why... (Score:2, Interesting)
Computer Systems Too (Score:4, Interesting)
Sometimes I had to go through a list of servers on a specific subnet, match that up with a list of servers from the routine network scans to see if all were accounted for, if not, I would try to find the server in the server room, get its ID from the chassis, go back to the inventory system, try to find the server there and hopefully an owner too.
Occasionally I could either not find the system in the inventory system or no owner was assigned. I would then have a private chat with the hardware people responsible for the area the servers were placed and suddenly the relevant servers would have an "unscheduled network outage", i.e. the operator would unplug the system and we would wait for somebody to start screaming. If that did not happen within a day or so, we would dump the system's data to the backup system and reassign the server. This time properly documented.
Re:History vs. Archaeology (Score:5, Interesting)
This is the absolute truth. Learn it easy now, or the hard way as you are booted out the door. I know this from experience. It is very true.
Meanwhile, I've done this. Actually, I'm in the middle of doing it again. I've thrown documents in the deep pit of the corporate Document Management System, however, I've tried to keep it all together by linking documents together by reference and hyperlinking. Fab stuff, except people keep saying that they can't find a thing without a link. *sigh* and this is for a mandated DMS.
So, now, I'm doing it again. This time with 'newer' technology (which shall not be named here, but of course is "Quick") .. same concept.. just easier to do instead of plain HTML. And on it goes.
Worst thing I have done is documented the system I used to admin.. I have been replaced by monkeys for whom management thinks is doing *my* job because they can follow basic documented procedures badly. However, I take secret delight that it is not *full* documentation - it is merely a how-to and a where-is than anything else. You could use it to maintain the existing system, but you still need to know how the system works... and why it works the way it does.
I have been kicked, and badly. By The Manager, and Publicly no less.
Yes, I am on my way Out (tm).
Yes, I was paid to build and maintain this and other systems, so that is in the past and done. I did my job, and did it well.
If these monkeys can't find the system documentation then it isn't my problem.
Now I am sitting back and watching the system degrade. Processes I created and documented are being sloppily followed, some not at all. Highly paid 'technical' people are spending days doing what takes me hours. Management doesn't care. Management doesn't want to know. Unless the system is down and someone is screaming then it Does Not Matter. Even then it is a case of 'fix this little problem now and I don't care why'.
What really sucks is that I am carrying quite a bit of useful information in my head.
So, what I am doing now is secretly bundling up what I know, putting documents and information together, linking it where it can be found and putting it where someone who - at some point in the future - will be able to find it and who will need to use it. I wish that person luck and happiness. It won't be tomorrow, or the day after, not next week or next month. Next year? Perhaps. Most likely in 3 to 4 years from now, when they find out that these twits should not be allowed near computers, let alone be allowed to administrate systems.. then this will be needed.
Until then, I am being paid. I am doing my job, and my primary job is to make the PHB manager happy.
I can no longer read Dilbert. It's too depressing, because it is too real.
Re:Know-how and know-why... (Score:5, Interesting)
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
A quick perusal of this thread shows no mention of George O. Smith's story Lost Art [wikipedia.org], which emphasizes precisely this aspect of engineering knowledge. A couple of humans archeologists are digging in the Martian ruins and come upon an ancient Martian device with the manual, which proves to be almost useless until they have done the systematic experimentation to understand how it actually works.
It was published in December, 1943, which suggests this kind of problem has been happening again and again for the better part of a century. Unfortunately the solution to it (value your people and don't treat them as interchangeable parts to be laid off the moment its convenient to outsource their work) is so completely counter-intuitive to the sociopaths who have always been the ones in charge of large organizations that it will never be implemented consistently.