Institutional Memory and Reverse Smuggling 312
An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."
This problem is quite common. (Score:5, Insightful)
Re:This problem is quite common. (Score:5, Insightful)
Yep, It happens all the time everywhere.
What I find most troubling about this is that most company don't recognize the problem or are not even aware of it? I think the mentality that "anyone can be replaced so we don't have to create incentive to retain them" is going a bit too far.
Because the real problem is in fact employee turnover.
So what's the problem then? (Score:3)
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem.
By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
salvage and surplus (Score:5, Interesting)
At my institution we routinely buy back equipment from salvage yards it was sold to. But this actually makes sense. We clear out lots of old stuff rather than pack ratting it. It's a small price to pay for the convenience of getting stuff we need back.
Know-how and know-why... (Score:5, Insightful)
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
I have witnessed some "technology transfers" where a set of working products was transferred with documentation, design data, and much background information to a group in another region (mostly EU/US transfers). The first group is then wound down or redeployed. The new group does fine at first, until a new version of a product is needed. Then they always make mistakes the original designers would consider childish or stupid. They don't know the why behind design decisions, and don't know what to avoid. In my experience, it takes 5-10 years to build a decent product design group from scratch. On the other hand, an existing design group will sustain itself by mentoring and guidance of new inductees by experienced members.
The only successful technology transfer which I participated in was one where a few senior designers were transferred with the set of products they were responsible for. It was far from cheap in up-front costs, but it avoided the crises and financial disasters which afflicted the other transfers after a year or two.
Re:Know-how and know-why... (Score:5, Interesting)
Re:Know-how and know-why... (Score:5, Interesting)
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
A quick perusal of this thread shows no mention of George O. Smith's story Lost Art [wikipedia.org], which emphasizes precisely this aspect of engineering knowledge. A couple of humans archeologists are digging in the Martian ruins and come upon an ancient Martian device with the manual, which proves to be almost useless until they have done the systematic experimentation to understand how it actually works.
It was published in December, 1943, which suggests this kind of problem has been happening again and again for the better part of a century. Unfortunately the solution to it (value your people and don't treat them as interchangeable parts to be laid off the moment its convenient to outsource their work) is so completely counter-intuitive to the sociopaths who have always been the ones in charge of large organizations that it will never be implemented consistently.
Re:So what's the problem then? (Score:5, Insightful)
Management don't care, why should you? It really can't be all that much of a problem.
The manager who didn't document anything finished ahead of schedule and below budget. He got a big bonus and moved on to another job at a different company based on his reputation for delivering early and cheap.
The manager who comes in to organise an upgrade to that software ends up taking much longer than expected and going well over budget because nothing was documented in the initial spaghetti code. He gets fired for being incompetent.
Re:So what's the problem then? (Score:4, Interesting)
Sounds familiar. I work in accountancy practice. The first year someone does a good job, gains an understanding of the business, thinks about it, documents the prep file well and puts important long-term stuff into the permanent file. Takes say 100 units of time.
Second year, someone does a good job, mostly gets to follow last year's files but keep up the documentation, updates it and the permanent file. Takes maybe 60 units of time, which sets a realistic bar.
Third year, someone follows the previous year file but doesn't bother documenting their own and fuck the permanent file. Takes 40 units time and a pat on the back. Resets the bar at 40 units.
Year 4, see year 1.
Re:So what's the problem then? (Score:4, Insightful)
I have seen a lot of examples where the problem was not lack of documentation but rather that the available documentation documented the obvious things (this is a piece of paper) instead of this is the formula for turning garbage to gold. It is very important to document what the code accomplishes instead of documenting each line of code.
Good
document once a purchase order has been approved, prevent changes to the line items
Bad
instead of if the PO status = 'APPR' and the item number or price is changed, raise an exception.
History vs. Archaeology (Score:5, Interesting)
Management don't care, why should you? It really can't be all that much of a problem. More a perception than a problem. By all means go ahead and create/use a documentation system for your group but clearly there just isn't a requirement for anything more comprehensive.
All too true. The sooner you learn not to care, the sooner you will stop feeling the pain. Surrender to mediocrity; obey all rules and your managers without question. Above all, do not let your work interfere with your life. You will be a happier person, and probably get promoted. Listen to me, oh young ones...heed the words of the Ancient One: Despair!
Once upon a time, I worked in a relatively small organization within a Very Large Company. The VLC was renowned for its inventiveness, until a new manager came who felt that inventing things was much too expensive; it would be more profitable to just to put the word "Invent" on the corporate logo, and then re-sell stuff from countries who make things cheaply. Because our corporate logo was on these re-branded products, we must obviously have invented them. Thus, marketing ceased to be a partner of R&D, and instead replaced it. Clever, eh?
One of my contributions to the VLC was a simple document management system (DMS). My boss mentioned that all the engineers in our lab kept complaining that "they can't find anything"—specifically, nobody could find old documentation for code that was written by vanished civilizations of programmers that had previously labored at the lab. In fact, it turned out that this was a general problem among the labs of the VLC, and that they all wished mightily for a solution. Apart from really ancient projects (from, say 10 years ago), it was pretty difficult to find documentation for even recent projects, as each lab had a different way of organizing its documentation, different document formats, and no channels for distributing the documentation to other groups who might have a legitimate interest. So I brought in an easy-to-use and easily maintained DMS, pitched it to the engineers at our lab, and got them to use it. My boss then made me contact a bunch of other labs, and offer them the use of the same DMS. A lot of their engineers liked it, and there was much happiness. I got lots of pats on the back for what was really a pretty simple idea, and my boss loved me for it. A year later, I was laid off.
As I was the only one maintaining this system, and no one ever asked me to train a replacement, I doubt that this DMS—and its contents—survived, save perhaps as digital ghosts on some unlabeled backup media in a faraway storage cave. In a sense, I'm one of the perpetrators of information decay; I caused all the knowledge deposited in "my" DMS to get black-holed. Come to think of it, this is probably not a problem; I'm pretty sure they've laid off all the engineers by now.
I understood very well that simply setting up a DMS was not the cure for the I-can't-find-anything problem. It was a beginning, at best. What was needed was a comittment by the organization as a whole to make sure the information continued to be available for as long as the company exists. Because engineering practices change, so do documentation practices; documentation technologies themselves change—not to mention physical storage media and data encoding. My DMS probably should have been replaced with something better by now (if it had continued in use), and that means someone would have had to plan the transition, and move the old docs to the new system. Maybe the best thing to do is print everything as hard copy from time to time, and hire a librarian to keep track of this backup.
Considering how much has been said and written (and blathered) about "knowledge management", it's amazing how little attention is ever given to the temporal aspect of institutional knowledge. Every engineering company should have an Office of Knowledge that has as part of its responsibility not
Re: (Score:3)
Do you have any idea how much such a library would cost?! No, we can't afford that.
Re:History vs. Archaeology (Score:5, Interesting)
This is the absolute truth. Learn it easy now, or the hard way as you are booted out the door. I know this from experience. It is very true.
Meanwhile, I've done this. Actually, I'm in the middle of doing it again. I've thrown documents in the deep pit of the corporate Document Management System, however, I've tried to keep it all together by linking documents together by reference and hyperlinking. Fab stuff, except people keep saying that they can't find a thing without a link. *sigh* and this is for a mandated DMS.
So, now, I'm doing it again. This time with 'newer' technology (which shall not be named here, but of course is "Quick") .. same concept.. just easier to do instead of plain HTML. And on it goes.
Worst thing I have done is documented the system I used to admin.. I have been replaced by monkeys for whom management thinks is doing *my* job because they can follow basic documented procedures badly. However, I take secret delight that it is not *full* documentation - it is merely a how-to and a where-is than anything else. You could use it to maintain the existing system, but you still need to know how the system works... and why it works the way it does.
I have been kicked, and badly. By The Manager, and Publicly no less.
Yes, I am on my way Out (tm).
Yes, I was paid to build and maintain this and other systems, so that is in the past and done. I did my job, and did it well.
If these monkeys can't find the system documentation then it isn't my problem.
Now I am sitting back and watching the system degrade. Processes I created and documented are being sloppily followed, some not at all. Highly paid 'technical' people are spending days doing what takes me hours. Management doesn't care. Management doesn't want to know. Unless the system is down and someone is screaming then it Does Not Matter. Even then it is a case of 'fix this little problem now and I don't care why'.
What really sucks is that I am carrying quite a bit of useful information in my head.
So, what I am doing now is secretly bundling up what I know, putting documents and information together, linking it where it can be found and putting it where someone who - at some point in the future - will be able to find it and who will need to use it. I wish that person luck and happiness. It won't be tomorrow, or the day after, not next week or next month. Next year? Perhaps. Most likely in 3 to 4 years from now, when they find out that these twits should not be allowed near computers, let alone be allowed to administrate systems.. then this will be needed.
Until then, I am being paid. I am doing my job, and my primary job is to make the PHB manager happy.
I can no longer read Dilbert. It's too depressing, because it is too real.
Re:This problem is quite common. (Score:4, Insightful)
Because the real problem is in fact employee turnover.
When you're talking about a scale of decades, including economic downturns with hiring freezes and whatnot, employee turnover is a fact of life. No one works in the same place forever, no matter how good the incentives are to stay. Even if you can stop them from changing jobs, they might retire. Even if you manage to make their job so compelling that they don't retire, death will eventually get them. And good luck getting ghosts to work for you in any meaningful way.
Its purely a management problem. Data retention over long periods of time is a real challenge, and one that most people don't care about because well.. out of sight, out of mind. Put a token effort towards data management when someone important is looking (investors, higher management, whatever). And then redirect the money to more immediate problems as soon as the heat is off.
Remember, the quarterly report is king. Trying to predict and prevent problems that may or may not crop up 10 years from now seems less relevant when you have to make not only profits, but record profits year after year or risk your stock price plummeting.
Re:This problem is quite common. (Score:5, Insightful)
"Because the real problem is in fact employee turnover."
That's not as much a problem as a reality you'd better learn to live with.
Look: there's only two ways for an employer-employee relationship to end up: either the company closes business or the employee goes out. Since the company as an entity isn't so interested on the afterthoughts of its own end, there's only an interesting case to consider: the employee will go away. That's as valid for the janitor as is for the general manager.
So it is not "anyone can be replaced" but "anyone *will* be replaced".
And so, civilisation ends. (Score:3)
And a new dark ages are born.
Oh enough hyperbole. Wouldn't worry about it, the companies in question will collapse and be replaced by someone else if the demand is high enough. Or not if it isn't.
Re: (Score:3)
But of course, this would imply a corporation with the ability to see beyond the tip of its own nose, which is rare.
Re:And so, civilisation ends. (Score:5, Insightful)
"But of course, this would imply a corporation with the ability to see beyond the tip of its own nose, which is rare."
The problem is not only corporate culture but societal.
Say Big Corp A has in fact the ability to see beyond the tip of its own nose and then spares efforts to prepare for the next decade which, of course, impacts its bottom line for this quarter. At the same time, Big Corp B doesn't see beyond the tip of its own nose and this quarter's results look better than that from Big Corp A. Then, all of a sudden, everybody, *you included*, sell their stocks from Big Corp A to buy stocks from Big Corp B which makes Big Corp B even bigger while Big Corp A is not a big corp any more.
Which, ironically, makes Big Corp B to be the one with the ability to see beyond the tip of its own nose because since Big Corp A is a big corp no more, its investment for the future is now moot and shouldn't have been done to start with.
That's exactly why you see from time to time companies exploding basically out of nothing (say Google or Facebook) despite of the fact that other Big Corps near the business niche have much more than enough muscle to crunch them -their stockholders wouldn't allow for the investment. And that's exactly way you see those Big Corps investing much more in rising the bars for new entries with lawyers, IP, patents and lobying legislators than in real R&D.
Re:And so, civilisation ends. (Score:4, Interesting)
This is actually why the space program is in the state it's in. They can't build more heavy lift rockets because there is nobody left who actually knows how they were built in the first place.
Re:And so, civilisation ends. (Score:4, Interesting)
I didn't say anything about "reinventing." I said building more of what they have.
It costs a lot more to invent, test, and deliver something than it does to build another of something which already works. There's likely to be a lot of efficiency improvements with newer heavy lift rockets because of advances in technology. That has nothing to do with the point I made, which apparently went whizzing over your head.
Re: (Score:3)
That's not really the point either. You can build on older technologies if you have the people who know how they work in the first place. All of the building blocks were there to develop better implementations, but that is no longer possible. It's being re-invented because there's absolutely no other option. Nobody can re-use any of the work that went into launching heavy rockets into space that might be applicable today. That might be a small amount of help, or a large amount. The point is, nobody knows an
Been there, done that: (Score:5, Funny)
But the NDAs keep me and everyone else involved from talking about it. :P
It's common (Score:2, Interesting)
A unnamed corp where I use to work would hire retired employees for contract work at high per hour wages for months due to issues like this.
Re:It's common (Score:5, Funny)
No (Score:4, Funny)
Speaking from experience, corporations are unable to give themselves just one name. They have to change their names regularly (because it obviously makes things better, like go faster stripes) and more, they have to change the names of departments even more regularly (again because it improves everything). The result is that any documentation system which is created rapidly becomes fragmented, out of date and lost as the paths to the documents are changed to match the names.
The conclusion is that using names only makes things worse.
Re: (Score:3)
There is much truth in this. My current unit (10 people) has had the same name for 15 years. Meanwhile the pyramid above has been through every permutation possible. People are still able to contact us because we did not let go of our name or branding.
Managers rarely have to produce a cost benefit analysis for name changes or Organisation chart changes. But they should. If you prove a profit in 12 months (the seeming time between reorganisations) then you can go ahead. Otherwise, No can do. Unfortuna
Re: (Score:3)
I think you may have it backwards. As near as I can tell, at my organization, changing the group names in the hierarchy is actually a source of income. See, if you can claim that the reorganization will leverage synergies or some other intangible, and get corperate to buy into it; they will give you a bunch of overhead money and secretarial support to implement the change. And free overhead money and secretarial support is like, no, better than gold-plated ponies.
Absolutely true (Score:5, Interesting)
Re:Absolutely true (Score:4, Informative)
Where I work, we've mined many products which failed to reach the market for good ideas. Often, there were many good ideas in there, it was just the package as a whole that didn't work out. (Either too early, too late, too big or too small.) But, we've also had good continuity among our key folks (myself included), so that probably explains why it ends up working. We understood why we put those features in failed product X, and so we understand how they'd work in new, more viable product Y. We also understand at least something about why X failed, so we can try to avoid it on Y.
But again, that does rely on institutional memory to make it work.
I see this in code I work on all the time (Score:5, Insightful)
Re:I see this in code I work on all the time (Score:4, Interesting)
Somebody writes the code, doesn't bother to comment it at all and then you come in years after the fact. You look at the code and wonder, "why did he do it that way instead of this way?" Then the big gotcha, you think I could ask him but he left the company 5 years ago(At this point slap your forehead and hope you don't break anything working on the code.)
One way to mitigate this is "extreme programming" where developers pair and constantly switch off projects and tasks. You end up with a larger body of people that understand the code and comments and code that is much more understandable because if you write it in a way that isn't, someone will notice the next week, complain and you have to go back and fix it.
Also, if you're hoping you don't break anything, hopefully you have a battery of unit tests and functional regression tests that will catch the lion's share of anything you might break and let you know in short order.
Re:I see this in code I work on all the time (Score:5, Interesting)
Or just plain boobytrap the code all over the place with sanity checks and known gotcha conditions, spiked with good comments and explanations.
I ran across that just last week when developing an importer for iTunes music library - reusing an xml class I wrote 3 yrs ago, it blew out on me with a "you're being STUPID" sort of error message just under a dozen times during development. Opening up said code block revealed comments I made when I wrote the sanity check, describing WHY it had coughed up the message, and more importantly, what was causing the problem.
I of course had little to no memory of ever writing them, but there they were, providing me an instant catch-me-up in the why's of my code, and saved me from having to relearn all the internals and subtleties that went into writing that bug-free, beautiful, and time-saving class of code.
That's one of the great things about having a poor long-term memory - it forces you to comment (and liberally sanity check) your own code well, not because someone else will need it, but because you are going to need it. You get to be the direct beneficiary of your own good practices.
Re:I see this in code I work on all the time (Score:4, Funny)
A post at bash.org [bash.org]:
Re: (Score:3)
There are some pieces of code at work where the "constantly switch off projects and tasks" has been applied almost as an anti-pattern. Glue scripts with code dating back 15 years, where probably a dozen or more different folks have gone in and made their "one little tweak", trying not to disturb the rest. We end up with a script with 50 flags, and who knows what subset of those flags is still relevant or really works.
Every so often, someone goes in there and removes some of the cruft, but it is still the
Re: (Score:3)
It doesn't work. You'd need people that follow the same coding practices and have enough experience to move at the same pace. Which is pretty much impossible.
Heh, it's funny since the place I work now has been making money doing it for a decade. People don't have to have the exact same level of experience to work on the same code and hand it off, especially when working as pairs in a group environment where you can always just ask the team if you don't understand something. You learn a whole lot faster when sharing a keyboard with someone who already has been working on the code base for a few years.
Re: (Score:3)
First, enough turnover will always kill you no matter your procedures. If a team turns over 75% of its developers in 6 months, there's nothing to be done. So step 1: make sure your developers don't want to quit.
Turnover is very harmful, but as I said spreading out the coding tasks across a whole team mitigates the damage and allows you to scale back up to efficiency in a reasonable timeframe. Brooks law states, "adding manpower to a late software project makes it later", but I've seen it proved false. With paired programming adding 50% more programmers on a late project did not slow our development at all and within a few weeks it had sped up significantly. The caveat being you have to be doing paired programming
Re:I see this in code I work on all the time (Score:5, Insightful)
Even worse, it sometimes turns out that YOU wrote the code 10+ years ago, and you have absolutely no clue why it was written that way.
Re:I see this in code I work on all the time (Score:5, Funny)
Re:I see this in code I work on all the time (Score:5, Informative)
It's a reference to this scene [youtube.com] from the Pixar movie "Up".
Re: (Score:3)
What the hell does this mean
I don't know. Let me think abou-SQUIRRELS.
Re: (Score:3)
What I really, really hate is looking at the code that I wrote 10 years ago
- finding it could be much more elegant and more efficient
- assuming I was an idiot
- rewriting the code to make it much more efficient
- and then finding out that the optimization misses a really subtle corner case
- and finally remembering that, yeah, I wasted 3 days hitting this stupid corner case years ago...
Now every time I do something even slightly non-obvious, I document *why*, not just *how*.
Re: (Score:3)
I have to remind of solved problems (Score:5, Insightful)
I design automated testing systems. I find myself in the position of re-solving problems that were solved as much as 10 years ago in prior automated testing systems. There's an odd tendency of people to simply passively accept whatever the manufacturer of the new system gives us as somehow "better" or more advanced that what we had, even when the new solution is an obvious fail, or (ahem) undeveloped. So I fix things to make up for the shortcomings of the new software, more or less the same way I fixed them 7 to 10 years ago. Things start working again. Everybody is happy.
I seem to have discovered job security by re-inventing the same thing every 7 years or so in a slightly different form using a different programming language. Of course, that sort of describes the entire computer industry, more or less.
You haven't discovered job security (Score:5, Funny)
Until you write it in cobol.
Why bother (Score:5, Insightful)
Why bother trying to smuggle them back in. Tell them you'll work on figuring out the bits you can't remember and will write a new manual. You get to charge them for time you don't need to spend and avoid getting caught.
Re: (Score:3)
Re:Why bother (Score:5, Insightful)
In this case it really is the safest solution, though. Companies don't want to know that someone has violated their precious document security policies, and would be much happier overpaying to have the data recreated than finding out that someone had a private copy. This guy giving engineers copies of the documents is probably the nice thing to do, but it's a risk that the incentives don't favor. Either he's naive, or knows the engineers he's dealing with well enough to trust that nothing will happen.
Re: (Score:3)
There are things he can't be *told*. Everyone knows that he already knows them, that's why he was hired!
Software Evolution (Score:3, Interesting)
One of the courses of my university study was called 'Software Evolution' in which tutors showcased all kinds of problems with legacy software. Software using languages or platforms no one knows about anymore or lost source code of mission critical systems. Especially older banks and insurance companies suffer from this problem, often choosing the 'wrap around the bit that we know that works' instead of a complete overhaul of a system no one really knows what it does but does it really well. I fear everything we build today will cause the same problems, but worse. Perhaps the old idea that readable code should help people understand its function saves us from this scenario...?
Re: (Score:3)
If you don't know what it does, how can you measure how well it does it?
I think what you meant to say, is that no one really knows what it does, but it continues to return answers that everyone likes.
Perhaps small reusable components might help (Score:4, Insightful)
They could do one thing, just one thing, but do it well...
Now... Where have I heard that philosophy before?
Dear Slashdot admins, (Score:5, Insightful)
More like this, please, and less about the Apple/Andoid/MS/Samsung/**AA suit-of-the-minute. I know, I know, flamewars == pageviews == Step 3, but you've occasionally gotta throw something out there for us old-timers.
Re:Dear Slashdot admins, (Score:5, Insightful)
How about you submit more like this? Or create more like this? Or find a site that is more about this? Or make a site that is more about this? I'd love to see it too but the fact of the matter is that the majority of the people here don't have these kinds of problems nor are they interested. Wait and see. Give this article about 3 days and look at the next closest article about Google/MS/Apple/RIAA and see what gets what comments. Granted, this article will probably get more insightful and thought out comments and a lot less of the memes and trolls than the others but page hits is what the management wants. They don't really care about content.
Outside of the code, all documentation is worthles (Score:5, Insightful)
I currently work for a company that has instituted an incredibly restrictive development methodology they bought from a big consulting firm. It requires multiple forms be filled out for every program written, every requirement for every program written, every request for every change to every program or application written. All of these reviewed by coworkers who are not about to alienate team members, and reviewed by lower management who want to look good to upper management by having everything go smoothly.
These documents are then stored on the LAN. To never, or rarely ever, be read again.
The one thing it does not enforce or require, is meaningful documentation in the code itself.
It doubles, or more, the time it takes to do everything. But it does nothing to stop the mistakes any better than the procedures that went before. If anything, it finds less errors, because we were not given more time to do this double or more amount of work. So time is so compressed, we do not have time to do anything other than get it working and get it in.
One thing it does do very well, is prevent problems from getting fixed. The only people that will start a change effort are those that notice a problem and are affected by it enough to have it cause them problems. Otherwise no one wants to go through the bureaucracy to kick off any sort of change effort, which leaves a lot of ticking time-bombs in the infrastructure configurations, application designs and application code.
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Documentation located anywhere else will be lost, or obsolete many more times than not, before you ever really need it.
If anything, documentation in code should be reviewed by people with absolutely no connection to the application it is for. If it is good enough for them to figure out an understand what is being done, and more importantly why it is being done, only then is it worth anything more than the bytes is written with in storage.
Re: (Score:3, Insightful)
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Deleting documentation is far more important than writing it. There is nothing worse than documentation that is clear, concise, convincing and that assumes something that is ever-so-subtly wrong due to the code base having been changed since the documentation was written. It's better to update the wrong documentation, but deleting it is far better than leaving it in in most cases. So no, people deleting comments are not necessarily fools, especially not if working to a deadline.
Opportunity for more pay (Score:5, Interesting)
I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works.
I certainly hope that you negotiated a much higher salary than before.
I have known people who were fired in a "chain saw" maneuver who proved to have core corporate competencies in their heads. They negotiated for a much higher rate the second time around...
Re:Opportunity for more pay (Score:5, Funny)
I know a guy who, after the company was bought, got the ax along with every other engineer in the place.
As he was working remotely, he had a local copy of the entire repo. With his severance check was a reminder to "destroy all company information in his possession".
Fortunately, he didn't do that because, 2 months later, they came back asking if he had gotten around to doing that because the salesdroids accidentally sold off the main repo server when they liquidated the office equipment.
He greatly enjoyed negotiating the fee for "recreating" the repo that he "didn't have".
Re: (Score:3)
Yes, this happens. A while ago I was made redundant by a Very Large Telephone Company. Said VLTC paid me a nice redundancy, and gave me a system I'd written they didn't want (which made me a nice living for a few years, thanks VLTC). No complaints. Before I left, I archived everything nicely, documented the archives, and handed over several copies.
A year or two later, they called me up and asked if I "happened" to have a copy of the source to a system I'd written. (Yes, it had been in those archived copies)
Re:Opportunity for more pay (Score:4, Insightful)
If you RTFA, you would know if s/he did.
If you had read it, you would know whether to use "he" or "she".
Company rules against removing documents (Score:5, Insightful)
Personally, I'd stick with just what's in my head and not keep any documents, files, etc. from a previous employer. It's just asking for trouble.
Re:Company rules against removing documents (Score:5, Interesting)
Yes, that's what makes it tricky. I wasn't supposed to have these documents, but some important internal drawings have been lost, and I have a copy. The safest solution would be to avoid bringing it up, I agree. Though I'm not blackmailing them for the documents or anything.
I've heard it happen several other times. I've had colleagues who consulted for a company, then a few years later get a call or email asking if they by any chance still have a copy of some old document. Of course they weren't supposed to have kept a copy, but if they accidentally did, it would be really nice if they could send a copy because the company seems to have lost it...
It mostly happens at lower levels of the org chart on a don't-ask/don't-tell basis I think. Lots of people keep personal archives, and sometimes the official archives have to get patched out of them.
Re: (Score:3)
Re: (Score:3)
I would suggest if you have continually worked for the company to simply co
Aliens. (Score:5, Funny)
If the history channel read this story, they would undoubtably conclude that the plant was built with the help of aliens.
Sometimes IT has to help the business (Score:3)
Re:Sometimes IT has to help the business (Score:5, Interesting)
An important task for Historians is to regularly remind the bosses what Historians are for so they don't get sacked
Documentation is key (Score:5, Insightful)
My situation is different, yet similar. I just have so much responsibility for so many things at work, I simply can't remember how everything works. At times it is frightening, especially since year after year there are more things happening and I get older, so it is already inherently more difficult.
I also find myself "protecting" my sanity by almost intentionally forgetting how some things operate, especially those things I don't like.
I think as a manager, one of the most important things to do is learn to effectively document everything. I always knew documentation was important, to the point that I have a log of everything I do at work, every day, that spans 23 years! But those are just "reminders", not proper documentation. It helps to jog memory or memorialize certain facts for reference. Nothing replaces *proper* documentation, though- the kind that is well thought-out, has great detail, and is organized properly. Further, it will have insights into defining the problem and the solution and WHY you selected certain paths and not others.
Proper documentation takes considerable time, and many people think it is a waste of time, especially when one is drowning in projects and deadlines. When you reach the point that you can't handle the load without continuing to document properly, you know you are in trouble. In the short-term, you can "win" by getting more things done. But in the LONG TERM, you are basically screwed, because you will spend a tremendous amount of wasted time later trying to relearn what you did and why. Or worse, trying to defend your decisions and can't because you can't find the information, it doesn't make sense, or it just doesn't exist.
that is why they need Technical writers / writers (Score:3)
to do the wiring part so that the tech people have time to do the tech work.
Control is the problem. (Score:4, Insightful)
The bean-counters and mindless penny pinchers who run most corporations like to keep everything secret and proprietary without fully appreciating the costs associated with maintaining that level of secrecy. The solution is simple, only keep something secret if you've first: taken the effort to understand what it will cost to do so and second: taken the effort to make sure the secret won't actually be forgotten. In short, keep fewer secrets, and keep them a lot better.
That's why the Feds declassify secret documents (Score:3)
It costs money to maintain secrets. If there is no reason to keep it secret anymore it is cheaper not to guard it.
Re: (Score:3)
I don't think the Federal Government is a good example of limiting secrets to save money.
Declassification is under funded, and mainly a fig leaf.
Classifying documents as secret is cheap, and has many bureaucratic benefits - making people and projects look import, shielding failures from review, etc. Thus the default is to classify everything.
I think this applies to other large institutions with secrecy programs as well, such as large corporations.
I don't comment my code anymore (Score:5, Insightful)
i did so slavishly at first but then my boss pointed out that comments are rarely updated with the actual code. a contrived example:
int foo() // always returns 4
{
return 5;
}
now I have a coding style that the simplest fool can understand. that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
assertions are better than comments (Score:5, Interesting)
now one does need to document the expected inputs and results of subroutines. better than commenting is to use assertions as well as unit tests. so for that contrived example, what I would actually do:
int foo()
{
int result = 5;
assert( 4==result);
return result;
}
if you don't maintain the assertions and unit tests with the payload code you'll find out about it right away.
Re:I don't comment my code anymore (Score:5, Interesting)
> that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
IMO, _proper__ commentting discusses WHY you did something. If someone wants to understand HOW it was done, read the code, as you correctly reason. Any half-decent coder should be able to figure out what the code is doing.
EXCEPTION: If you are doing something tricky, then document that. i.e. http://www.codemaestro.com/reviews/9 [codemaestro.com]
Re: (Score:3)
There are often situations when it makes sense to add comments to explain why you did something a certain way based on knowledge of things outside of the local code, like database schema or quirky behavior of a function you're calling. It's easy to comment too much or too little; the right balance is an exercise in judgment and almost an art form itself.
Old news (Score:5, Informative)
this is why you shouldn't outsource (Score:5, Insightful)
if your own employees do the work your company retains the knowledge of how to get it done.
if you outsource you never learn how to do your own work, instead you finance your outsourcing firms efforts to learn how to operate your business. not only do you not gain that institutional memory but that outsourcing firm can now perform that same work for your competitors.
Oh, that hidebound government bureaucracy! (Score:3)
Something like this would never happen in private enterprise, where the invisible hand of the free market weeds out such inefficiencies.
Oh, wait ...
A similar experience (Score:3)
I had a similar experience, from the other side of the spectrum. I was the summer intern doing a project that required digging through the paper archives to find some hardware data that was to be entered into a computer simulation.
It was nowhere to be found. What was documented for this perticular plant was little more than the type of equipment used. Not it's technical ratings or anything. Just what was there. And good luck finding any of that documentation only.
I also found full documentation for a plant that'd been shut down decades ago. Then demolished. And was now a blank wasteland. "In the hope they would be useful"
It is a solved problem (Score:5, Funny)
I'm that for my ex employer... (Score:3)
I worked for an organization from day 1 and was involved in every project we ever worked on. After several years there, that organization wound up merging with another organization and I decided it was time for me to move on.
It's been almost a year since I left, and every day I get 2-3 emails asking me for information, and as a way for them to not feel embarrassed for asking, they actually created a 1/10th time paid position for me and the job title is, literally, "Institutional Memory."
I go in for a couple of hours every few weeks and do a data dump with one team or another, and they're slowly starting to amass documentation as to what was done and why it was done that way.
At my new job the first thing I did when I came onboard was to set us up with an internal wiki for every single project we have and everyone puts notes on the projects, why they're doing things the way they are, etc. on that. It's a good way to have as much of a paper trail as possible.
Ask intel (Score:3, Interesting)
One of the things they do very well is... no, not designing chips, they suck at that, but duplicating fab plants. Down to getting everyone that's gonna work in there up to speed before the plant is finished.
The key is of course not to just build a plant. It's to build at least two plants, at least one in reality and one in the documentation. Fred Brooks (yes, exactly in his book _the mythical man-month_) explains how this works for software and computer hardware. It won't be different for processing plants, except of course that your production numbers are lower. But even with just one physical plant, you need full documentation on how to replicate it, with every task and every job described. And then, of course, you update the documentation along with any updates to the plant, bring it back up to the level where you can xerox the paper into reality again.
The test is exactly that: Can we give this to another engineer and have him duplicate the thing from that? Yes, most software documentation fails that test and is utterly useless as api docs or user manual to boot-- as if those code monkeys don't even read what they wrote, the illiterate bums. And yes, you probably should check every five or ten years if it's still understandable. If that's expensive, well, you get to pay more thirty years down the line. Or you can start from scratch, if you prefer.
Notwithstanding that this documentation archaeology an interesting excercise, but as TFA illustrates, it makes for rather schizophrenic situations, is way out of the usual corporate flow of how things normally go, cost a bundle, takes a lot of time, and is fraught with uncertainty. Care and feeding of the corporate library is future-proofing the IP assets, far more so than filling a war-chest with patents. It's about as sexy as caring for the backup tapes, but you have those too for a very good reason.
"New and improved" disease (Score:4, Insightful)
Accounting, for instance, has been done the same way for almost 500 years, yet there are thousands of accounting packages available, written in dozens of different languages, even to run on the same hardware. Some programs (Quickbooks, for instance) have been "improved" beyond comprehension while still preserving their deficiencies and shortcomings.
Once a problem is solved in programming, it should never have to be solved again. Most of the underlying code in computer systems is based on programs written back in the early days of digital computers. By adding the basic "components" together, more complex systems problems can be solved at a higher scale of interaction and complexity. So, scaling can be a problem. And, how the heck are we supposed to know where to find the original solution to the problem? (Note: There have been attempts to "hash" the common solutions and create unique, searchable identifiers, but I don't know what ever happened to those experiments. The ACM maintains and makes available a "library" of fundamental algorithims, but it is getting unmanageable.).
Now the person coming in to review or improve legacy code has a "complexity" obstacle that prevents him from easily determining the purpose and scope of the code.
In the digital world, programs work on data. Every digital program relies on sequence, alternation, and repetition. Many researchers (Djikstra, Jacopini, Martin and more) have shown that these are directly comparable to concepts in mathematical logic (sequential proof, decision rule, and quantification) and have demonstrated ways to "prove" programs correct. In today's world, we have the power to actually analyze programs, derive the structures, and prove the validity of the results at a meta-level (giving a nod to Goedel on the way). A meta system of this kind should discover what a program or system does, whether it does it correctly, and whether it does ONLY what it is supposed to. Now the problem becomes one of making the results understandable. The various packages that analyze source code and produce UML documentation are a very good start in that direction. I am concerned, because, (as recent demonstrations of Stuxnet and Duqu show) the interaction of flawed systems can have accumulating ill effects on those of us who have to depend more and more on computer systems to control our cars, our environment and the tools we use in our lives.
Not every industrial process is as easy to analyze as software. And the complexity of modern industrial processes makes it even harder to understand once the process has been analyzed. Much of industrial behavior is analog; much of it is probabilistic; and much of it depends on uncontrollable inputs such as human behavior, and human mis-interpretation of the situation. If I was part of the investigating team trying to preserve lost company know-how, I'd probably go back to anything resembling requirements analysis, and work forward from there.
labour relations (Score:4, Interesting)
Many companies have a tiny number of people who manage things, and then a sea of consultants who do things. Often the management of the consultant groups are consultants themselves.And even the managers of the managers are often consultants. The result? Corporate Memento syndrome. No long term memories get formed because the people who do the remembering are gone. Stuff is rar'd or stuf'd or tar balled into a series of docs that sit on a server than is run by an bunch of consultants in IT who have no idea what is on the drive.
Then the drive either fails or is dumped into a bigger storage drive and labeled in a folder "management data 2009" or whatever, and there it rots because no one has any idea what's inside, and the people with access are consultants who don't/can't know.
and why? Because it's cheaper to hire a river of consultants for 6 months than it is to hire people and keep them around for years. The result? Permanent amnesia. All of it will be lost, and no one will care. the essence of 21st century culture - digital business - will be erased and forgotten. Forever. Some would say "Good riddance", but I'm enough of an armchair anthropologist to feel the loss.
Not just engineers, either. (Score:3)
Haha, wrong! Sure, they retained the drawings... digitally. But nobody bothered to check that all of the cross-referenced CAD files got saved, or that the links inside of them them weren't broken when they got moved to near-line storage, or that consultant drawings were saved, or that Jimmy The Intern hadn't linked in a dozen files from his desktop for the sake of expediency and never got around to moving them to central storage before he went back to college, and then IS wiped his account. What should have been a quick dive into the archives turned into a three-month-long wild goose chase. Let's just say that I got really familiar with the large-format scanner and the AutoCAD Reference Manager tool.
Happened to me at NASA... (Score:5, Informative)
I've seen it happen a different way (Score:5, Insightful)
I saw a company lose probably 60% of their IT knowledge base. They announced outsourcing six months in advance, and directed current employees to document their jobs so that offshore workers could take over. Employees instead used the time to find other jobs, as one would expect. Transition was a disaster, and a year later they're still discovering things that the company no longer knows how to do.
One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.
The basic misunderstanding, I believe is thinking that IT is generic procedure-based stuff that anyone could do. There are cases where this is true, but if you are in the business of providing a web based business, your IT *is* your value add, and your success is probably based on in what way your team does things that *aren't* generic.
Re: (Score:3, Informative)
One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.
I've seen the same. I took over a fairly large database project with distributed client software from a previous team. I had about a day to ask questions of that team. I learned most of it on my own; I added a lot of code comments, logging, and documentation as I went. The project was outsourced; they had 30 days of daily sessions and Q&A to learn the project. The team it was outsourced to supposedly had experience with this particular database (it has a client/user interface built on top of the
Re: (Score:3)
When IT cratered after transition, the survivors (of which I was one) attended an all-hands meeting where the suits tried to explain the problems. Their first position was that the former IT people had not documented the processes properly before the layoff, and that's why things were so messed up now. They were shouted down by the crowd. It was wonderful. :-)
Burn the documents? (Score:3)
Re: (Score:3)
Yep. And legal will tell you: You must have a document destruction policy in place and follow it consistently. If not, a the destruction of documentation in the face of a potential lawsuit will look all the more suspicious.
Burn it all. That way a prosecutor can't impute some nefarious intent when some critical evidence goes missing.
The Real Problem - Corporate Attitudes (Score:4)
There have been a lot of nice and informative comments to this point about the various facets and symptoms of the problem, but here is the problem itself: Corporate Attitudes.
1: Companies don't give you time to write peer-reviewed documentation because that doesn't contribute to Time To Market and overall productivity.
2: Companies seldom ask you to train your replacement in the I.T. world because they don't want you around once you realize that you're about to be laid off/fired. Who in I.T. hasn't had the experience yet of being marched off the property under the watchful eye of somebody (I've had it done where they guy actually carried a gun) within an hour of being told that you don't work here any longer. You find that you passwords were disabled during your 15 minute (at most) exit interview and given a box to clean out your work-area under guard.
3: Most importantly, the corporate belief is that nobody is indispensable, and they're willing to prove it with you no matter how much you know that no one else does because the more senior that you were, the less anyone else was watching over what you did in the first place.
In short, you're gone and they simply don't appreciate or value what they lost with you. And find this far more prevalent in I.T. than in most other areas.
The flip-side is that you've probably also been hired at least once to pick up and complete an undocumented project of someone else that they previously let go. Isn't that fun?
Same issue, different format (Score:3)
Some higher up got the bright idea of merging the duplicate documents to save storage space. That was fine. Then they decided to expand it to eliminate entries that hadn't been accessed in 6 months or longer. That was moronic. Sure enough, it wasn't long before it turned around and bit us in the backside. But management wouldn't budge. They refused to restore the old documents, even when it was on something we now needed, nor would they stop deleting stuff that hadn't been accessed for a while.
To add insult to injury, they would then use the count of how many documents you had in the KB against you if it wasn't high enough. Well, Soandso, you only have 120 documents, why should I listen to you instead of Otherguy, he's got 14657 documents and is in charge of the KB cleanup.
You see, the guys doing the KB cleanup and deduplication were going through all the documents and replacing the owner/creator names with their own so they could bloat their own credit, and management (at least the part I had to deal with) was too stupid to understand this. I have no idea how they rationalized that somebody who'd only worked for the company for less than 2 years could have written thousands of documents that were over 4 years old.
Yes, in a somewhat different form, this probably affects all companies in one way or another.
History majors (Score:3)
Maybe that B.A. in History isn't such a bad degree to have after all. Now all you need to do is convince corporations they need to have a history department to manage all of their documentation in such a way that it is discoverable and accessible.
One of my favorite lines is, "Your data is secure. We just can't access it." In other words, if you are going to use a particular medium as an archive, you need to ensure that the data can be read from that medium, in perpetuity. If the technology is in danger of becoming obsolete, the archived data must be moved forward to new media.
Of course, that isn't enough. The data must be indexed in such a way that the information you need can be found. Finding a needle in a haystack is easy with modern technology; finding a hand-drawn flow chart explaining how a particular "black box" in your manufacturing plant converts X to Y (and why it's painted blue) among hundreds of thousands of pages of poorly indexed documentation is exceedingly difficult.
Pump 6 (Score:3)
This reminds me of Pump 6, by Paolo Bacigalupi: "...it made me nervous thinking about all those maintenance warnings glowing down there in the dark: Mercury Extender Seal, Part #5970-34, Damaged, replace. Whatever the hell that meant."
Re: (Score:3)
And "Like looking at Egyptian Hieroglyphs." Anonymous author of TFA should definitely read this.
I've been there (Score:3)
For my entire adult life I worked in the medical diagnostic device industry and somewhere in the late late 80's and electronic documentation & email really started to take over. Then following a series of lawsuits the corporate SOP began to change. We went from loose organization in directories to using versioning tools for documents. And we went from what was essentially unlimited email storage to smaller and smaller... eventually ending up in 2005 with mandated culling policies. (mostly as a proactive defensive legal strategy).
By my nature, I am digital packrat. I still have all the email I have ever received or sent, in curated archives. I still have all the documents I have created. I still have all the code I have ever written. I still have all the design docs I have ever created. And I still have the knowledge management system I created to curate all of that data.
So, my nature and corporate policy really began to conflict more and more strongly. For about 12 years I used my own hardware for backups with my management looking the other way. Eventually I was told the backup strategy had to go and to take all my stuff home. That was replaced by corporate supplied laptop which I routinely took home to backup.
I took early retirement in 2009 and in late 2010 was asked back to resolve a thorny problem with some of the in-house equipment I had a hand designing. The current site manager, who I have a lot of disagreements with but is a nice guy, assessed the parts of my personal archive that I brought in with me as "The largest and most frightening example of industrial espionage he had ever seen"... and wanted to buy it from me so he could destroy it.
Computer Systems Too (Score:4, Interesting)
Sometimes I had to go through a list of servers on a specific subnet, match that up with a list of servers from the routine network scans to see if all were accounted for, if not, I would try to find the server in the server room, get its ID from the chassis, go back to the inventory system, try to find the server there and hopefully an owner too.
Occasionally I could either not find the system in the inventory system or no owner was assigned. I would then have a private chat with the hardware people responsible for the area the servers were placed and suddenly the relevant servers would have an "unscheduled network outage", i.e. the operator would unplug the system and we would wait for somebody to start screaming. If that did not happen within a day or so, we would dump the system's data to the backup system and reassign the server. This time properly documented.
Re:NASA (Score:5, Informative)
NASA has lost ALL of it's Saturn V knowledge.
Uh, no it hasn't. It may not be readily accessible, but there are (literally) tons of Saturn V information still available.
The scientist have died, the documents have rotten and been lost.
That'll be a surprise to the people who've put many gigabytes of Apollo-era documents on ntrs.nasa.gov.
What they have lost is much of the software; the Apollo Guidance Computer software only exists because some of the programmers had old listings that were OCR-ed, and the Saturn guidance computer software seems to be long gone.. there are a couple of computers left that may still have the code in their core memory, but so far no-one has been brave enough to try reading it out since that's a destructive operation and if you screw up you won't get a second chance.
Fortunately if someone was to try to rebuild a Saturn V they'd use completely new electronics and software anyway. Heck, it would probably have to run Windows...
Re:NASA (Score:5, Informative)
This is incorrect. When I was working for Boeing, who was contracting to NASA, I saw the Saturn V drawings with my own eyes, in the data repository building at the Marshall Space Flight Center in Huntsville, AL. They are punch card aperture cards, which is IBM style punch cards with a piece of microfilm stuck into it. If you need a paper copy, they print from the microfilm. There are about 3 million of those cards, filling stacks of drawers, with all the drawings and many of the documents in them.
On the Space Station project, which I worked on for many years, we had a data vault at Boeing, which was literally a vault in the basement, where all the project data got copies stored, and we of course had to supply copies to NASA, who then put it in their repository. If there is one thing government is good at, is storing documents.