Institutional Memory and Reverse Smuggling 312
An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."
This problem is quite common. (Score:5, Insightful)
I see this in code I work on all the time (Score:5, Insightful)
Re:This problem is quite common. (Score:5, Insightful)
Yep, It happens all the time everywhere.
What I find most troubling about this is that most company don't recognize the problem or are not even aware of it? I think the mentality that "anyone can be replaced so we don't have to create incentive to retain them" is going a bit too far.
Because the real problem is in fact employee turnover.
I have to remind of solved problems (Score:5, Insightful)
I design automated testing systems. I find myself in the position of re-solving problems that were solved as much as 10 years ago in prior automated testing systems. There's an odd tendency of people to simply passively accept whatever the manufacturer of the new system gives us as somehow "better" or more advanced that what we had, even when the new solution is an obvious fail, or (ahem) undeveloped. So I fix things to make up for the shortcomings of the new software, more or less the same way I fixed them 7 to 10 years ago. Things start working again. Everybody is happy.
I seem to have discovered job security by re-inventing the same thing every 7 years or so in a slightly different form using a different programming language. Of course, that sort of describes the entire computer industry, more or less.
Why bother (Score:5, Insightful)
Why bother trying to smuggle them back in. Tell them you'll work on figuring out the bits you can't remember and will write a new manual. You get to charge them for time you don't need to spend and avoid getting caught.
Dear Slashdot admins, (Score:5, Insightful)
More like this, please, and less about the Apple/Andoid/MS/Samsung/**AA suit-of-the-minute. I know, I know, flamewars == pageviews == Step 3, but you've occasionally gotta throw something out there for us old-timers.
Outside of the code, all documentation is worthles (Score:5, Insightful)
I currently work for a company that has instituted an incredibly restrictive development methodology they bought from a big consulting firm. It requires multiple forms be filled out for every program written, every requirement for every program written, every request for every change to every program or application written. All of these reviewed by coworkers who are not about to alienate team members, and reviewed by lower management who want to look good to upper management by having everything go smoothly.
These documents are then stored on the LAN. To never, or rarely ever, be read again.
The one thing it does not enforce or require, is meaningful documentation in the code itself.
It doubles, or more, the time it takes to do everything. But it does nothing to stop the mistakes any better than the procedures that went before. If anything, it finds less errors, because we were not given more time to do this double or more amount of work. So time is so compressed, we do not have time to do anything other than get it working and get it in.
One thing it does do very well, is prevent problems from getting fixed. The only people that will start a change effort are those that notice a problem and are affected by it enough to have it cause them problems. Otherwise no one wants to go through the bureaucracy to kick off any sort of change effort, which leaves a lot of ticking time-bombs in the infrastructure configurations, application designs and application code.
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Documentation located anywhere else will be lost, or obsolete many more times than not, before you ever really need it.
If anything, documentation in code should be reviewed by people with absolutely no connection to the application it is for. If it is good enough for them to figure out an understand what is being done, and more importantly why it is being done, only then is it worth anything more than the bytes is written with in storage.
Company rules against removing documents (Score:5, Insightful)
Personally, I'd stick with just what's in my head and not keep any documents, files, etc. from a previous employer. It's just asking for trouble.
Re:I see this in code I work on all the time (Score:5, Insightful)
Even worse, it sometimes turns out that YOU wrote the code 10+ years ago, and you have absolutely no clue why it was written that way.
Documentation is key (Score:5, Insightful)
My situation is different, yet similar. I just have so much responsibility for so many things at work, I simply can't remember how everything works. At times it is frightening, especially since year after year there are more things happening and I get older, so it is already inherently more difficult.
I also find myself "protecting" my sanity by almost intentionally forgetting how some things operate, especially those things I don't like.
I think as a manager, one of the most important things to do is learn to effectively document everything. I always knew documentation was important, to the point that I have a log of everything I do at work, every day, that spans 23 years! But those are just "reminders", not proper documentation. It helps to jog memory or memorialize certain facts for reference. Nothing replaces *proper* documentation, though- the kind that is well thought-out, has great detail, and is organized properly. Further, it will have insights into defining the problem and the solution and WHY you selected certain paths and not others.
Proper documentation takes considerable time, and many people think it is a waste of time, especially when one is drowning in projects and deadlines. When you reach the point that you can't handle the load without continuing to document properly, you know you are in trouble. In the short-term, you can "win" by getting more things done. But in the LONG TERM, you are basically screwed, because you will spend a tremendous amount of wasted time later trying to relearn what you did and why. Or worse, trying to defend your decisions and can't because you can't find the information, it doesn't make sense, or it just doesn't exist.
Control is the problem. (Score:4, Insightful)
The bean-counters and mindless penny pinchers who run most corporations like to keep everything secret and proprietary without fully appreciating the costs associated with maintaining that level of secrecy. The solution is simple, only keep something secret if you've first: taken the effort to understand what it will cost to do so and second: taken the effort to make sure the secret won't actually be forgotten. In short, keep fewer secrets, and keep them a lot better.
I don't comment my code anymore (Score:5, Insightful)
i did so slavishly at first but then my boss pointed out that comments are rarely updated with the actual code. a contrived example:
int foo() // always returns 4
{
return 5;
}
now I have a coding style that the simplest fool can understand. that's quite a different thing than the arrogant coders who feel that their colleagues are incompetent if they cannot follow uncommented code. if someone cannot follow my code then I feel I have done something wrong, not them.
this is why you shouldn't outsource (Score:5, Insightful)
if your own employees do the work your company retains the knowledge of how to get it done.
if you outsource you never learn how to do your own work, instead you finance your outsourcing firms efforts to learn how to operate your business. not only do you not gain that institutional memory but that outsourcing firm can now perform that same work for your competitors.
Re:Dear Slashdot admins, (Score:5, Insightful)
How about you submit more like this? Or create more like this? Or find a site that is more about this? Or make a site that is more about this? I'd love to see it too but the fact of the matter is that the majority of the people here don't have these kinds of problems nor are they interested. Wait and see. Give this article about 3 days and look at the next closest article about Google/MS/Apple/RIAA and see what gets what comments. Granted, this article will probably get more insightful and thought out comments and a lot less of the memes and trolls than the others but page hits is what the management wants. They don't really care about content.
Re:Why bother (Score:5, Insightful)
In this case it really is the safest solution, though. Companies don't want to know that someone has violated their precious document security policies, and would be much happier overpaying to have the data recreated than finding out that someone had a private copy. This guy giving engineers copies of the documents is probably the nice thing to do, but it's a risk that the incentives don't favor. Either he's naive, or knows the engineers he's dealing with well enough to trust that nothing will happen.
Re:Outside of the code, all documentation is worth (Score:3, Insightful)
The only place documentation is good, is if it is meaningful, and in the code, where it is readily findable and far less likely to get lost, short of some fool deleting it.
Deleting documentation is far more important than writing it. There is nothing worse than documentation that is clear, concise, convincing and that assumes something that is ever-so-subtly wrong due to the code base having been changed since the documentation was written. It's better to update the wrong documentation, but deleting it is far better than leaving it in in most cases. So no, people deleting comments are not necessarily fools, especially not if working to a deadline.
Perhaps small reusable components might help (Score:4, Insightful)
They could do one thing, just one thing, but do it well...
Now... Where have I heard that philosophy before?
Re:Opportunity for more pay (Score:4, Insightful)
If you RTFA, you would know if s/he did.
If you had read it, you would know whether to use "he" or "she".
Re:So what's the problem then? (Score:5, Insightful)
Management don't care, why should you? It really can't be all that much of a problem.
The manager who didn't document anything finished ahead of schedule and below budget. He got a big bonus and moved on to another job at a different company based on his reputation for delivering early and cheap.
The manager who comes in to organise an upgrade to that software ends up taking much longer than expected and going well over budget because nothing was documented in the initial spaghetti code. He gets fired for being incompetent.
Know-how and know-why... (Score:5, Insightful)
One of the uncomfortable truths (uncomfortable for MBA cost minimizers) is that know-how is between the ears. It is not in the manuals or specifications, which merely prove that their writers had the know-how. Even more important is the know-why, which is part of the institutional memory which also resides between the ears.
I have witnessed some "technology transfers" where a set of working products was transferred with documentation, design data, and much background information to a group in another region (mostly EU/US transfers). The first group is then wound down or redeployed. The new group does fine at first, until a new version of a product is needed. Then they always make mistakes the original designers would consider childish or stupid. They don't know the why behind design decisions, and don't know what to avoid. In my experience, it takes 5-10 years to build a decent product design group from scratch. On the other hand, an existing design group will sustain itself by mentoring and guidance of new inductees by experienced members.
The only successful technology transfer which I participated in was one where a few senior designers were transferred with the set of products they were responsible for. It was far from cheap in up-front costs, but it avoided the crises and financial disasters which afflicted the other transfers after a year or two.
Re:I don't comment my code anymore (Score:2, Insightful)
int foo()
{
return 5;
}
Hmm. // always returns 4
$ svn blame
1001 mcrawford int foo()
1001 mcrawford {
1001 mcrawford
1001 mcrawford return 5;
1001 mcrawford }
As I suspected.
"New and improved" disease (Score:4, Insightful)
Accounting, for instance, has been done the same way for almost 500 years, yet there are thousands of accounting packages available, written in dozens of different languages, even to run on the same hardware. Some programs (Quickbooks, for instance) have been "improved" beyond comprehension while still preserving their deficiencies and shortcomings.
Once a problem is solved in programming, it should never have to be solved again. Most of the underlying code in computer systems is based on programs written back in the early days of digital computers. By adding the basic "components" together, more complex systems problems can be solved at a higher scale of interaction and complexity. So, scaling can be a problem. And, how the heck are we supposed to know where to find the original solution to the problem? (Note: There have been attempts to "hash" the common solutions and create unique, searchable identifiers, but I don't know what ever happened to those experiments. The ACM maintains and makes available a "library" of fundamental algorithims, but it is getting unmanageable.).
Now the person coming in to review or improve legacy code has a "complexity" obstacle that prevents him from easily determining the purpose and scope of the code.
In the digital world, programs work on data. Every digital program relies on sequence, alternation, and repetition. Many researchers (Djikstra, Jacopini, Martin and more) have shown that these are directly comparable to concepts in mathematical logic (sequential proof, decision rule, and quantification) and have demonstrated ways to "prove" programs correct. In today's world, we have the power to actually analyze programs, derive the structures, and prove the validity of the results at a meta-level (giving a nod to Goedel on the way). A meta system of this kind should discover what a program or system does, whether it does it correctly, and whether it does ONLY what it is supposed to. Now the problem becomes one of making the results understandable. The various packages that analyze source code and produce UML documentation are a very good start in that direction. I am concerned, because, (as recent demonstrations of Stuxnet and Duqu show) the interaction of flawed systems can have accumulating ill effects on those of us who have to depend more and more on computer systems to control our cars, our environment and the tools we use in our lives.
Not every industrial process is as easy to analyze as software. And the complexity of modern industrial processes makes it even harder to understand once the process has been analyzed. Much of industrial behavior is analog; much of it is probabilistic; and much of it depends on uncontrollable inputs such as human behavior, and human mis-interpretation of the situation. If I was part of the investigating team trying to preserve lost company know-how, I'd probably go back to anything resembling requirements analysis, and work forward from there.
I've seen it happen a different way (Score:5, Insightful)
I saw a company lose probably 60% of their IT knowledge base. They announced outsourcing six months in advance, and directed current employees to document their jobs so that offshore workers could take over. Employees instead used the time to find other jobs, as one would expect. Transition was a disaster, and a year later they're still discovering things that the company no longer knows how to do.
One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.
The basic misunderstanding, I believe is thinking that IT is generic procedure-based stuff that anyone could do. There are cases where this is true, but if you are in the business of providing a web based business, your IT *is* your value add, and your success is probably based on in what way your team does things that *aren't* generic.
Re:This problem is quite common. (Score:4, Insightful)
Because the real problem is in fact employee turnover.
When you're talking about a scale of decades, including economic downturns with hiring freezes and whatnot, employee turnover is a fact of life. No one works in the same place forever, no matter how good the incentives are to stay. Even if you can stop them from changing jobs, they might retire. Even if you manage to make their job so compelling that they don't retire, death will eventually get them. And good luck getting ghosts to work for you in any meaningful way.
Its purely a management problem. Data retention over long periods of time is a real challenge, and one that most people don't care about because well.. out of sight, out of mind. Put a token effort towards data management when someone important is looking (investors, higher management, whatever). And then redirect the money to more immediate problems as soon as the heat is off.
Remember, the quarterly report is king. Trying to predict and prevent problems that may or may not crop up 10 years from now seems less relevant when you have to make not only profits, but record profits year after year or risk your stock price plummeting.
Re:This problem is quite common. (Score:5, Insightful)
"Because the real problem is in fact employee turnover."
That's not as much a problem as a reality you'd better learn to live with.
Look: there's only two ways for an employer-employee relationship to end up: either the company closes business or the employee goes out. Since the company as an entity isn't so interested on the afterthoughts of its own end, there's only an interesting case to consider: the employee will go away. That's as valid for the janitor as is for the general manager.
So it is not "anyone can be replaced" but "anyone *will* be replaced".
Re:And so, civilisation ends. (Score:5, Insightful)
"But of course, this would imply a corporation with the ability to see beyond the tip of its own nose, which is rare."
The problem is not only corporate culture but societal.
Say Big Corp A has in fact the ability to see beyond the tip of its own nose and then spares efforts to prepare for the next decade which, of course, impacts its bottom line for this quarter. At the same time, Big Corp B doesn't see beyond the tip of its own nose and this quarter's results look better than that from Big Corp A. Then, all of a sudden, everybody, *you included*, sell their stocks from Big Corp A to buy stocks from Big Corp B which makes Big Corp B even bigger while Big Corp A is not a big corp any more.
Which, ironically, makes Big Corp B to be the one with the ability to see beyond the tip of its own nose because since Big Corp A is a big corp no more, its investment for the future is now moot and shouldn't have been done to start with.
That's exactly why you see from time to time companies exploding basically out of nothing (say Google or Facebook) despite of the fact that other Big Corps near the business niche have much more than enough muscle to crunch them -their stockholders wouldn't allow for the investment. And that's exactly way you see those Big Corps investing much more in rising the bars for new entries with lawyers, IP, patents and lobying legislators than in real R&D.
Re:So what's the problem then? (Score:4, Insightful)
I have seen a lot of examples where the problem was not lack of documentation but rather that the available documentation documented the obvious things (this is a piece of paper) instead of this is the formula for turning garbage to gold. It is very important to document what the code accomplishes instead of documenting each line of code.
Good
document once a purchase order has been approved, prevent changes to the line items
Bad
instead of if the PO status = 'APPR' and the item number or price is changed, raise an exception.