Institutional Memory and Reverse Smuggling

An anonymous reader writes "Anyone who's worked in a large engineering firm is familiar with institutional amnesia. Things get built, and then forgotten. Documentation is supposed to help, but rots, is lost, or uses obsolete methods and notation nobody understands anymore. I recently found myself in a strange position, rehired as a consultant with the unofficial job of reminding the company how an old plant works. I even have some personal copies of documents they seem to have lost, which I have to awkwardly smuggle back in. I don't find these kinds of experience written about often, but I'm convinced they're more common than you'd think."

Old news

[DingerX]

Primo Levi, "Chrome" in The Periodical Table describes a similar problem with a paint formula at his factory in the 1950s. Evidently, during the war, they had a QC problem, included some chemical to correct for that, and the formula became a cargo cult, long after anyone knew what it was for. Someone changes some other factor, and their batch of paint gets the consistency of liver. So our hero had to reverse-engineer the trade secrets of a previous generation, back when he was working in the lab at Auschwitz.

Re:NASA

[0123456]

NASA has lost ALL of it's Saturn V knowledge.

Uh, no it hasn't. It may not be readily accessible, but there are (literally) tons of Saturn V information still available.

The scientist have died, the documents have rotten and been lost.

That'll be a surprise to the people who've put many gigabytes of Apollo-era documents on ntrs.nasa.gov.

What they have lost is much of the software; the Apollo Guidance Computer software only exists because some of the programmers had old listings that were OCR-ed, and the Saturn guidance computer software seems to be long gone.. there are a couple of computers left that may still have the code in their core memory, but so far no-one has been brave enough to try reading it out since that's a destructive operation and if you screw up you won't get a second chance.

Fortunately if someone was to try to rebuild a Saturn V they'd use completely new electronics and software anyway. Heck, it would probably have to run Windows...

Re:I see this in code I work on all the time

[heypete]

It's a reference to this scene [youtube.com] from the Pixar movie "Up".

Happened to me at NASA...

[JetScootr]

I worked 30 years in astronaut training facility (full-fidelity simulators), and wrote many many documents on software that I wrote. I always kept my own digital copies, of course. Over the years, the contracts changed hands many times, and different document systems were implemented, and "all" documents were "always" converted from old to new. I was never able to later re-locate *any* document I had submitted to *any* of the document systems. So my copies of my documents were the only ones that actually existed that I knew of. This included meeting minutes, peer review notes, design and 'as-delivered' documents. So I think institutional amnesia is more the norm, and actual memory beyond 3-5 years is rare.

Re:NASA

[DanielRavenNest]

This is incorrect. When I was working for Boeing, who was contracting to NASA, I saw the Saturn V drawings with my own eyes, in the data repository building at the Marshall Space Flight Center in Huntsville, AL. They are punch card aperture cards, which is IBM style punch cards with a piece of microfilm stuck into it. If you need a paper copy, they print from the microfilm. There are about 3 million of those cards, filling stacks of drawers, with all the drawings and many of the documents in them.

On the Space Station project, which I worked on for many years, we had a data vault at Boeing, which was literally a vault in the basement, where all the project data got copies stored, and we of course had to supply copies to NASA, who then put it in their repository. If there is one thing government is good at, is storing documents.

Re:Absolutely true

[Mr Z]

Where I work, we've mined many products which failed to reach the market for good ideas. Often, there were many good ideas in there, it was just the package as a whole that didn't work out. (Either too early, too late, too big or too small.) But, we've also had good continuity among our key folks (myself included), so that probably explains why it ends up working. We understood why we put those features in failed product X, and so we understand how they'd work in new, more viable product Y. We also understand at least something about why X failed, so we can try to avoid it on Y.

But again, that does rely on institutional memory to make it work.

Re:I've seen it happen a different way

[brhalltx]

One of the mistakes made was to assume that everything the old team did could be described as a procedure that a junior offshore employee could follow. This is a fallacy at a basic level, and shows a basic misunderstanding of IT. Knowledge is more than just memorization of individual procedures, it's information about topology, architecture, the flow of information, how the business works, and where the knowledge points are in the organization. (Who knows what.) Degradation of that knowledge base began on the day outsourcing was announced, and by the time transition arrived, there wasn't much left.

I've seen the same. I took over a fairly large database project with distributed client software from a previous team. I had about a day to ask questions of that team. I learned most of it on my own; I added a lot of code comments, logging, and documentation as I went. The project was outsourced; they had 30 days of daily sessions and Q&A to learn the project. The team it was outsourced to supposedly had experience with this particular database (it has a client/user interface built on top of the database engine). They asked a lot of questions (all answered), received all of the documentation, had access to test servers for testing code changes, and managed to screw it up within a couple of days of taking it over. The support part of the team starting contacting me for help... After I'd been laid off. Not much I could do to help for free... They, of course, then promptly blamed me, claiming they hadn't been trained on whatever they screwed up. I heard it took them six months to get it straightened out again. You get what you pay for.