Most Digital Content Not Stable

brunes69 writes "The CBC is running an article profiling the problems with archiving digital data in New Brunswick's provincial archives. Quote from the story: 'I've had audio tape come into the archives, for example, that had been submerged in water in floods and the tape was so swollen it went off the reel, and yet we were able to recover that. We were able to take that off and dry it out and play it back. If a CD had one-tenth of one per cent of the damage on one of those reels, it wouldn't play, period. The whole thing would be corrupted'. Given the difficulties with preserving digital data, is it really the medium we should be using for archival purposes?"

oh, just

[superwiz]

let's play it all by memory. Seriosly, do we really have a choice? The more densely we pack the information that more of a chance it has for corruption. The "CD" mentioned by the article has effectively 700 minutes of music of the same quality as the 60 minute tape.

First

[WormholeFiend]

we need to realise that nothing lasts forever.

Then, we can figure out the most cost-effective medium to record stuff on, with determined re-archival cycles.

They could try harder

[Waffle Iron]

The CD wouldn't play with an off-the-shelf CD player. That doesn't mean that a special "archaeological" CD player can't be built that would perform extensive microscopic image analysis of the disk surface in order to read the data in the face of extensive corruption.

Some analog technologies, like old color films, have also degraded and need image enhancement to recover the original content.

1% = Total Loss?

[JesseL]

If losing 1% of the data on a CD means the data is a total loss, doesn't that say to you that you should be using a file system and data formats with more redundancy and parity?

Of course for the ultimate in durable electronically readable storage you should be burning everything to PROMs [wikipedia.org].

Re:No.

[Anonymous Coward]

Yup, DDS tapes renewed every 5-7 years is the only way to go.

Tape, whether analog or digital, usually suffers "bleed through" effects after about 7 years.

Of course, CD and DVD are disposable, temporary media, though their durability depends a lot on the quality (price) of the media.

I guess Denis Noel is not an archivist after all, despite being hired as one. Government has a knack for hiring the lest competent people possible. I suspect it is a strategy as the less a government actually achieves, the smaller the political risk and the greater the chances of re-election. Don't you just love democracy?

Remember the "Domesday Book"

[hopbine]

In the 1980's they digitized the Domesday Book. Trouble was the format they used is now obsololete. The good news (apart from still having the origional) they have re-inveted the wheel. http://news.bbc.co.uk/2/hi/technology/2534391.stm [bbc.co.uk] for details.

Re:Precisely

[skoaldipper]

Here's [computerworld.com] an alternate article which might shed some light:

"Unlike pressed original CDs, burned CDs have a relatively short life span of between two to five years, depending on the quality of the CD,"[...]The problem with hard drives, he said, is not so much the disk itself as it is the disk bearing, which has a positioning function similar to a ball bearing.[...]Gerecke (a physicist and storage expert at IBM Deutschland GmbH) suggests using magnetic tapes, which, he claims, can have a life span of 30 to 100 years, depending on their quality.

Which raises a few questions:

1. Even if a 1000 backups are made today, unless each successive backup (say) 2-5 years from now includes _all_ the information from today, those original 1000 backups are quite useless.

2. Having been a victim of HD fluid bearing loss (from a brand new Maxtor drive only lasting 16 months), even HD(s) aren't reliable.

3. As long as item 1 is handled by ever increasing storage capacities, it's not an issue. However, redundancy doesn't stop at 2 (hd -> CD). It's better to have a long term solution like magnetic tapes (or other) imo.

data type is more important than medium

[benmoreassynt]

This is a dual problem:

1) Digital data needs to be moved about once every 5 years onto a new physical store, disk, whatever. Think of the amount of data sitting around on floppy disks that is being lost as we speak.

2) Data has to be recorded in a way that that presumes whatever software you use to create it will not exist in the future. Anyone who saved their life's work in some ancient binary word processor file will know what I mean. For most computer-based data storage that requires data be stored somewhere in plain text, and using as open a format of 'markup' as possible, if any.

In effect, from a historical/archival point of view, data does not exist unless it is kept in at least two places at all times, and unless whatever bit of software you use to create it can also save it in a non-binary format of some sort for access for future generations who don't have a copy of your software.

Ok, that does not pertain to sound recordings or images, but even then some sort of 'permanent' standard is essential for all data.

I used to work with medieval documents written on vellum - sheep skin. The original Domesday book was written on vellum, and is as readable today as it was in 1150. (It also doesn't need a power supply to work!) Meanwhile the digital 'Domesday' Laser Disk made in the early 80s in the UK had to be saved from oblivion a few years ago (with a great deal of work) because the computers and hardware that it was created to work with were utterly obselete. Fortunately, and unusually, someone realised the problem before it was too late.

Re:That's nothing, think of DRM

[CyberLord Seven]

See last months "Wired" for a nice article on Mayan/Aztec (don't remember which) practice of using KNOTS as a method of storage/writing.

You are indeed correct that "there are many methods of storage that could be easily overlooked and mistaken for simple art."

When our house burned down...

[dclozier]

About 6 years ago our home burned down. It was a complete loss. Once we were able to pick through what remained I came across some jewel cases containing some backup data. These cases were next to some cassette tapes. The jewel cases had warped considerably but many of the CDs inside were still flat and usable. The cassette tapes were warped as well but the tape inside looked like it had shriveled from the heat. What ever the type of plastic the CDs were made from withstood heat the cassettes could not.

Just tossing this out there. The topic made me remember the pleasure of finding some stuff in tact. :)

Re:It's already happened/happening.

[dhasenan]

So don't convert the content to the new format -- port the viewer / player / codec. Then if the new formats are that much better (or simply more common), convert to the new one whenever you play back anything in the old format. That'd be much cheaper on computation and disk transfers (or at least it would deamortize them), though it might incur additional lag in playback.

Article title incorrect

[RedBear]

I think what the original poster meant to say was, "With digital you either get a perfect copy, or a corrupt copy. With analog you always get a corrupt copy."

Digital content isn't unstable, it's just more sensitive to corruption because in general software expects to be able to extract a perfect copy every time, rather than a near-perfect copy. Whether you can recover partially corrupted digital data depends on several things:

A) Choice of filesystem (journaling, error correction, built-in redundancy)
B) Choice of media (CD/DVD bad unless multiple copies you have, hmm?)
C) Choice of physical storage method and location (store CD/DVD out of sunlight, vertical in jewel case)
D) Choice of archival file formats (PAR2, anyone?)
E) Choice of hardware (some hardware is more robust)
F) Choice of software used to read the media (most software gives up too easily)

The cure:
1. Use the right media (with phsyical redundancy measures to counter physical damage).
2. Use a robust filesystem (preferably with error correction and redundancy measures also to counter minor physical damage).
3. Use a robust file format specifically designed for archiving data (again with built-in redundancy measures and compartmentalized structure that can work around partial corruption).
4. Use hardware that has a high tolerance for physical or digital media corruption.
5. Use software specifically designed to keep trying to extract data even after encountering partial corruption (like Unstoppable Copier [roadkil.net]).

All that being said, if you were to say that digital media, file formats, filesystems, hardware and software are too fragile, I would have to agree. There is far too little fault tolerance and redundancy built into digital storage media, hardware, software, filesystems and file formats. A lot could definitely be improved for the future. But calling most digital content unstable because a CD got scratched is disingenuous at best.

Re:Multiple identical copies?

[mgabrys_sf]

Of course I wrote the above muse before checking google. Here's the theory in practice. Elegant as hell I might add:

http://www.norsam.com/hdrosetta.htm [norsam.com]