Slashdot Log In
Most Digital Content Not Stable
Posted by
Zonk
on Tue Mar 20, 2007 11:55 AM
from the define-stable dept.
from the define-stable dept.
brunes69 writes "The CBC is running an article profiling the problems with archiving digital data in New Brunswick's provincial archives. Quote from the story: 'I've had audio tape come into the archives, for example, that had been submerged in water in floods and the tape was so swollen it went off the reel, and yet we were able to recover that. We were able to take that off and dry it out and play it back. If a CD had one-tenth of one per cent of the damage on one of those reels, it wouldn't play, period. The whole thing would be corrupted'. Given the difficulties with preserving digital data, is it really the medium we should be using for archival purposes?"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
That's nothing, think of DRM (Score:5, Insightful)
Re:That's nothing, think of DRM (Score:5, Funny)
Parent
Re: (Score:3, Informative)
I think that was the point behind "depending on how you define American" -- the GP was referring to the urbanized cultures of Mexico, Central and South America that had writing systems that they were forced to give up along with the rest of their culture.
Re:That's nothing, think of DRM (Score:4, Insightful)
Ballsy words for an Anonymous Coward. Hopefully you'd stick to them if your hometown were invaded.
Parent
Re: (Score:3, Informative)
Best summed up by Chief Seattle, in 1854: "This we know: the earth does not belong to man, man belongs to the earth. All things are connected like the blood that unites us all. Man did not weave the web of life, he is merely a strand in it. Whatever he does to the web, he does to himself."
Re: (Score:3, Insightful)
Well, I'm 1/8th Native American (but 7/8ths White) if that counts for anything, but this is always overblown. Whites/europeans came in and conquered the land. That's what people have done throughout all of recorded history. The Romans Conquered the Greeks, the Normans conquered the Saxons, etc. The list goes on and on. The case has ALWAYS been that if some other nation wanted your land and you couldn't stand up to them in a mil
how did we get so far offtopic? (Score:3, Insightful)
That being said. What's done is done. It should be remembered so we learn from those horrible mistakes. It shouldn't be a constant source of guilt to be used against people that had no part in it. The same goes for slavery, genocid
Re:That's nothing, think of DRM (Score:5, Insightful)
As a person who loves to study European antiquity I would point out some flaws in this thinking...
1. When the Romans conquered the Greeks they actually adopted Greek culture and didn't kill off the Greeks.
2. When the Normans conquered the Saxons they didn't kill off the Saxons nor really conquered their land as much as just intermarried with them (Hence Anglo-Saxon Culture)
The only whole sale Genocides that history can come up with is the Crusaders massacre of Jerusalem (which wasn't really as much as hatred of Muslims as it was starving Europeans killing off everyone in the city regardless of religion out of rage of having to starve in the desert for several months) and then the Mongol sack of Baghdad which wasn't over so much as land, but out of spite of the execution of Mongol diplomats (considering they burned and salted the lands made the "take your lands" point of conquering sort of a non-issue).
The genocide and seizure of lands in this scale was never really seen before until the colonization of Americas. It wasn't as much as the Indians could not defend them as much as it was that the westerners thought they were subhuman.
Which sadly we saw again in the European theatre in WW2.
Parent
Re: (Score:3, Insightful)
Multiple identical copies? (Score:5, Insightful)
Re:Multiple identical copies? (Score:5, Informative)
Parent
Every Superman has his Kryptonite (Score:5, Insightful)
Parent
Precisely (Score:3, Insightful)
As you said, the great thing about digital data is that is can be replaced cheaply, perfectly, and spread around. It's resilience isn't in the one copy lasting 1000 years, it is in having copies everywhere, so no even short of nuclear war can eliminate them all, and maybe not even then.
This also is th
It's already happened/happening. (Score:5, Insightful)
I think you're missing an important element here. As you move along in time, the volume of data that must be converted to the format du jour only gets bigger and bigger.
For a single person, it's probably not too bad. I, too, have pretty much everything I ever wrote since I first got a computer, and every few years I've committed to rolling the whole thing onto new media. So I've gone from offline backups on floppies, to Zip disks (in retrospect a mistake), to CDs, to DVD-R, and now to DVD+R (the -R discs were crappy and I've since heard that +R is a superior format anyway). This isn't much trouble, because the amount of data I have to backup hasn't really grown that much faster than the data density of available media. I'm probably up to a couple of DVDs for the stuff I really, really care about, maybe a binder if I include all the photos and video.
But what's a basic Saturday-afternoon copy-and-burn job for an individual is a Sisyphean task for a large government agency or library, particularly one who is constantly generating new content. I've seen places that could barely keep up with archiving the stuff they were producing, much less roll their vast archives forward onto new media. So they'd have vaults of hard drives, sitting next to DLT cassettes, next to IBM 3480, next to racks of old half-inch open-reel tapes. Probably back in some dark corner there were piles of punched cards; it really wouldn't surprise me. The problem of data loss due to unreadable formats isn't some abstract 'maybe,' it's already happened in a lot of places (but nobody really wants to talk about it, so it mostly gets buried and whatever's on the tapes gets written off).
The reason why there's so much interest in preservable formats is because while it may not be strictly impossible to constantly roll old backups and archives forward, it's very hard, and requires vast amounts of effort and expense. If you have a backup that's being written into a format that you know is going to be readable for a long time, even if it's more expensive to write initially, you can save a lot of money and time down the road by not having to copy it forward as often.
People may get a little shrill when they're talking about these issues, but they're quite real.
Parent
Re:Multiple identical copies? (Score:5, Insightful)
Also I'm certain for every analog horror story there is a digital lucky story (and vice versa). Not to mention digital encodings usually have some kind of redundancy. A small scrach does nothing but the same scratch on an lp forever destroys some part of the track. I wont even go into the magic of data restoration (which the author ignores). There's really no 'tough medium for the ages' out there that can do it all. Just complaints and blind-luck stories.
Parent
Re: (Score:3, Funny)
What I do is take files I care about, encrypt them, rename the file to something tempting like "Cheerleader Sex Orgy XXXIV.avi," note the MD5 (sticky note on the next of the monitor), and share it on a P2P network.
Instant distributed backup! 8D
Re: (Score:3, Funny)
Re: (Score:3, Insightful)
Re:Multiple identical copies? (Score:4, Funny)
The larger systems could also provide mirroring by interfacing with each other as directed by chemical interactions in order to preserve original data as well as integrate new data that may be useful in assuring that future units are even more resilient to any sorts of flaws or possible malfunction caused by inappropriate chemical input. The key to all of this is going to be to make sure that the larger units are impelled to continue the duplication and exchange of data ad infinitum. To do that, there should be some sort of mutual benefit that the engaged units acquire from the mirroring. Multiple levels of mutual benefit would likely be more successful than just one level. So I propose that at a base level, the units should be programmed with routines that make them feel more or less successful whenever a mirroring connection is attempted. I know that sounds strange, but it should be a pretty simple subroutine and will at least get the units to attempt mirroring.
The next level would also be an expansion of the data mirroring to the actual manufacture of a tertiary (or even more) unit that contains selected data from both origination units. As part of the mutual benefit relationship between units, the origination units should be programmed to protect the manufactured unit in order to safeguard its data as it would be the freshest copy (chemically speaking) and therefore more viable. So the relationship between origination units and next generation manufactured units would be that of security and stability from the origination units as applied to the next generation.
Another aspect to all of this that would add even more value would be to provide the larger units with various sensors that would store ANY and ALL possible forms of energy radiation and chemical exposure to the environment. This would assure that the units would not only contain the originally stored data, but would be constantly gathering the data in a parallel fashion in every corner of the world where the units are deployed.
As you can see, this would ensure after several generations, that all the original data is in tact and could simply be retrieved by reading all units chemical stores simultaneously and reassembling the original data as well as newly stored information. Imagine that... a sensor array that spans the planet with historical functions as well. And all self-sustaining and chemically based.
Parent
Re: (Score:3, Insightful)
You're assuming the source is analog... what about material that is no different in digital then in analog... if I write a book, or an application, what if the source is a picture, video or audio but one that was originally created o
Re:Multiple identical copies? (Score:5, Funny)
I'm confused.
Parent
Stone tablets (Score:5, Funny)
Now we just need a large enough area to store them
Re: (Score:3, Insightful)
data type is more important than medium (Score:3, Interesting)
This is a dual problem:
1) Digital data needs to be moved about once every 5 years onto a new physical store, disk, whatever. Think of the amount of data sitting around on floppy disks that is being lost as we speak.
2) Data has to be recorded in a way that that presumes whatever software you use to create it will not exist in the future. Anyone who saved their life's work in some ancient binary word processor file will know what I mean. For most computer-based data storage that requires data be stored s
Re:Stone tablets (Score:5, Funny)
That's out of the original 15.
Parent
oh, just (Score:3, Interesting)
3.5" (Score:5, Funny)
We can take this seriously. (Score:4, Informative)
Emphasizing the “I” in RAID [8k.com].
Parent
It's the messanger, not the message (Score:4, Insightful)
Re: (Score:3, Insightful)
Those audio tapes were "recoverable", but I bet they didn't sound all that great. Good enough to be understood, but nowhere near the original quality. An analog signal that is "garbled" is still usable.
If there had been *digital* data on those tapes, then it's pretty likely that enough of the data had been corrupted that the files would have been *unusable*. Once the bits are gone, they're gone. Throw in the fact that there no guaranteed that the encoding and file for
Re: (Score:3, Informative)
It depends on the content as well. Content that is inherently analog tends to be more 'robust' in analog form. For instance, in the military they say [wired.com], "A computer with a bullet in it is just a paperweight. A map with a bullet in it is still a map."
But what you got off the tape... (Score:5, Insightful)
First (Score:3, Interesting)
Then, we can figure out the most cost-effective medium to record stuff on, with determined re-archival cycles.
Crush and Preserve! (Score:4, Funny)
wring recovery method (Score:5, Informative)
Just because it's harder to recover the data doesn't mean it's impossible.
Of course, anyone using CDs or DVDs for large data backup must have a lot of interns to do the disc swapping.
Re:wring recovery method (Score:5, Informative)
To recover data from a CD, you can simply photograph it at high enough resolution. Even with huge scratches, even with parts of the disc physically missing, you can recover the data exactly as it was encoded. How? Reed Solomon code [wikipedia.org] .
Quoth wikipedia:
Parent
They could try harder (Score:4, Interesting)
Some analog technologies, like old color films, have also degraded and need image enhancement to recover the original content.
Re: (Score:3, Informative)
have people already forgotten? (Score:5, Informative)
1% = Total Loss? (Score:4, Interesting)
Of course for the ultimate in durable electronically readable storage you should be burning everything to PROMs [wikipedia.org].
Re:1% = Total Loss? (Score:4, Informative)
Parent
Apples and Oranges (Score:3, Informative)
Sad to say, tape dies too.
What is more interesting is the use of compression (and rights management, though if your originals are encrypted you deserve to get screwed - physical security comes first). With analog and simple stream encoding of time domain data (such as audio recordings) much data can be recovered using an external benchmark for the time code. Compress that data and lose your parity and you're totally hosed.
I've never been a proponent of compressed or encoded backups. Sure they save space and add a layer of "security", but that comes at the cost of flexibility should damage occur.
Of course, as has certainly already been mentioned - with digital data, you have the luxury of making multiple perfect copies as well as the ability to perform automated checks of that data, mostly possible without user interaction necessary.
Othwise, stone tablets have the best track record so far, though the storage density is a bit on the light (or should I say heavy?) side.
I've said it before and I'll say it again... (Score:4, Funny)
Re: (Score:3, Funny)
Re: (Score:3, Funny)
That service is already available [magrathea.px]. However, only the ultra-rich can afford it, and what with the whole galaxy in a bit of a rec
Remember the "Domesday Book" (Score:4, Interesting)
Umm.. (Score:5, Insightful)
Mission-critical archives and backups (Score:3, Insightful)
A combination of multiple sets of magneto-optical and tape backups maintained in separate locations, all temperature and humidity-controlled environments should easily yield 25~30 years shelf life, which guarantees that by then we'll hopefully have found better long-term options to transfer these to.
I am transferring most of my 15 to 20-year old audio DAT tapes digitally with no problems. Good brand-name CD-R's (like Tayo-Yuden) kept out of the light and at a steady temperature seem fairly resilient so far, but there has been batches which over time have developed 'rot' or layer oxydation, which sometimes renders them partially or wholly unusable.
DLT tapes are so far the most trouble-free type of media I have encountered, but with only 10 years to go back on, not sure that is accurate.
Z.
Sorry to spoil the fun (VXA tape format) (Score:3, Informative)
I know I'm offtopic, injecting facts into this debate, but I thought it might be interesting to bring up the VXA tape format. It allegedly survives all kinds of abuse like freezing, see Freezing Test [exabyte.com]
I have never tried these drives, and would love to hear from someone independent who has.
So You've Lost a $38 Billion File (Score:3, Insightful)
Chappies in New Brunswick:
From an earlier /. article:
Quick someone tell the author of: 'So You've Lost a $38 Billion File [slashdot.org]' that everything is alright! New Brunswick had data that was submerged in water, tape so swollen it was off the reel; they still managed to recover it.
And don't come out with that: 'Polar Bear ate the backup tape' excuse again!