Most Digital Content Not Stable 353
brunes69 writes "The CBC is running an article profiling the problems with archiving digital data in New Brunswick's provincial archives. Quote from the story: 'I've had audio tape come into the archives, for example, that had been submerged in water in floods and the tape was so swollen it went off the reel, and yet we were able to recover that. We were able to take that off and dry it out and play it back. If a CD had one-tenth of one per cent of the damage on one of those reels, it wouldn't play, period. The whole thing would be corrupted'. Given the difficulties with preserving digital data, is it really the medium we should be using for archival purposes?"
That's nothing, think of DRM (Score:5, Insightful)
Re:That's nothing, think of DRM (Score:5, Funny)
Re: (Score:3, Funny)
That might mean something if DRM magically retroactively destroyed all non-DRM copies of the content it contains. Like, say, the original.
Ten years ago my VCR ate my copy of Citizen Kane, which might have been a cultural tragedy, but fortunately someone had the foresight to give me a copy on VHS instead of the original print.
Re: (Score:2)
Re: (Score:3, Informative)
I think that was the point behind "depending on how you define American" -- the GP was referring to the urbanized cultures of Mexico, Central and South America that had writing systems that they were forced to give up along with the rest of their culture.
Re:That's nothing, think of DRM (Score:4, Insightful)
Ballsy words for an Anonymous Coward. Hopefully you'd stick to them if your hometown were invaded.
Re: (Score:3, Insightful)
Um. Slashdot itself refers to you as an 'Anonymous Coward', ostensibly in an effort to incite account registration while allowing anonymous posting.
And I'll admit myself to having a bit of a bias against ACs. Sometimes they're insightful, but most of the time when you see 'Anonymous Coward' in the byline, you can guess you're going to see something stupid or trollish.
So, yeah. Statistically, 'Anonymous Coward'
Re: (Score:3, Informative)
Best summed up by Chief Seattle, in 1854: "This we know: the earth does not belong to man, man belongs to the earth. All things are connected like the blood that unites us all. Man did not weave the web of life, he is merely a strand in it. Whatever he does to the web, he does to himself."
Re: (Score:3, Interesting)
You are indeed correct that "there are many methods of storage that could be easily overlooked and mistaken for simple art."
Re: (Score:3, Insightful)
Well, I'm 1/8th Native American (but 7/8ths White) if that counts for anything, but this is always overblown. Whites/europeans came in and conquered the land. That's what people have done throughout all of recorded history. The Romans Conquered the Greeks, the Normans conquered the Saxons, etc. The list goes on and on. The case has ALWAYS been that if some other nation wanted your land and you couldn't stand up to them in a mil
Re: (Score:2)
1: Indians are still around in recognisable form, their culture is recovering.
2: You didn't wipe them out completelly.
3: It didn't happen that long ago
4: You made all those dumb 'indians are bad guys' films that actually end up revealing the conptempt the white european settlers had for the native population.
Probably point 4 would be irrelevent if point 3 wasn't there.
The English did a far worse job on the Aborigines of Australia. We have the advantage
how did we get so far offtopic? (Score:3, Insightful)
That being said. What's done is done. It should be remembered so we learn from those horrible mistakes. It shouldn't be a constant source of guilt to be used against people that had no part in it. The same goes for slavery, genocid
Re:That's nothing, think of DRM (Score:5, Insightful)
As a person who loves to study European antiquity I would point out some flaws in this thinking...
1. When the Romans conquered the Greeks they actually adopted Greek culture and didn't kill off the Greeks.
2. When the Normans conquered the Saxons they didn't kill off the Saxons nor really conquered their land as much as just intermarried with them (Hence Anglo-Saxon Culture)
The only whole sale Genocides that history can come up with is the Crusaders massacre of Jerusalem (which wasn't really as much as hatred of Muslims as it was starving Europeans killing off everyone in the city regardless of religion out of rage of having to starve in the desert for several months) and then the Mongol sack of Baghdad which wasn't over so much as land, but out of spite of the execution of Mongol diplomats (considering they burned and salted the lands made the "take your lands" point of conquering sort of a non-issue).
The genocide and seizure of lands in this scale was never really seen before until the colonization of Americas. It wasn't as much as the Indians could not defend them as much as it was that the westerners thought they were subhuman.
Which sadly we saw again in the European theatre in WW2.
Roman & Greeks != European & Native Americ (Score:3, Insightful)
Re: (Score:3, Insightful)
That's a pretty fine hair to split between genocide and ethnic cleansing. What is the real difference between successful ethnic cleansing and unsuccessful genocide?
I do believe I have friends that have some native american ancestry...
All this means is that the genocide was not complete.
The Nazis attempted a genocide against the Jews but did not complete the job. If they had started out simply with a mission of ethnic cleansing and achiev
Re: (Score:3, Insightful)
Multiple identical copies? (Score:5, Insightful)
Re:Multiple identical copies? (Score:5, Informative)
Like what? (Score:2)
Re: (Score:2, Informative)
DLT
reel-to-reel
Mini8mm
SAN
CD/DVD
etc...
Depends on how deep your pockets go and your calculation for the value of the data if lost. You are doing the math on loss of data, riggghhhhttt?
Re: (Score:3, Funny)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Insightful)
You're assuming the source is analog... what about material that is no different in digital then in analog... if I write a book, or an application, what if the source is a picture, video or audio but one that was originally created o
Re: (Score:3, Insightful)
Re:Multiple identical copies? (Score:4, Funny)
The larger systems could also provide mirroring by interfacing with each other as directed by chemical interactions in order to preserve original data as well as integrate new data that may be useful in assuring that future units are even more resilient to any sorts of flaws or possible malfunction caused by inappropriate chemical input. The key to all of this is going to be to make sure that the larger units are impelled to continue the duplication and exchange of data ad infinitum. To do that, there should be some sort of mutual benefit that the engaged units acquire from the mirroring. Multiple levels of mutual benefit would likely be more successful than just one level. So I propose that at a base level, the units should be programmed with routines that make them feel more or less successful whenever a mirroring connection is attempted. I know that sounds strange, but it should be a pretty simple subroutine and will at least get the units to attempt mirroring.
The next level would also be an expansion of the data mirroring to the actual manufacture of a tertiary (or even more) unit that contains selected data from both origination units. As part of the mutual benefit relationship between units, the origination units should be programmed to protect the manufactured unit in order to safeguard its data as it would be the freshest copy (chemically speaking) and therefore more viable. So the relationship between origination units and next generation manufactured units would be that of security and stability from the origination units as applied to the next generation.
Another aspect to all of this that would add even more value would be to provide the larger units with various sensors that would store ANY and ALL possible forms of energy radiation and chemical exposure to the environment. This would assure that the units would not only contain the originally stored data, but would be constantly gathering the data in a parallel fashion in every corner of the world where the units are deployed.
As you can see, this would ensure after several generations, that all the original data is in tact and could simply be retrieved by reading all units chemical stores simultaneously and reassembling the original data as well as newly stored information. Imagine that... a sensor array that spans the planet with historical functions as well. And all self-sustaining and chemically based.
Every Superman has his Kryptonite (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Precisely (Score:3, Insightful)
As you said, the great thing about digital data is that is can be replaced cheaply, perfectly, and spread around. It's resilience isn't in the one copy lasting 1000 years, it is in having copies everywhere, so no even short of nuclear war can eliminate them all, and maybe not even then.
This also is th
Re: (Score:2)
It's a stupid fucking argument anyway. I think I am not alone when I say that, for example, I have a C= 1541 lying around. That's an old-ass format, but there's probably tens of thousands of them sprinkled around through geeks' bedrooms alone.
Hobbyists alone are sufficient to maintain the means to read old data formats, and som
Re: (Score:2)
I lost a lot of my childhood stuff (since stuff like Splash doesn't even have a reader for the format anymore), but if I cared enough I would have deliberately saved it. Just like I could save my children's "fridge art".
Archival quality data is easily saved if you care. (DRM or not)
Re: (Score:2, Interesting)
It's already happened/happening. (Score:5, Insightful)
I think you're missing an important element here. As you move along in time, the volume of data that must be converted to the format du jour only gets bigger and bigger.
For a single person, it's probably not too bad. I, too, have pretty much everything I ever wrote since I first got a computer, and every few years I've committed to rolling the whole thing onto new media. So I've gone from offline backups on floppies, to Zip disks (in retrospect a mistake), to CDs, to DVD-R, and now to DVD+R (the -R discs were crappy and I've since heard that +R is a superior format anyway). This isn't much trouble, because the amount of data I have to backup hasn't really grown that much faster than the data density of available media. I'm probably up to a couple of DVDs for the stuff I really, really care about, maybe a binder if I include all the photos and video.
But what's a basic Saturday-afternoon copy-and-burn job for an individual is a Sisyphean task for a large government agency or library, particularly one who is constantly generating new content. I've seen places that could barely keep up with archiving the stuff they were producing, much less roll their vast archives forward onto new media. So they'd have vaults of hard drives, sitting next to DLT cassettes, next to IBM 3480, next to racks of old half-inch open-reel tapes. Probably back in some dark corner there were piles of punched cards; it really wouldn't surprise me. The problem of data loss due to unreadable formats isn't some abstract 'maybe,' it's already happened in a lot of places (but nobody really wants to talk about it, so it mostly gets buried and whatever's on the tapes gets written off).
The reason why there's so much interest in preservable formats is because while it may not be strictly impossible to constantly roll old backups and archives forward, it's very hard, and requires vast amounts of effort and expense. If you have a backup that's being written into a format that you know is going to be readable for a long time, even if it's more expensive to write initially, you can save a lot of money and time down the road by not having to copy it forward as often.
People may get a little shrill when they're talking about these issues, but they're quite real.
Re: (Score:3, Interesting)
There was life before CDs. (Score:3, Informative)
Erm
There are terabytes (quite literally tons) of data sitting around on everything from old 7- and 9-track 1/2" open reel tape, to old 8" and 5-1/4" floppies, and other formats that are basically dead. [I'm not familiar with anything older than that, but I'm sure there are some real greybeards around that could enlighten you as to what cam
Re: (Score:2)
I've got maybe 400 DVD's full of stuff here. If you think I'm copy that lot to Blu-ray or whatever, you've got another thing coming. I got to a point a couple of years ago where I decided I'd only stick stuff on DVD by making 2 copies on 2 different brands & keeping them in a dark cupboard. I also only bothered backing up stuff that I wouldn't be *that* upset if I lost. The really important stuff is mirrored over 3 hard drives so if one dies, I have 2 copies to create
Re:Multiple identical copies? (Score:5, Insightful)
Also I'm certain for every analog horror story there is a digital lucky story (and vice versa). Not to mention digital encodings usually have some kind of redundancy. A small scrach does nothing but the same scratch on an lp forever destroys some part of the track. I wont even go into the magic of data restoration (which the author ignores). There's really no 'tough medium for the ages' out there that can do it all. Just complaints and blind-luck stories.
LOCKSS (was Re:Multiple identical copies?) (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Funny)
What I do is take files I care about, encrypt them, rename the file to something tempting like "Cheerleader Sex Orgy XXXIV.avi," note the MD5 (sticky note on the next of the monitor), and share it on a P2P network.
Instant distributed backup! 8D
Re: (Score:3, Funny)
Article title incorrect (Score:3, Interesting)
Digital content isn't unstable, it's just more sensitive to corruption because in general software expects to be able to extract a perfect copy every time, rather than a near-perfect copy. Whether you can recover partially corrupted digital data depends on several things:
A) Choice of filesystem (journaling, error correction, built-in redundancy)
B) Choi
Re:Multiple identical copies? (Score:5, Funny)
I'm confused.
Stone tablets (Score:5, Funny)
Now we just need a large enough area to store them
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:Stone tablets (Score:5, Funny)
That's out of the original 15.
Re: (Score:3, Insightful)
Re: (Score:2)
data type is more important than medium (Score:3, Interesting)
This is a dual problem:
1) Digital data needs to be moved about once every 5 years onto a new physical store, disk, whatever. Think of the amount of data sitting around on floppy disks that is being lost as we speak.
2) Data has to be recorded in a way that that presumes whatever software you use to create it will not exist in the future. Anyone who saved their life's work in some ancient binary word processor file will know what I mean. For most computer-based data storage that requires data be stored s
oh, just (Score:3, Interesting)
Contents isn't supposed to be stable (Score:2)
The real problem is more that the media is not stable. Optical disks are certainly not a long term archival strategy.
I wonder if there's a good way to convert digital video into black and white film (maybe with one frame per color channel) since it's got a proven archival record.
Re: (Score:2)
As long as you're not using nitrate based (celluloid) film stock -- a lot of pre-1950s footage has disintegrated. For that matter even acetate-based stocks tend to deteriorate over time.
There are other film types -- and the stuff they use (used to use?) for microfiche is rated in centuries if kept in climate-controlled conditions. Under hot and
3.5" (Score:5, Funny)
We can take this seriously. (Score:4, Informative)
Emphasizing the “I” in RAID [8k.com].
Re: (Score:2)
It's the messanger, not the message (Score:4, Insightful)
Re: (Score:3, Insightful)
Those audio tapes were "recoverable", but I bet they didn't sound all that great. Good enough to be understood, but nowhere near the original quality. An analog signal that is "garbled" is still usable.
If there had been *digital* data on those tapes, then it's pretty likely that enough of the data had been corrupted that the files would have been *unusable*. Once the bits are gone, they're gone. Throw in the fact that there no guaranteed that the encoding and file for
Re: (Score:3, Informative)
It depends on the content as well. Content that is inherently analog tends to be more 'robust' in analog form. For instance, in the military they say [wired.com], "A computer with a bullet in it is just a paperweight. A map with a bullet in it is still a map."
But what you got off the tape... (Score:5, Insightful)
Exactly (Score:2)
First (Score:3, Interesting)
Then, we can figure out the most cost-effective medium to record stuff on, with determined re-archival cycles.
Re: (Score:2)
Tell that to the Egyptians, the Romans, or any other ancient civilization. Many of their creations still exist today. Can the same be said for ours?
Crush and Preserve! (Score:4, Funny)
wring recovery method (Score:5, Informative)
Just because it's harder to recover the data doesn't mean it's impossible.
Of course, anyone using CDs or DVDs for large data backup must have a lot of interns to do the disc swapping.
Re:wring recovery method (Score:5, Informative)
To recover data from a CD, you can simply photograph it at high enough resolution. Even with huge scratches, even with parts of the disc physically missing, you can recover the data exactly as it was encoded. How? Reed Solomon code [wikipedia.org] .
Quoth wikipedia:
Doesn't help degradation (Score:2)
But it can not compensate well at all for even medium amounts of random bit errors. These are the exact kinds of errors that occur on CD and DVD media over time as it degrades. That is what is being referred to here.
If you have a piece of analogue data, and it degrades, you can still get enough meaning from the original to make it worth archiving. A piece of digital data with even a relatively small
They could try harder (Score:4, Interesting)
Some analog technologies, like old color films, have also degraded and need image enhancement to recover the original content.
Re: (Score:3, Informative)
have people already forgotten? (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
Just out of curiousity, was your first clue when he said "If you have an analog tape"?
Re: (Score:2)
Re: (Score:2)
1% = Total Loss? (Score:4, Interesting)
Of course for the ultimate in durable electronically readable storage you should be burning everything to PROMs [wikipedia.org].
Re: (Score:2)
Re:1% = Total Loss? (Score:4, Informative)
Apples and Oranges (Score:3, Informative)
Sad to say, tape dies too.
What is more interesting is the use of compression (and rights management, though if your originals are encrypted you deserve to get screwed - physical security comes first). With analog and simple stream encoding of time domain data (such as audio recordings) much data can be recovered using an external benchmark for the time code. Compress that data and lose your parity and you're totally hosed.
I've never been a proponent of compressed or encoded backups. Sure they save space and add a layer of "security", but that comes at the cost of flexibility should damage occur.
Of course, as has certainly already been mentioned - with digital data, you have the luxury of making multiple perfect copies as well as the ability to perform automated checks of that data, mostly possible without user interaction necessary.
Othwise, stone tablets have the best track record so far, though the storage density is a bit on the light (or should I say heavy?) side.
Re: (Score:2)
BEar in mind, we are not talking about cassette here.
I've said it before and I'll say it again... (Score:4, Funny)
Re: (Score:3, Funny)
Re: (Score:3, Funny)
That service is already available [magrathea.px]. However, only the ultra-rich can afford it, and what with the whole galaxy in a bit of a rec
Remember the "Domesday Book" (Score:4, Interesting)
Re: (Score:2)
Umm.. (Score:5, Insightful)
Mission-critical archives and backups (Score:3, Insightful)
A combination of multiple sets of magneto-optical and tape backups maintained in separate locations, all temperature and humidity-controlled environments should easily yield 25~30 years shelf life, which guarantees that by then we'll hopefully have found better long-term options to transfer these to.
I am transferring most of my 15 to 20-year old audio DAT tapes digitally with no problems. Good brand-name CD-R's (like Tayo-Yuden) kept out of the light and at a steady temperature seem fairly resilient so far, but there has been batches which over time have developed 'rot' or layer oxydation, which sometimes renders them partially or wholly unusable.
DLT tapes are so far the most trouble-free type of media I have encountered, but with only 10 years to go back on, not sure that is accurate.
Z.
Here's how to make your digital backups.... (Score:2)
1) Make a digital copy
2) Repeat step 1 until your digital copy takes up as much physical space as an analog copy would
3) For no reason, lay out all your digital copies in such a way that the whole of them create an analog copy
4) For fun, Pretend what you've done is "holographic storage"
Sorry to spoil the fun (VXA tape format) (Score:3, Informative)
I know I'm offtopic, injecting facts into this debate, but I thought it might be interesting to bring up the VXA tape format. It allegedly survives all kinds of abuse like freezing, see Freezing Test [exabyte.com]
I have never tried these drives, and would love to hear from someone independent who has.
So You've Lost a $38 Billion File (Score:3, Insightful)
Chappies in New Brunswick:
From an earlier /. article:
Quick someone tell the author of: 'So You've Lost a $38 Billion File [slashdot.org]' that everything is alright! New Brunswick had data that was submerged in water, tape so swollen it was off the reel; they still managed to recover it.
And don't come out with that: 'Polar Bear ate the backup tape' excuse again!
Incompetent archivist alert. (Score:2)
Sometimes I've not been able to recover disks that have been damaged beyond a certain point. But I've never lost a CD because it got wet, or had one become unplayable because it warped. I keep backup tapes in a water-resistant container (
The obvious solution (Score:2)
Error-correcting codecs (Score:2)
For instance Reed-Solomon codes or Tornado codes can be used to break data up so that you can use a subset of the pieces to reconstruct the original signal. After chunking things up into small enough pieces that these codes are practical to apply, you can scatter the chunks across the disk or across multiple disks. This general sort of thing a
Tapes are vulnerable too (Score:2)
Now I'm not going to suggest that a box of DVD's bought at Walmart in a 50/$25 pack is a good replacement for tapes, but in cases with proper handling and storage, optical-based
Not the same thing... (Score:2)
Who backs up all their data on CD/DVDs? I don't know of any enterprise who puts their long term backups on CD/DVDs. Everyone still uses tape. It is just in digital format vs. analog format.
And like other posters have pointed out, there are more serious concerns such as DRM and equipment resources.
And again, like other posters have pointed out, you can make perfect digital copies. You cannot do that with analog.
But do we really WANT permanency on everything? (Score:2)
Is this really that bad a thing? Do we really want everything preserved forever? Obviously, there are important exceptions (historical records, artwork, family "heirloom"-type information, etc. but do we really want all the day-to-day minutia to be so d
TV DVD recorders (Score:3)
Our one, and so far only, experience with our DVD recorder (the TV/Video kind) illustrates why we haven't gotten rid of our VHS tapes yet.
Least steps to record onto a new VHS:
1) pop tape in
2) press record
Least steps to record onto new DVD (-RW in our case):
1) pop DVD in
2) wait 10 seconds before format options come up
3) wait 1 min for format to finish
4) select recording option (quality setting, etc)
5) begin recording
At the end of an hour-long show, I finally hit "stop" on the DVD recorder. In earlier, shorter tests it took about 30 seconds to write out the information for that hour. This time, it failed for some reason.
End result: the whole hour of recording was lost.
All the other nice features that would've come with recording to DVD were flushed right down the drain, for the simple reason the damn thing can't even guarantee that what I recorded would, in the end, actually be available to play back!
Re: (Score:2)