What Do You Do When the Cloud Shuts Down?

jbrodkin writes "Can you trust your data to the cloud? For users of an online storage service called The Linkup, formerly known as MediaMax, the answer turned out to be a resounding 'no.' The Linkup shut down on Aug. 8 after losing access to as much as 45% of its customers' data. 'When we looked at some individual accounts, some people didn't have any files, and some people had all their files,' The Linkup CeO Steve Iverson admits. None of the affected users will get their lost data back. Iverson called it a 'worst-case scenario.'"

Backups, backups, backups!

Like anything else, including local technology, the key is to create a backup strategy. The cloud creates special problems for performing and managing backukps, so you need to understand your chosen compute or storage cluster provider's options, as well as other options specific for your application in regards to backups.

Re:Backups, backups, backups!

Redundancy, redundancy, redundancy. One backup to rule them all is not going to work. And for mission critical files would have to be backed up several times.

Re:Backups, backups, backups!

No kidding. Why do you think I said 'backups' three times in the subject line? ;) That's what I mean by a 'backup strategy' -- backup strategies, which are sometimes called 'disaster recovery plans', though that's really a bigger plan that includes a backup strategy, include making multiple redundant backups, offsite storage of backups, considerations for multiple different media, etc. There are several 'best practices', but the best strategy is going to be different for each company or department and often even for each application.

The best thing to do is to examine what kind(s) of data there is in the set, how large that data set is, how often that data gets updated, how often it needs to be accessed, and what are the potential costs for losing a day's, week's, month's, year's etc. worth of that data. That will point you in the direction as to frequency of backups, types of backups, etc.

Offsite backups are essential for any data requiring backup.

Re:Backups, backups, backups!

"No kidding. Why do you think I said 'backups' three times in the subject line?"

I think he was just backing up your statement!

Re:Backups, backups, backups!

"No matter how much we preach to the choir, it seems that most managers simply don't want to pay for it."

Fixed that for you.

Re:Backups, backups, backups!

There, fixed that for you. Backups aren't worth a damn if the building is blown up.

Yeah, I already thought of that. *smug* I have a script that backs up all my files from our servers in WTC1 to our servers in WTC2. What are the odds we could lose both sets of servers?

Re:Backups, backups, backups!

I used to back up my data in the mustache-and-goatee universe, but it kept coming back with the Evil bit set.

Re:Backups, backups, backups!

Don't forget to have a RESTORE strategy in place, too, and one that can be executed by others. Redundant backups don't do any good if you don't know how to restore from them, and know approximately how long it will take to restore.

We set up a test system identical to a few of our servers and had non-IT people execute the restoration plan for the core applications/data our business needs. There were a few flaws in the plan but it was a GREAT learning tool.

Re:Backups, backups, backups!

This is why I always thought that cloud computing based on servers would be disastrous. What if the server goes down? Well, here's a case in point. You lose everything.

I proposed an idea [halfbakery.com] like a P2P backup. Say you have some 20 GB you want to back up. You make 20 GB available on your system, and fire up a P2P backup program. You partner with people who want to backup also, trade backup space, and voila! You have a distributed backup system. It's all encrypted, so you can't get into other people's stuff on your system, and vice-versa. Periodically, the app checks to make sure that all your backup partners are available. If not, it starts negotiating a backup with a new partner.

Of course, you don't want to lose your stuff to a single host going down, you would have a ratio of 3:1 or 4:1 to make sure that you have high availability.

Re:Backups, backups, backups!

That idea is already quite prevalent and many programs such as BuddyBackup [databarracks.com] already use this idea.

Test restore, test restore, test restore!

I don't have any way of verifying this story, but I worked with an old guy once who told me that he had been at a startup in the UK that was, by the sound of it, creating a kind of IMDB in about 1994. They had a team of researchers and a bunch of seed capital to create a large film database. Everything was ticking along for about 18 months and they had researched thousands of films.

Then one day, the database shut down and they traced it to some bad hardware. They replaced the hardware and restored the database from the previous night's backup. Nothing doing - the backup tape (he said it was DAT) was corrupt. So they tried the other one. Nada. Same corruption. So they tried the off-site one. Same thing. Turned out all the backups they had made seem to have transferred the same corruption resulting in nothing significant recoverable.

Had they tried a test restore at some point, they might have found out. As it was, a week after the crash, they shut the business down.

Which reminds me of another (maybe apocryphal) story: the head of IT as a large company was fond of organising disaster recovery practices by walking into the data centre, physically removing a (pre-ordained) server and leaving a note in its place with the words "The server crashed" written on it. The support staff (and presumably management) knew that this would happen, but not when, or which machine (or dependent services) would be affected. Interesting test I would say.

Not a new problem!

What do you do when your local computer shuts down? How about a server on your company intranet? The cloud is no different. Backups are your friend!

Re:Not a new problem!

Kinda funny when you think about it, the backups are stored locally and the working copies are stored far away.

Backup, Storage

I can't believe this article. The number of places you store your data is directly related to the level of which it's important to you. People put all their data in once place then cry when it's gone? How is this new?

Isn't this akin to dumping all you money into one stock then whining when it tanks?

Re:Backup, Storage

Isn't this akin to dumping all you money into one stock then whining when it tanks?

Sure, but that doesn't stop people from doing exactly that.

What Do You Do When the Cloud Shuts Down?

Open the curtains and let the sunshine in, and water the garden.

Oh, you mean the network... what kind of fool trusts his data with someone else?

Re:

A fool and his data are soon to be parted.

Re:

Oh, you mean the network... what kind of fool trusts his data with someone else?

I build my own hard drives and power supply too!

Re:What Do You Do When the Cloud Shuts Down?

You don't generate your own electricity, too? Slacker!

Re:What Do You Do When the Cloud Shuts Down?

Insurance means nothing. Once your data is lost, it is lost. Whether or not you get money out of them in compensation for the lost data is almost non-important. I would say that anything you lost would be completely non-producable, even if you had all the money in the world. A picture of your family on vacation, can't be reproduced. You can go on another vacation, but it won't be the same vacation. Any document you have typed out, could be typed out again, but it would be different each time. Unless you are talking about lost music files, in which case, you could download them again, but that's kind of the same as having a backup. Any data that's really important isn't going to reproducible.

The critical flaw

The critical flaw of cloud computing is that you entrust your data to a third party. If you are at all concerned with privacy you will think cloud computing is a terrible idea.

OK, so I'm a dinosaur, but...

...they can have my local computing and storage capabilities when they pry them from my cold, dead hands. Google is great for looking things up -- and webmail accounts are great for portability -- but the old saying applies: If you want something done (backed up, available), do it yourself. Much more secure that way, too.

Besides, with Remote Desktop, FOUSs*, and continuous 'Net connections, it's pretty easy to take it with you.


* (8GB on a microSDHC the size of my fingernail is a Flashdrive Of Unusual Size in my book!)

Same happened to me.

I had this overly insightful comment... but it all got lost when I submitted it.

And now look what I'm left with!

Link to the article on ONE PAGE

This version [networkworld.com] may be easier to read.

Bottom line: The Linkup is blaming Nirvanix (a third-party service provider) which is, of course, blaming The Linkup. FTA:

Nirvanix says it has not deleted any customer data, and promises that its Storage Delivery Network is immune to the problem that plagued The Linkup. At The Linkup, a "system administrator ran a script that misidentified active account data and disassociated physical files from their owners," Nirvanix says. "This led to files being marked offline in the old Streamload/MediaMax file system when they shouldn't have been." Iverson, meanwhile, claims it was a Nirvanix engineer who caused the data loss.

Summary: "He did it." "No, he did it." "No, it was him!" "You did it FIRST!" "Idiot!" "Moron!" "Jackass!" ** customers shoot them both **

Re:Well DUH!

I back up everything to /dev/null. Restoration is somebody else's job.