Slashdot Log In
Ma.gnolia User Data Is Gone For Good
Posted by
kdawson
on Fri Feb 20, 2009 11:48 AM
from the lost-in-the-clouds dept.
from the lost-in-the-clouds dept.
miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."
Related Stories
[+]
Hardware: The Sidekick Failure and Cloud Culpability 246 comments
miller60 writes "There's a vigorous debate among cloud pundits about whether the apparent loss of all Sidekick users' data is a reflection on the trustworthiness of cloud computing or simply another cautionary tale about poor backup practices. InformationWeek calls the incident 'a code red cloud disaster.' But some cloud technologists insist data center failures are not cloud failures. Is this distinction meaningful? Or does the cloud movement bear the burden of fuzzy definitions in assessing its shortcomings as well as its promise?"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Mac reliability (Score:5, Funny)
Re:Mac reliability (Score:4, Insightful)
I had no idea anyone actually used Mac's as servers. Sure, I bet you can get apache running or something but I didn't realize anyone had. Therefore, this is my first bit of exposure to this idea of Macs as servers and its all negative!
Woe is me.
Parent
Re:Mac reliability (Score:5, Informative)
Parent
Re:Mac reliability (Score:5, Insightful)
Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.
Seriously, do people still not realise that OS X is just UNIX with a pretty UI?
Parent
Re:Mac reliability (Score:5, Funny)
Parent
Re: (Score:3, Informative)
No, but Mac OS 10.5.x can properly be called Unix [opengroup.org], but only the Intel version, not the PPC version.
Re:Mac reliability (Score:5, Informative)
Yeah, but that's exactly the surprising part. Why would you pay Apple $3000 for a xserve running Apache and MySQL, with a crappy service contract (no next-day service, no on-site service-- I've looked into it), when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL, and get a next-day and on-site service contract?
Anyone who buys an xserve is an idiot.
Parent
!equivalent (Score:4, Informative)
Parent
Re:Mac reliability (Score:5, Informative)
a crappy service contract (no next-day service, no on-site service-- I've looked into it)
Not very hard, apparently.
http://www.apple.com/server/support/ [apple.com]
You get 24/7 telephone and email support with 30-minute response. For hardware repairs, Apple-certified technicians provide onsite response within four hours during business hours and next-day onsite response when you contact Apple after business hours.
Parent
Re:Mac reliability (Score:5, Informative)
Fine; what company do you trust? HP? IBM? Replace "Dell" with them, and my example still applies. The fact is, *every* server vendor can do better than Apple. Even IBM does better, and they suck.
Oh, and BTW, all servers will have hardware problems from time-to-time. When that happens with your Dell, HP, IBM server, the guy is there in his truck in 4 hours. When that happens to your Apple server, you're SOL.
Parent
Re:Mac reliability (Score:5, Informative)
Even IBM does better, and they suck.
One morning I came in and was looking at the logs. SMART was reporting that one of the disks in one of the servers was going to go bad soon. Not 15 minutes after i even noticed this in the logs, an IBM tech was there with a fresh one ready to replace it.
How? The server called home, told IBM about the error, and they disbatched a tech immediately.
If that "sucks", your service must come with free hookers or something.
Parent
Re:Mac reliability (Score:5, Informative)
Except its the same hardware...well no, that's not true. You can get a Dell with actual hardware RAID when you're stuck with software RAID on an Xserve.
Furthermore Dell also has a 4-hour onsite 24/7 support package if I'm not mistaken.
I love my MacBook and the OS X desktop experience but you simply can't use an Xserve on business critical operations.
Parent
Re:Mac reliability (Score:4, Insightful)
Wait, you mean apples makes all their own hardware? Really?
No, they use CPUs from Intel, hard drives from WD/Seagate/Maxtor/whoever, graphics chips from nVidea/ATI, etc.
So, no, you do NOT get what you pay for hardware wise. You are getting something identical or very similar to everybody else, from the same manufactures, you are just paying a little more.
Parent
Re:Mac reliability (Score:4, Insightful)
I think you are missing the importance of a disaster recovery plan, with backups, for any mission critical hardware, regardless of vendor. Why didn't Mag have any sort of backup plan that was tested? Clustered hardware does not equal a backup plan - thanks for trying there.
Was there in fact a schedule of backups of the operational system? This seems like a rubber band and duct tape operation to me.
Parent
Re:Mac reliability (Score:5, Interesting)
Are any of the free BSDs or Linux variants certified Unixes?
(Honest question, I don't know.)
Parent
Re:What is this "UNIX" you speak of... (Score:4, Interesting)
Apple is 1% hardware and 99% Marketing. Not too much they do can't be done on a Dell or HP. They just make it appear to do it better/slicker/faster, that's all.
I'd peg it at 10% hardware, if not more. The internal hardware layout of Apple's desktop towers borders on beautiful. Beats Dell and HP hands down.
And, while its hardware failures tend to be more spectacular, I've generally found Apple hardware to be more reliable than any of the Wintel vendors. (...speaking as someone who has been supporting computers since before MS-DOS or the Mac...)
Parent
Re: (Score:3, Interesting)
It's hard to know whether you're trolling or not.
There are OS X servers out there and they perform rather well. I know because I admin 50 of them, and have met hundreds of others who administer them in school systems across the state.
You may also be familiar with iTunes, or Apple's movie trailer website. I'm sure a large part of those are Xserves and Raids.
I'm not saying they are maintenance free, but they are out there.
Furthermore, a few years back there was a rather large beowulf cluster of mac towers tha
Re:Mac reliability (Score:5, Insightful)
Mac servers are pretty. They do okay, they have nice swanky data enclosures, and the form factor is roughly the same as anyone elses.
It's just whether or not you want to use OS X. I disagree that OS X is "just unix," however. It's not even "just linux" or "just bsd". OS X has it's own warts, and while it may be stable and friendly, I'd rather have a real *nix running on less pretty hardware.
The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine (If you say windows mac volume, I'm mailing a dead fish to your house).
Otherwise, you know, you can install apache, whatever, but it's not any different from using a regular linux server in terms of increased functionality, and there are some significant OS update issues that can cause problems. Mac updates are of the all or nothing school, and they WILL break stuff, so you need to be careful.
Parent
Re:Mac reliability (Score:4, Funny)
(If you say windows mac volume, I'm mailing a dead fish to your house).
Why, so it will attract the penguins?
Parent
Re:Mac reliability (Score:4, Funny)
and it comes with a free black turtleneck!
Parent
Re:Mac reliability (Score:4, Informative)
This is meaningless today. Most Unixlike systems today are not certified Unix systems.
OS X has some significant differences from traditional Unixlike systems and Linux - not necessarily disadvantages:
You can certainly add grid computing software to other operating systems. OS X is missing some functionality that a "regular linux server" may have. Even when considering third-party software, there are many things that can be done in Linux but not in OS X.
Mac OS X security updates certainly are "all or nothing" - you have to install all of the patches included in the package or install none of them. Each package includes many fixes, and sometimes they break things [slashdot.org]. The updates are not available as individual pkacages. You cannot select which updates are applied to the system.
RHEL/CentOS has point releases, but there are plenty of individual package updates in between (to fix bugs, compatibility, and security issues.) Individual package updates are released when they are ready, not as part of a large security update bundle or a monthly schedule.
Parent
Re:Mac reliability (Score:4, Informative)
It means you don't have to pay the performance penalty that netatalk has from resource fork handling since HFS+ is a native file system.
So if you're dealing with lots of small files with both forks, you're going to pay a penalty. What that says to me is that there are certain limited cases in which you might still need to use an Apple fileserver for performance, but in almost every real-world case where you actually have macs as clients, is it really an issue?
Parent
Re: (Score:3, Funny)
Re:Mac reliability (Score:4, Funny)
What you don't realise is that Microsoft hates Apple, so they must have been Psystar clones. Everyone knows that Psystar couldn't exist without help from a major corporation. And Microsoft is a major corporation. So that proves it: they weren't Mac Minis -- they were Psystar clones, recased inside gutted Mac Minis.
In fact, the whole reason Ma.gnolia's servers bombed (hey, Macs don't crash), is because of Psystar. They distracted Apple from creating beautiful products. So it's Microsoft's fault.
Obviously.
Parent
Re:Mac reliability (Score:5, Funny)
I want to say something here, but I get the feeling that no matter what I say, I just wont be herd.
Parent
Re:Mac reliability (Score:5, Funny)
Parent
In other news (Score:5, Funny)
Re:In other news (Score:5, Funny)
Parent
Re:In other news (Score:5, Funny)
Parent
Food for Stallman (Score:3, Insightful)
This bad news is delicious food for Stallman's argument against "cloud" services.
Re:Food for Stallman (Score:5, Insightful)
Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.
Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.
Parent
Re:Food for Fault-Tolerance (Score:3, Insightful)
It's food for any argument against any web service that doesn't publish it's reliability information or publicize the data for what types of mechanisms it has in place in case of disasters like a corrupt database, fried motherboard, or busted hard drive.
There's a design methodology that's used by NASA for manned missions: Any individual component should be able to fail without compromising the mission. Of course, in the last few decades we've seen 2 out of 5 Shuttles go ka-boom! so obviously this NASA gu
Needless loss (Score:5, Insightful)
Argh, why not just add a backup or replication database on one of the spare Mac Minis?
That way you would have needed a complete server farm disaster to mess things up irretrievably.
Re:Needless loss (Score:5, Insightful)
I definitely have my doubts that someone who could make this mistake is all that capable "lessons learned."
Parent
Re:Needless loss (Score:5, Insightful)
A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.
Parent
Re:Needless loss (Score:5, Insightful)
If you read the transcript, that's what they were doing, a simple firewire DB dump.
The problem is that they never tested the backups, and they didn't keep versioned backups. So they'd been backing-up the corrupted database for awhile before the site finally crashed for good. When it crashed, they only had the corrupted database backup. Additionally, the DB server was on RAID but of course the corrupted DB would just get saved to both HDs, so that's no good in a situation like this.
Basically, when the site crashed, he had three copies (2 RAID, 1 backup) of the data: all corrupt. The guy wasn't totally retarded when it came to backups, just 80% retarded.
Parent
Re:Needless loss (Score:5, Insightful)
Argh, why not just add a backup or replication database on one of the spare Mac Minis?
That way you would have needed a complete server farm disaster to mess things up irretrievably.
Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.
What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.
Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!
Parent
Who the hell is Ma.gnolia (Score:5, Insightful)
And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?
lesson #1 (Score:5, Interesting)
lesson #2, trust no-one with your data
lesson #3 disaster recovery capability only exists after it's been tested
lesson #4 backups are useless unless you can prove you can recover from them
Not the platform's fault... (Score:3, Interesting)
Good backup strategies are critical to any operation, regardless of platform. I've seen similar things happen with MSSQL server databases as well as Oracle running on the most powerful Sun box you can get (circa 2001).
One database backup strategy I've seen used rather successfully is doing a straight SQL dump every night and then copying the sql file over to somewhere else; even if the database became hopelessly corrupted there's still a way to re-import everything.
Of course, this is in *addition* to mirroring, tape backups, etc.
Lesson? (Score:5, Insightful)
discussed the crash and the lessons to be learned
Lessons such as "Regularly monitor and maintain backups like and business should?"
Re:Lesson? (Score:5, Interesting)
I love it when people say things like this. It shows me that they've never actually had to set up an enterprise backup strategy. I'm certainly not defending the Ma.gnolia guys, but I also can't stand it when people are on a shakier soapbox than they realize.
I'm sorry, but when you are used to the whiz-bang-pretty of Web2.0, the state of enterprise-level backups is horrifically archaic and dismal. And, btw, given the size of today's hard drives and databases, for pretty much all intents and purposes "Enterprise" == "More than one computer with more than just a few files on a drive".
Compare and contrast: a 1 TB hard drive will run you roughly $100. Do you know how much it then costs to backup that TB?
LTO-4 tapes, 800GB each, $50-$150 each tape plus roughly $2500 for the drive. Figure 2 tapes/day * 10 days backups = 20 tapes * $100 = $2000 in tapes alone. Congrats, that 1 TB just cost you $4500 in enterprise backups ... not to mention the time involved each day in doing a backup. You might save yourself some time and money by doing incrementals ... but then you have to balance that risk with complexity of backups and difficulty in restores.
NAS is trickier. The cheap NAS solutions, sub $1000 such as Buffalo and LaCie, aren't going to get you much more than a TB or two. And at that point, are you really any better off than the RAID solution? Maybe, maybe not. As you start to scale into IBM or Dell solutions, you are almost immediately beyond a $2500 price point before you even get to hard drives. Oh, and don't forget the cost of a gigabit switch so that it doesn't take you days to do a single backup.
iSCSI? Seriously? Not an option for SOHO businesses.
Then there's backup software to contend with. It's not just as simple as "go buy a copy of BackupExec" -- there's different licensing for databases, and network backups, and whatever arcane rules they want today. I'm a PC guy so I can't talk much about Enterprise-level Mac backup solutions, but I can without a doubt say that Time Machine is not one of them.
It's even more dismal when it comes to Open Source solutions. Have you ever actually tried to setup Bacula? It may be the 600lb gorilla of OS backup solutions, but it's still a royal pain. And to the "just set up a cron job for rsync" guys, c'mon, really? Good luck with that.
So, please, let's dispense with the thought that backups are easy. Backups really suck. Hard. That's why so many people want to think of RAID as a backup solution -- because the step from one hard drive to two or three is easy, but the next step is much farther away than you think.
Parent
Lessons Learned?? (Score:5, Insightful)
What's with the OS X users losing data? (Score:4, Insightful)
I mean, just because a few medium-profile sites running on Macs have experienced a failure causing data loss doesn't make them unique. Every OS and every type of hardware will, at some point, experience a failure. It's the PEOPLE that make the failure a problem, and it sure looks like this tard was a problem.
Who the hell doesn't back up their data? Seriously? This is "Slashdot worthy" because some hapless Mac user lost their data. BOO FUCKING FAIL. Move on.
"Private relaunch?" (Score:5, Interesting)
"Gee, Bob, we have the proof that this thing works. Why don't we sell it already?"
"Well, Bill, nobody wants to buy it and grandfather in all the whining freeloaders and their data."
"It's too bad we can't just drop all the data and start fresh."
"Well, why not, Bill? All we have to do is say it's been lost and can't be recovered. We can tell the buyer what's actually happening so they don't think we're total IT rejects who couldn't figure out a data retention policy."
"That's why I like working with you, Bob. You always have a way around the problem."
Have fun with it. The names have been changed (one changed anyway and one added), well, because it probably has nothing to do with reality. It sure is fun to ponder, though.
Transcript of interview (Score:5, Informative)
Rather than watch the video or download the 23MB MP3, you can read the full transcript here:
http://ratafia.info/post/78915439/transcript-and-commentary-for-whither-magnolia [ratafia.info]
I can read much faster than I can listen.
Finally! A privacy solution! (Score:5, Funny)
All right, let me get this straight: First you people bitch and moan when Facebook says they'll save user data forever. NOW you people bitch and moan when this site loses user data forever! You're never happy, are you?!?
Re:Finally! A privacy solution! (Score:5, Funny)
You have a 5 digit UID and you are just realizing this now?!?
Parent
Hardware RAID isn't magic, mirrors aren't archives (Score:4, Insightful)
Since the file system and database were corrupted, it wouldn't matter if it was hardware RAID or software RAID. That's not the problem at all, the problem is there was no archival backup, and their only backup was a file sync... that replicated the database errors on the backup.
To backup a database, you dump it in a serialized form, or maintain a serialized form of the data in parallel with the database.
Free service (Score:3, Insightful)
Simple as that.
The flip side is that this guy's service will probably be the MOST reliable going forward.
Of course he should have had reliable backups; now he is the poster child for backups. Remember, nobody pays you for backups, only for restores.