Ma.gnolia User Data Is Gone For Good 450
miller60 writes "The social bookmarking service Ma.gnolia reports that all its user data was irretrievably lost in the Jan. 30 database crash that knocked the service offline. Ma.gnolia founder Larry Halff recently discussed the crash and the lessons to be learned from Ma.gnolia's experience. A lesson for users: don't assume online services have lots of staff and servers, and always keep backup copies of your data. Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis."
Mac reliability (Score:5, Funny)
Re:Mac reliability (Score:4, Insightful)
I had no idea anyone actually used Mac's as servers. Sure, I bet you can get apache running or something but I didn't realize anyone had. Therefore, this is my first bit of exposure to this idea of Macs as servers and its all negative!
Woe is me.
Re:Mac reliability (Score:5, Informative)
Re: (Score:3, Informative)
It would appears so:
Memory
* 800MHz DDR2 ECC fully buffered DIMM (FB-DIMM) memory
* Eight FB-DIMM slots support up to 32GB of memory
* 256-bit-wide memory architecture
Re:Mac reliability (Score:5, Insightful)
Yes, because they don't come with apache and php pre-installed, only a ticky box away from running.
Seriously, do people still not realise that OS X is just UNIX with a pretty UI?
Re:Mac reliability (Score:5, Funny)
Re: (Score:3, Informative)
No, but Mac OS 10.5.x can properly be called Unix [opengroup.org], but only the Intel version, not the PPC version.
Re: (Score:3, Informative)
But that's more of a PR thing than anything. If I raise cows in the pasture behind my hose then they aren't "USDA Certified Organic" or any other such thing, but that doesn't really change what they are - it just means the haven't been inspected an labeled by some committee.
Same with Mac OS X being "Unix". It's more of a stamp of approval than anything.
Re:Mac reliability (Score:5, Informative)
Yeah, but that's exactly the surprising part. Why would you pay Apple $3000 for a xserve running Apache and MySQL, with a crappy service contract (no next-day service, no on-site service-- I've looked into it), when you could buy an equivalent Dell server for $2100, running the exact same Apache and MySQL, and get a next-day and on-site service contract?
Anyone who buys an xserve is an idiot.
!equivalent (Score:4, Informative)
Re: (Score:3, Insightful)
You wouldn't. It's a "right tool for the job" situation and XServes aren't the right tool for running Apache and MySQL. They have the flexibility to run Apache and Mysql, which is nice if you buy them for some other purpose and then either no longer need them for that purpose or find that you have spare capacity and want to use it that way. But
Re:Mac reliability (Score:5, Informative)
a crappy service contract (no next-day service, no on-site service-- I've looked into it)
Not very hard, apparently.
http://www.apple.com/server/support/ [apple.com]
You get 24/7 telephone and email support with 30-minute response. For hardware repairs, Apple-certified technicians provide onsite response within four hours during business hours and next-day onsite response when you contact Apple after business hours.
Re: (Score:3, Informative)
I'm as much of a Mac fanboy as the next guy, but I do want to point out that the "on-site" service isn't as amazing as it sounds.
I have a Mac Pro and recently discovered that the on-site service is provided at the discretion of the local store/repair center and not Apple. If you call with a problem and want on-site service for it, they'll give you a list of local stores that you can then call and try and convince them to come out on a Saturday (it doesn't work, btw). I imagine if you bought all your systems
Re:Mac reliability (Score:5, Informative)
Fine; what company do you trust? HP? IBM? Replace "Dell" with them, and my example still applies. The fact is, *every* server vendor can do better than Apple. Even IBM does better, and they suck.
Oh, and BTW, all servers will have hardware problems from time-to-time. When that happens with your Dell, HP, IBM server, the guy is there in his truck in 4 hours. When that happens to your Apple server, you're SOL.
Re:Mac reliability (Score:5, Informative)
Even IBM does better, and they suck.
One morning I came in and was looking at the logs. SMART was reporting that one of the disks in one of the servers was going to go bad soon. Not 15 minutes after i even noticed this in the logs, an IBM tech was there with a fresh one ready to replace it.
How? The server called home, told IBM about the error, and they disbatched a tech immediately.
If that "sucks", your service must come with free hookers or something.
Re: (Score:3, Informative)
Actually, I priced it out after writing that post, and IBM is not only beating Apple, but is competitive with Dell on the mid-high end of rackmount servers. Much of the reason is that Apple nickel-and-dimes you to death-- for example, they charge you an additional $200 for a secondary PSU which all Apple's competitors in that range have the second PSU as standard equipment (IBM, Dell, HP.)
Equivalent hardware to a basic $3000 xserve from IBM is about $2700, and that includes 2 PSUs and better service.
Re:Mac reliability (Score:5, Informative)
Except its the same hardware...well no, that's not true. You can get a Dell with actual hardware RAID when you're stuck with software RAID on an Xserve.
Furthermore Dell also has a 4-hour onsite 24/7 support package if I'm not mistaken.
I love my MacBook and the OS X desktop experience but you simply can't use an Xserve on business critical operations.
Re:Mac reliability (Score:4, Insightful)
Wait, you mean apples makes all their own hardware? Really?
No, they use CPUs from Intel, hard drives from WD/Seagate/Maxtor/whoever, graphics chips from nVidea/ATI, etc.
So, no, you do NOT get what you pay for hardware wise. You are getting something identical or very similar to everybody else, from the same manufactures, you are just paying a little more.
Re:Mac reliability (Score:4, Insightful)
I think you are missing the importance of a disaster recovery plan, with backups, for any mission critical hardware, regardless of vendor. Why didn't Mag have any sort of backup plan that was tested? Clustered hardware does not equal a backup plan - thanks for trying there.
Was there in fact a schedule of backups of the operational system? This seems like a rubber band and duct tape operation to me.
Re: (Score:3, Informative)
Six Sigma is a quality methodology.
http://en.wikipedia.org/wiki/Six_sigma [wikipedia.org]
In short, you use statistical methods to improve quality to a target goal of no more than 3.4 defects per million opportunities. There's a process you go through - define, measure, analyse, improve, control (DMAIC), and the analysis part is kind of fun for the mathematically-inclined.
You can go as high as twelve sigma, which is some ridiculously small defect rate used in aircraft engineering (for example).
I'm a Six Sigma Green Belt, al
Re: (Score:3, Interesting)
"Seriously, do people still not realise that OS X is just UNIX with a pretty UI?"
Actually, I prefer to think of OS X as UNIX with a good UI. Alas, I can't say the same for the OS X Server tools.
A.
(on topic: at my company we back up our database to three different boxes, in two different physical locations, every day. It's also replicated across the country to a secondary facility in realtime. The backups are periodically written to DVD and stored in a safety deposit box. Oh yea, all this is encrypted.
Re:Mac reliability (Score:5, Interesting)
Are any of the free BSDs or Linux variants certified Unixes?
(Honest question, I don't know.)
Re: (Score:2)
Actually, that pretty much was the defintion of Unix for a long time. Thus why things like Linux, which is not Unix, became popular with the masses.
Unix - You have to buy a million dollar vendor supplied computer, pay a hundred thousand in licensing fees, and were only allowed to run approved utilities with out violating your service contract.
Linux - You could toss it on your garden variety PC, likely one you already own, the cost of acquisition was the cost of the floppies or bandwidth, and you could do wh
Re: (Score:3, Interesting)
Unix - You have to buy a million dollar vendor supplied computer, pay a hundred thousand in licensing fees, and were only allowed to run approved utilities with out violating your service contract.
Would you care to explain that bit to oldSCO (aka Santa Cruz, not the SCOundrels)?
I believe SCO was certified SVID compliant.
Re:What is this "UNIX" you speak of... (Score:4, Interesting)
Apple is 1% hardware and 99% Marketing. Not too much they do can't be done on a Dell or HP. They just make it appear to do it better/slicker/faster, that's all.
I'd peg it at 10% hardware, if not more. The internal hardware layout of Apple's desktop towers borders on beautiful. Beats Dell and HP hands down.
And, while its hardware failures tend to be more spectacular, I've generally found Apple hardware to be more reliable than any of the Wintel vendors. (...speaking as someone who has been supporting computers since before MS-DOS or the Mac...)
Re: (Score:3, Interesting)
It's hard to know whether you're trolling or not.
There are OS X servers out there and they perform rather well. I know because I admin 50 of them, and have met hundreds of others who administer them in school systems across the state.
You may also be familiar with iTunes, or Apple's movie trailer website. I'm sure a large part of those are Xserves and Raids.
I'm not saying they are maintenance free, but they are out there.
Furthermore, a few years back there was a rather large beowulf cluster of mac towers tha
Re:Mac reliability (Score:5, Insightful)
Mac servers are pretty. They do okay, they have nice swanky data enclosures, and the form factor is roughly the same as anyone elses.
It's just whether or not you want to use OS X. I disagree that OS X is "just unix," however. It's not even "just linux" or "just bsd". OS X has it's own warts, and while it may be stable and friendly, I'd rather have a real *nix running on less pretty hardware.
The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine (If you say windows mac volume, I'm mailing a dead fish to your house).
Otherwise, you know, you can install apache, whatever, but it's not any different from using a regular linux server in terms of increased functionality, and there are some significant OS update issues that can cause problems. Mac updates are of the all or nothing school, and they WILL break stuff, so you need to be careful.
Re: (Score:3, Informative)
The best use I've ever had for the big Mac servers is running as a file server in a windows/mac environment. If you still have any pre-OS X machines around, that's about the only way to get them all on the same machine
Negatory - the best answer there is samba+netatalk. I did this at my house and then proceeded to do it again when I was the network admin at a spot with a mix of PCs and various-vintage Macs. Since you are generally running such a solution on a free Unix system (I did it on Linux both times) you also have access to pretty much ever other network filesystem too. Ostensibly it should be easy to add Appletalk DDP support to a modern Novell system running on SuSe, and it's definitely been done on various small
Re:Mac reliability (Score:4, Informative)
It means you don't have to pay the performance penalty that netatalk has from resource fork handling since HFS+ is a native file system.
So if you're dealing with lots of small files with both forks, you're going to pay a penalty. What that says to me is that there are certain limited cases in which you might still need to use an Apple fileserver for performance, but in almost every real-world case where you actually have macs as clients, is it really an issue?
Re:Mac reliability (Score:4, Funny)
(If you say windows mac volume, I'm mailing a dead fish to your house).
Why, so it will attract the penguins?
Re:Mac reliability (Score:4, Funny)
and it comes with a free black turtleneck!
Re:Mac reliability (Score:4, Informative)
This is meaningless today. Most Unixlike systems today are not certified Unix systems.
OS X has some significant differences from traditional Unixlike systems and Linux - not necessarily disadvantages:
You can certainly add grid computing software to other operating systems. OS X is missing some functionality that a "regular linux server" may have. Even when considering third-party software, there are many things that can be done in Linux but not in OS X.
Mac OS X security updates certainly are "all or nothing" - you have to install all of the patches included in the package or install none of them. Each package includes many fixes, and sometimes they break things [slashdot.org]. The updates are not available as individual pkacages. You cannot select which updates are applied to the system.
RHEL/CentOS has point releases, but there are plenty of individual package updates in between (to fix bugs, compatibility, and security issues.) Individual package updates are released when they are ready, not as part of a large security update bundle or a monthly schedule.
Re: (Score:3, Informative)
> I bet you can get apache running
Every Mac comes with apache - "getting it running" means checking a single box in the system preferences dialog.
Same goes for Samba for example.
Re: (Score:3, Funny)
Re: (Score:3, Informative)
Um, how is that relevant? MS isn't using those minis to run an internet service of any sort, they're using them for brute force automated testing of a desktop application that was specifically designed to run on desktop-class Macs.
Comment removed (Score:4, Funny)
Re:Mac reliability (Score:5, Funny)
I want to say something here, but I get the feeling that no matter what I say, I just wont be herd.
Re:Mac reliability (Score:5, Funny)
In other news (Score:5, Funny)
Re:In other news (Score:5, Funny)
Re: (Score:2, Funny)
Re: (Score:2)
Re:In other news (Score:5, Funny)
Re: (Score:2)
Despair not, Mr Furry was revived and now running Blizzard's instance servers.
Re: (Score:2)
And nothing of value was lost.
But now who will know of my status updates, you insensitive clod!
Food for Stallman (Score:3, Insightful)
This bad news is delicious food for Stallman's argument against "cloud" services.
Re:Food for Stallman (Score:5, Insightful)
Stallman's argument is more that cloud services are almost always non-open. He does not have a per se objection to cloud services - and if you were to reveal all your source code and protocols, I doubt it would be objectionable to him.
Of course it's impossible to free cloud services in the sense of modification and distribution, but if the source is open you have the chance to make your own.
Re: (Score:2)
In fact Stallman is against software as a service, he said so when he was in my city.
Even if he didn't say it, it goes against the FSF phylosophy. The whole point is giving freedom to the user. If someone else has your data, you lose some freedom to do what you want with your data.
Re: (Score:3, Insightful)
Well, that and the fact that many cloud services destroy your privacy. RMS argues that we are being shortsighted to trade our privacy for "kewl!"
He's right, you know! About everything.
http://www.guardian.co.uk/technology/2008/sep/29/cloud.computing.richard.stallman [guardian.co.uk]
Re:Food for Fault-Tolerance (Score:3, Insightful)
It's food for any argument against any web service that doesn't publish it's reliability information or publicize the data for what types of mechanisms it has in place in case of disasters like a corrupt database, fried motherboard, or busted hard drive.
There's a design methodology that's used by NASA for manned missions: Any individual component should be able to fail without compromising the mission. Of course, in the last few decades we've seen 2 out of 5 Shuttles go ka-boom! so obviously this NASA gu
Re: (Score:2, Insightful)
NASA guideline isn't enough and it's *REALLY* hard to prevent failure when a perfect storm of multiple systems experience failure at the same time.
I'm not saying that saving Apollo 13 wasn't hard, or an extremely great accomplishment, however I am going to say "slick and pretty" (the shuttle) is generally the opposite of "robust" or "fault tolerant." Slick and pretty is also usually more expensive.
The basic, non-pimped xserve is $2999. An identically configured node from eRacks, running your choice of BSD (the default on these quad-core Xeons seems to be OpenBSD) or Linux, $1894 -- leaving you with plenty of room in the budget to build a bigger, bad
Re: (Score:3, Insightful)
Neither the Challenger nor the Columbia represented simultaneous multiple failures. They *did* represent cascade failures that should have been planned for, but weren't.
Re: (Score:2)
Needless loss (Score:5, Insightful)
Argh, why not just add a backup or replication database on one of the spare Mac Minis?
That way you would have needed a complete server farm disaster to mess things up irretrievably.
Re:Needless loss (Score:5, Insightful)
I definitely have my doubts that someone who could make this mistake is all that capable "lessons learned."
Re: (Score:3, Insightful)
Except cron+tar isn't sufficient. You need versioning. Otherwise if your database is corrupted and you don't notice immediately, your backup gets corrupted automatically.
I back up my web sites using cron + rsync + rdiff-backup.
Re: (Score:2)
I can't believe anyone would run a commercial system without backing things up. Hell, even home users, if they have anything of value, need to do backups.
It's just not that hard these days especially with cheap NAS boxes, low-cost hard drives, etc.
Re: (Score:2)
Why is it that I feel the stuff I have setup at home is more robust than some "professional" shops? Is the world more like The Daily WTF [thedailywtf.com] than I've been lead to believe?
I'm not saying my system is perfect, but it's redundant in at least two locations.
Laptop<->Server<->Server HD 2<->Dreamhost.
My MySQL databases which just keep stuff like weather and temp (from my 1-Wire system) is dumped nightly and sent to my Gmail account. (It's also not a few TB server...) but seriously. How hard is it to
Re:Needless loss (Score:5, Insightful)
A simple periodic dump to an external hard drive would have at least been something. I know that small-time operations shouldn't be expected to have robust backup schemes, but if your primary purpose is to store other people's data, the FIRST thing on your mind should be how to back it up. Once you lose someone's data, they'll never use anything you put out again, and they'll tell all their friends not to either.
Re: (Score:2, Insightful)
Small time operations should be expected to back up just the same as large ones. I wrote simple routines to backup my db nightly, compress it, upload it and on the receiving end decompress it and restore it. If any step fails it emails me. I check it manually every week and save backups for 2 years (quite a bit of data but for legal reasons it's necessary).
The whole setup took me maybe a day to get working. There is NO excuse for not having backups.
I lost one of my primary servers on a sunday at around 5pm.
Re:Needless loss (Score:5, Insightful)
If you read the transcript, that's what they were doing, a simple firewire DB dump.
The problem is that they never tested the backups, and they didn't keep versioned backups. So they'd been backing-up the corrupted database for awhile before the site finally crashed for good. When it crashed, they only had the corrupted database backup. Additionally, the DB server was on RAID but of course the corrupted DB would just get saved to both HDs, so that's no good in a situation like this.
Basically, when the site crashed, he had three copies (2 RAID, 1 backup) of the data: all corrupt. The guy wasn't totally retarded when it came to backups, just 80% retarded.
Re:Needless loss (Score:5, Insightful)
Argh, why not just add a backup or replication database on one of the spare Mac Minis?
That way you would have needed a complete server farm disaster to mess things up irretrievably.
Replication gives you redundancy, much like RAID does. It lets you survive a hardware failure or two. It is not a backup. If the building burns down, or a tree falls on your server room, or lightning fries everything you are still screwed.
What they needed was a backup. A tape, or removable HDD, or a flash drive, or a CD, or something that can be taken out of the building on a regular basis. Once a day, once a week, once a month... Whatever.
Then, no matter what happens to your live hardware, you've got a backup you can restore from. Buy some new hardware, throw your backup at it, done!
Re: (Score:2)
Who the hell is Ma.gnolia (Score:5, Insightful)
And how can they be slashdot worthy when they are a social networking site with ONLY a half a terabyte of data? In short, who cares?
Re: (Score:3, Funny)
At least if you had privacy concerns with them, you have nothing to worry about now.
lesson #1 (Score:5, Interesting)
lesson #2, trust no-one with your data
lesson #3 disaster recovery capability only exists after it's been tested
lesson #4 backups are useless unless you can prove you can recover from them
Re: (Score:2, Funny)
Re: (Score:2)
lesson #1 on the 'net you can't tell the major corporation from the kid in a garage
lesson #2, trust no-one with your data
1 and 2 don't really matter if you've got a backup. Who cares if it's some kid in a garage if you've got a backup? If it's more convenient to have your data on some kind of web service, go for it! But make sure you've got a backup.
lesson #3 disaster recovery capability only exists after it's been tested
lesson #4 backups are useless unless you can prove you can recover from them
This is really where things fall apart over and over again. I've seen tons of clients with no backup at all... Or a backup that they've never tested and they just assume it's working correctly.
It isn't a backup unless you can take it off-site, and it isn't a backup unless you
Re: (Score:3, Funny)
I love whens someone has a replicated DB as a "backup". I like to say okay.. "Drop table users". And then it dawns on them that the drop command would replicate.... ;)
Not the platform's fault... (Score:3, Interesting)
Good backup strategies are critical to any operation, regardless of platform. I've seen similar things happen with MSSQL server databases as well as Oracle running on the most powerful Sun box you can get (circa 2001).
One database backup strategy I've seen used rather successfully is doing a straight SQL dump every night and then copying the sql file over to somewhere else; even if the database became hopelessly corrupted there's still a way to re-import everything.
Of course, this is in *addition* to mirroring, tape backups, etc.
Re: (Score:2)
Agreed. We have a mirror that we do weekly EXPorts from, as not to slow production environment. On prod have a second safety net of RMAN, but I've never trusted it. I've taken all the bloody courses, it just seems too failure prone. Heaven forbid you open your database with reset logs. It mucks the SCN up, or something irrevocably small. I'm still not confident about changing Oracle versions and have backwards compatibility.
In short, yeah, exp is tried and tested for recovery.
Re: (Score:2)
Opening oracle with resetlogs resets the online redo logs and sets the log sequence number to 1; also called creating a new incarnation of the database.
That prevents applying archive logs from before the reset (i.e. previous incarnations ) but which may contain more recent data than what's in the datafiles.
I use EXP/IMP myself, but for larger databases it can be impractical. One of my systems takes around 120 hours of processing time to read in an export and write it to a blank schema ( which we tested whe
Re: (Score:2)
This I know. But, heaven forbid you open your database like that, and you are screwed. I believe this oversight has been correct in Oracle 11g, but I'm not entirely sure, as I'm trained in 10g. We tend to keep our OLTP databases relatively clean via purge processes, and offload required data to our OLAP. As we have to maintain 7 years for litigation purposes on tape backup, having the EXP is basically yeah, the only thing I trust. It has never taken over 24hrs to perform an EXP, and this meets our dai
Re: (Score:2)
Yes, but straight dumps take time, a lot of time. They need to be consistent to be useful and then you have to hold table/db locks which can interfere with the operation. Even if you can dump it without locks, 500GByte of data over a GBit link takes al least 1.2 hours. And that assumes that you can get the data that fast, let alone transport it. Mysql is slow at doing dumps on innodb (myisam can be copied rather easily).
When databases and tables get large, things start to suck big time if you want real back
Re: (Score:3, Informative)
With a transactional database who cares how long it takes - the state isn't going to change. If you're backing up your 500GB MyISAM tables, well, you're asking for trouble. Since you mention MySQL, use innodb tables with the dump option "--single-transaction".
Lesson? (Score:5, Insightful)
discussed the crash and the lessons to be learned
Lessons such as "Regularly monitor and maintain backups like and business should?"
Re: (Score:2)
Some other web 2 site died a month or two ago.
The story was on /. but I don't remember the services name. Turned out the guy had a single copy on a RAID array which got wiped, game over.
Lesson still not learned apparently.
Re: (Score:2)
"Ma.gnolia was a one-man operation running on two Mac OS X servers and four Mac minis"
So what? Hard drives are cheap. Buy a couple and make backups.
Re:Lesson? (Score:5, Interesting)
I love it when people say things like this. It shows me that they've never actually had to set up an enterprise backup strategy. I'm certainly not defending the Ma.gnolia guys, but I also can't stand it when people are on a shakier soapbox than they realize.
I'm sorry, but when you are used to the whiz-bang-pretty of Web2.0, the state of enterprise-level backups is horrifically archaic and dismal. And, btw, given the size of today's hard drives and databases, for pretty much all intents and purposes "Enterprise" == "More than one computer with more than just a few files on a drive".
Compare and contrast: a 1 TB hard drive will run you roughly $100. Do you know how much it then costs to backup that TB?
LTO-4 tapes, 800GB each, $50-$150 each tape plus roughly $2500 for the drive. Figure 2 tapes/day * 10 days backups = 20 tapes * $100 = $2000 in tapes alone. Congrats, that 1 TB just cost you $4500 in enterprise backups ... not to mention the time involved each day in doing a backup. You might save yourself some time and money by doing incrementals ... but then you have to balance that risk with complexity of backups and difficulty in restores.
NAS is trickier. The cheap NAS solutions, sub $1000 such as Buffalo and LaCie, aren't going to get you much more than a TB or two. And at that point, are you really any better off than the RAID solution? Maybe, maybe not. As you start to scale into IBM or Dell solutions, you are almost immediately beyond a $2500 price point before you even get to hard drives. Oh, and don't forget the cost of a gigabit switch so that it doesn't take you days to do a single backup.
iSCSI? Seriously? Not an option for SOHO businesses.
Then there's backup software to contend with. It's not just as simple as "go buy a copy of BackupExec" -- there's different licensing for databases, and network backups, and whatever arcane rules they want today. I'm a PC guy so I can't talk much about Enterprise-level Mac backup solutions, but I can without a doubt say that Time Machine is not one of them.
It's even more dismal when it comes to Open Source solutions. Have you ever actually tried to setup Bacula? It may be the 600lb gorilla of OS backup solutions, but it's still a royal pain. And to the "just set up a cron job for rsync" guys, c'mon, really? Good luck with that.
So, please, let's dispense with the thought that backups are easy. Backups really suck. Hard. That's why so many people want to think of RAID as a backup solution -- because the step from one hard drive to two or three is easy, but the next step is much farther away than you think.
Re: (Score:2)
Thats all fine and dandy, except, where I work someone has to prioritize our assignments, one of those is getting the backup scripts up and working for nightly backups, but so far that job has been prioritized way below adding new fizzbang to the interface.
Currently I'm just grabbing a new snapshot of the database once in a while.
(On the positive side we know we can recover our backups, that at least is verified)
I see this all too often with our clients... Backups are considered a low priority because everything is up and running right now. Nobody wants to waste time/money on something that isn't necessary right now.
Then they have some kind of horrible software crash that eats all their data... Or the hardware goes up in a puff of smoke... Or the building floods... And all of a sudden they're scrambling to get things back up and running.
Usually after a disaster like that they'll take backups more seriously...
Lessons Learned?? (Score:5, Insightful)
Macs (Score:2, Funny)
You shouldn't use shiny plastic ornaments for serious business.
Re: (Score:2)
You shouldn't use shiny plastic ornaments for serious business.
Or fruits.
Time Machine Anyone? (Score:2, Interesting)
Re: (Score:2)
Presumably because the database was stored in a single .sql file, mirrored by each server, Time Machine wouldn't be particularly effective, because it would copy across a (new) copy of the massive database every time.
Time Machine is excellent for backing up lots of little files (on a home PC, say, or a web server's /httpd) but for backing up big files, it's very inefficient. Additionally, Time Machine wasn't included with OS X 10.4 (both distributions), so if it wasn't running Leopard or Snow Leopard, you'd
Re: (Score:2)
flyback [flyback-project.org] - I bet you could get it to work on OSX :D
What's with the OS X users losing data? (Score:4, Insightful)
I mean, just because a few medium-profile sites running on Macs have experienced a failure causing data loss doesn't make them unique. Every OS and every type of hardware will, at some point, experience a failure. It's the PEOPLE that make the failure a problem, and it sure looks like this tard was a problem.
Who the hell doesn't back up their data? Seriously? This is "Slashdot worthy" because some hapless Mac user lost their data. BOO FUCKING FAIL. Move on.
"Private relaunch?" (Score:5, Interesting)
"Gee, Bob, we have the proof that this thing works. Why don't we sell it already?"
"Well, Bill, nobody wants to buy it and grandfather in all the whining freeloaders and their data."
"It's too bad we can't just drop all the data and start fresh."
"Well, why not, Bill? All we have to do is say it's been lost and can't be recovered. We can tell the buyer what's actually happening so they don't think we're total IT rejects who couldn't figure out a data retention policy."
"That's why I like working with you, Bob. You always have a way around the problem."
Have fun with it. The names have been changed (one changed anyway and one added), well, because it probably has nothing to do with reality. It sure is fun to ponder, though.
Transcript of interview (Score:5, Informative)
Rather than watch the video or download the 23MB MP3, you can read the full transcript here:
http://ratafia.info/post/78915439/transcript-and-commentary-for-whither-magnolia [ratafia.info]
I can read much faster than I can listen.
Finally! A privacy solution! (Score:5, Funny)
All right, let me get this straight: First you people bitch and moan when Facebook says they'll save user data forever. NOW you people bitch and moan when this site loses user data forever! You're never happy, are you?!?
Re:Finally! A privacy solution! (Score:5, Funny)
You have a 5 digit UID and you are just realizing this now?!?
Hardware RAID isn't magic, mirrors aren't archives (Score:4, Insightful)
Since the file system and database were corrupted, it wouldn't matter if it was hardware RAID or software RAID. That's not the problem at all, the problem is there was no archival backup, and their only backup was a file sync... that replicated the database errors on the backup.
To backup a database, you dump it in a serialized form, or maintain a serialized form of the data in parallel with the database.
Free service (Score:3, Insightful)
Simple as that.
The flip side is that this guy's service will probably be the MOST reliable going forward.
Of course he should have had reliable backups; now he is the poster child for backups. Remember, nobody pays you for backups, only for restores.
Lesso learned (Score:2)
Users: if you're trusting your data to someone else, you need to insure one of two things. Either you need a signed, iron-clad written contract guaranteeing service with nasty penalty clauses requiring the service to compensate you fully for all the costs of data loss (and sufficient insurance and/or confidence that the service has the wherewithall to pay those penalties and not just flee into bankruptcy leaving you holding the bag anyway), or you need a backup of your data under your own control and in a f
0,5 TB = 500 GB (Score:2)
Backup Testing? (Score:2, Insightful)
Ouch... Isn't part of a backup strategy to sometimes attempt a recovery from a backup, on a test system?
Re: (Score:2)
Yes. He addresses this and acknowledges he did not either deliberately fail his system or conduct extensive tests to ensure his backup scheme was adequate.
He acknowledges this was one of many 'lessons learned' (aka huge mistakes made).
Social Bookmarking? (Score:2)
User Error (Score:2)
Oh don't worry it's in the 'cloud' somewhere (Score:2)
After all, it's the new paradigm.