Facebook Unveils Details of Downtime

An anonymous reader writes "Facebook officially gave out more technical details on the endless loop in a database control mechanism that forced a 2.5-hour shutdown of the social site, and the resulting combination of a productivity burst, increased fertility (check back on June 25, 2011) and mass hysteria all around the world."

not very technical

[datapharmer]

The technical details are that I have an incompatible browser? Really Slashdot? Did you even check the links? of course not...

Re:not very technical

[Anonymous Coward]

Correct link to technical details:

http://www.facebook.com/note.php?note_id=431441338919&id=9445547199&ref=mf [facebook.com]

(anon because I'm not a karma whore)

Re:

[OzPeter]

Correct link to technical details:

Sounds like someone didn't do any testing or that their testing is in adequate.

Re:not very technical

[ProdigyPuNk]

You've got to read some of the comments posted in that thread, it's hilarious.

Anoesj Sadraee It's great to hear and see that big companies like Facebook are so open with what they do. That's rare, very rare. Thanks!

Anne Uriarte ~facebook is stiLL sooo sLow for uz irr! >;'((

Phil McBride this site is becoming less secure lately... hackers are becoming more and more intelligent, i would know, cuz im a white hat lol

Re:

[EdIII]

If you think that is funny, trying reading the article summary this way:

Slashdot officially gave out more technical details on the endless loop in a database control mechanism that forced a 2.5-hour shutdown of the geek news site, and the resulting combination of a productivity burst, increased fertility (check back on June 25, 2011) and mass hysteria all around the world."

Re:

[adamofgreyskull]

Hilarious? I feel sick. Names not changed to implicate the stupid:

Paul Diaz: Will What i Say is get a front page they say facebook down due to server's and we are working hard to fix it get free cash ?:)

Mouhssine Freedom Elmezyani It's very easy to rape facebook !! i know some friends can hack your compt through your electronic adress !& they hacked my compt several time in the pretext of kidding !!

Mauro Guberti I'd like to know what's the necessary qualifications to work like moderator.

And the one that prompted me to close the tab:

Joanne Bozik The following link is the problem.........these people have been sending my name and pic to many stating that I purchased this product and I also in return am receiving the following link in my friends names......Please get after these people

http://www.facebook.com/facebook?v=wall#!/note.php?note_id=431441338919&id=9445547199&ref=mf [facebook.com]

(note, the link is the link to the explanation for the outage)

Re:

[Xest]

"I would know, cuz im a white hat lol"

I'm going to see how many times I can use this line in conversation at work today. It's just brilliant.

Re:

[Twinbee]

Well that's daft because I can't see much difference in real value whether you have 5000 karma points or 5,000,000 'points' unless you know of a way to convert that to cash.

Re:not very technical

[elewton]

You could spend it on Karma whores or play Karma poker.

Re:

[MichaelSmith]

But all the Karma whores offer their services for free anyway.

Re:

[GameboyRMH]

I can't see much difference in real value whether you have 5000 karma points or 5,000,000 'points'

The difference is if you have 5,000 karma points...well, you have 5,000 virtual points on a site only geeks care about or even know exist.

If you have 5,000,000 points, on the other hand, THAT'S OVER NINE THOUSAAAND!!!

Re:

[ultranova]

Well that's daft because I can't see much difference in real value whether you have 5000 karma points or 5,000,000 'points' unless you know of a way to convert that to cash.

Points let you level up, upgrading your abilities and attributes, while karma points let you select a more powerful base class on your next playthrough.

Re:

[Anonymous Coward]

What was he trying to do? The only reason I could think of would be to demonstrate how broken the code is, to prove some point to the project management presumably.
But breaking a production system for that is either stupid or malicious.

I'd guess he had a nerd rage fit, targeted at management most likely, something I think most of the people here can somewhat relate to, but really, what he did is way over the top.

OH NOES

[Pojut]

Meh, it happens...just like a power company, no one says a word when the thing works fine for weeks or months at a time...but when it goes down for a couple hours, people act like it never works.

Re:

[Mitchell314]

But it's a helluva a lot more important for a power company to stay up than FB, no power can cause serious problems. But FB down for two hours, man, the gods forbid you actually are productive or something . . .

Re:

[negRo_slim]

But it's a helluva a lot more important for a power company to stay up than FB, no power can cause serious problems. But FB down for two hours, man, the gods forbid you actually are productive or something . . .

This. FB needs perspective [bit.ly].

Paywall

[tepples]

FB needs perspective [New York Post].

That'd be a bit more convincing if it were more than just an ad for a $3.95 article.

Re:

[SudoGhost]

I actually think Facebook being down would make them LESS productive...because instead of working, they're constantly checking to see if Facebook is back up. Those farms won't...whatever they do with those things...themselves.

Official Technical Details

[ProdigyPuNk]

You are using an incompatible web browser. Sorry, we're not cool enough to support your browser. Please keep it real with one of the following browsers:

Obviously, the error was caused by too many people not keeping it real.

Re:

[jdong]

I've got a great idea! Why don't we have every slashdot reader go in and try to fix the broken link? Then the problem will correct itself in no time!

Link to Facebook Blog Post

[ryanleary]

Since the link in the summary is broken, this is the facebook blog post [facebook.com].

Post contents:
Early today Facebook was down or unreachable for many of you for approximately 2.5 hours. This is the worst outage we’ve had in over four years, and we wanted to first of all apologize for it. We also wanted to provide much more technical detail on what happened and share one big lesson learned.

The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration values ended up causing much more damage than it fixed.

The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store. This works well for a transient problem with the cache, but it doesn’t work when the persistent store is invalid.

Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

To make matters worse, every time a client got an error attempting to query one of the databases it interpreted it as an invalid value, and deleted the corresponding cache key. This meant that even after the original problem had been fixed, the stream of queries continued. As long as the databases failed to service some of the requests, they were causing even more requests to themselves. We had entered a feedback loop that didn’t allow the databases to recover.

The way to stop the feedback cycle was quite painful - we had to stop all traffic to this database cluster, which meant turning off the site. Once the databases had recovered and the root cause had been fixed, we slowly allowed more people back onto the site.

This got the site back up and running today, and for now we’ve turned off the system that attempts to correct configuration values. We’re exploring new designs for this configuration system following design patterns of other systems at Facebook that deal more gracefully with feedback loops and transient spikes.

We apologize again for the site outage, and we want you to know that we take the performance and reliability of Facebook very seriously.

Re:Link to Facebook Blog Post

[Charliemopps]

Wow, I'm impressed by the detail they provided. More companies should handle outages like this. Makes them look like they know what they're doing, they figured it out, and it wont happen again. Instead of the typical stance of pretending it never happened.

Re:

[inKubus]

This shows the beginning of the end for Facebook. Reading the summary they provided provides many details such as the fact they don't have a QA environment or regional segments or anything. It's pretty dangerous to run a site that big like that. And I've read much more they've released that basically says they just hacked mysql replication to update their caches to get it real time across regions. What they should have done to horizontally scale is to implement regional shards and then some type of inter

Re:

[DiEx-15]

We apologize again for the site outage, and we want you to know that we take the performance and reliability of Facebook very seriously.

Just like their commitment to security, I must ask this:

SINCE WHEN?

Wifey was gutted... I wasn't

[rabidjoe]

In the downtime I tried to show the woman the virtues of an IRC channel she could share with family and friends and how much less bullshit she would find there..... "does it have farmville?". There is no hope for the sheeples, the planet is doomed

Re:Wifey was gutted... I wasn't

[Allnighte]

I was going to disagree with you until I read one of the Facebook comments on the blog post talking about the error:

"John Marshall: how do i get job workin with facebook i live in newcastle in uk can any one from facebook staff get or can some one give me a email address that i can use to contact facebook please"

:|
very doomed.

Does the woman have a Wii?

[tepples]

If the woman has a GameCube or Wii, and you want to get her off FarmVille, try buying her a copy of Harvest Moon: Magical Melody.

Re:

[neumayr]

While Harvest Moon has always been a very girl-compatible game even before it had all those social features modern iterations of the game has, it cannot get her away from FarmVille unless her and everyone she's playing FarmVille with has as much access to a Wii as to a webbrowser.
Not going to happen.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:

[rubycodez]

so you, being of independent mind, married a sheeple? you're a sheeple-fucker! and your children will be sheeple!

conclusion: the sheeple aren't doomed....

Re:

[ibmjones]

In the downtime I tried to show the woman the virtues of an IRC channel she could share with family and friends and how much less bullshit she would find there

Right. . . . [bash.org]

"mass hysteria"?

[SuperBanana]

and mass hysteria all around the world.

[citation needed].

First I knew was when I read about it on another tech blog, hours after it'd happened...and I use Facebook. And I work with a ton of people who use it (grad students.)

There wasn't mass hysteria; there was mass ambivalence. I'm now reading all these blog/news postings about how "everyone" went crazy. Nobody was talking about it where I ate dinner. Nobody was talking about it where I had coffee that evening. It didn't make my city newspaper- no "Facebook down, residents in despair" stories to be found.

All this coverage claiming that everyone went nuts seems like a desperate attempt by Facebook PR to make something positive out of this...namely, trying to convince us that Facebook is so integral to the people who use it, it must, of course, be to us as well.

Re:"mass hysteria"?

[Anonymous Coward]

To me, Facebook is about as integral to my life as the toilet is. It's there, it's gonna be used every once in a while and it involves a bit of dirty business that you just can't avoid.

Re:"mass hysteria"?

[microbee]

It's a toilet all right, but a very annoying one: you hear every flush coming from your friends' toilet as well.

Re:

[IANAAC]

It's a toilet all right, but a very annoying one: you hear every flush coming from your friends' toilet as well.

Really?

The first time I see notice of anyone's flushes, I block the flush notices.

Seems to me, if you're complaining about it, you don't really know what you're doing (and that's considering all you have to do is hover your mouse over their post).

Re:

[Lumpy]

You've never used a Bidet have you.

It makes the "dirty business" downright civilized.

That said, you do NOT have to use the toilet paper that is facebook. really.

Re:

[Alien1024]

And if it becomes unavailable for 2 hours, I do the dirty business on the street.

Re:"mass hysteria"?

[ForexCoder]

[citation needed].

[citation] [wikipedia.org]

Re:

[kurokame]

To be fair, most grad students use Facebook primarily to help remind them that they need to come up for air occasionally. If it goes down and temporarily stops vying for their attention, they're likely to continue being absorbed with analyzing the data from their last attempt to apply an epicycle-based model to the sociology of small town karaoke sessions given a behavioral-political tensor formulation of motivation in a multidimensional vector space representing cheese.

Re:

[bsane]

They 'need to come up for air' every 2.5 hours? I don't know where to start.

Re:

[Klinky]

Haven't you seen Waterworld?

Re:

[bsane]

OK- I guess I should have started...

I get the idiom- I think its sad that taking a break after 2.5 is worthy of the phrase... Is 2.5 hours of concentrating outside the realm of normal?

Re:

[DerekLyons]

All this coverage claiming that everyone went nuts seems like a desperate attempt by Facebook PR to make something positive out of this...namely, trying to convince us that Facebook is so integral to the people who use it, it must, of course, be to us as well.

Actually, all the coverage I've seen has been slanted like the summary above - that is, an attempt to denigrate and marginalize those that use Facebook. See the comments in Slashdot's previous coverage for some pretty clear examples of this.

Re:

[kesuki]

you are so on the right track there. the desire to spread news, especially bad news, is screwing with 'the system' there were places built to be safely ignorant... and some to just be quiet and relaxing... oh well.

Re:

[neumayr]

Uh. I took the summary as sarcasm of sorts. Which made your reaction seem like quite the overreaction. Then you got +5 Insightful...

No one is thinking about the big losses here...

[ProdigyPuNk]

Let's look at the important thing here with this outage: How many cows, pigs, chickens, cats, goldfish, etc were made to suffer? I know my girlfriend couldn't take care of her virtual cats, and their litterbox ended up full. They were not at all happy. I'm sure the same thing played out across thousands of FarmVille, MyPets, etc accounts. Please, won't someone think of the animal?

Re:

[roman_mir]

please, what are these animals and why when I hear about FB I hear about farms?

Re:

[loxosceles]

They are a harbinger of the apocalypse.

http://g4tv.com/thefeed/blog/post/702668/DICE-2010-Video-Design-Outside-The-Box.html [g4tv.com]

Re:

[Grapplebeam]

If they really wanted to make some money, they'd sell virtual shotguns so trolls could end the cows, pigs, chickens, goldfish, and your girlfriend's cats.

Re:

[frizzantik]

your girlfriend can change my cat's litterbox if she wants

The metaphysical reason was karmic payback

[Anonymous Coward]

If you suck so bad on a global scale long enough, eventually the universe tries to step in.

Farrmville continued to run?

[Animats]

Does the clock stop for Farmville if Facebook goes down?

Re:

[phantomfive]

If you care, go to http://farmville.com/ [farmville.com]
Only if you care.

Re:

[daveime]

The data for these games are hosted on Zynga servers, not Facebook servers. So, yes, the clock keeps running.

Re:

[kesuki]

nope farmville.com -- the fb net protocol didn't go down just part of it's db so farmville was still playable.

Twitter...

[PmanAce]

I wonder if Twitter had a noticable increase in usage during the Facebook outage, or other social portals?

Re:

[Jedi Alec]

Let's just say that #failbook was rather high in the Twitter trends during the outage ;-)

less productivity actually

[Valpis]

instead of people checking facebook every 5 minutes for the latest, very important, updates as they always do they now constantly was hitting reload for 2.5 hour

Re:

[MobileTatsu-NJG]

Facebook is just a waste of time.

... said the Slashdot commentor on a Saturday.

Huh?

[trasgu]

What is this Facebook thing? Isn't that something kids do on computers?

Fertility was not affected

[michaelmalak]

Unless the particular arrangement of pixels on a Facebook webpage caused a powerful alignment of EM radiation, fertility was not affected. Perhaps fecundity was, but not fertility.

My Favorite Downtime

[WankersRevenge]

My favorite server downtime story occurred back in early 2000 when I was working for Disney's Internet Group. All the message boards for the film and television websites ended up crashing. No one knew the cause and as the web-ops team investigated, we learned that the messageboard server wasn't even housed in any of Disney' server farms. After a lot of hair pulling, we found the server was located in a satellite office in Sunnyvale. Evidently, the server was just on an engineers desk. When that engineer left the company he neglected to tell anyone about the box so when the new engineer took his spot, she found she didn't like the noise from the machine. So one day, she pulled the plug, and put it in some out of the way spot in the office. There wasn't a lot of traffic on it, but it still makes me laugh to think of all the Tim Allen fans in distress over a misplaced box.

Summary FTW!!

[bigrockpeltr]

best summary on /. ever!

Eerie sight

[NicknamesAreStupid]

As Facebook went off-line, I witnessed the unthinkable at an Internet cafe. Young men and women, innocently engaging in social networking intercourse, were suddenly thrown out of their Facebook world and into the reality of the real world, as though all had taken the red pill. Images distorted into 3D with a startling range of colors, sounds beyond stereo, and smells -- odors for the new fifth sense. Everyone looked around to witness "super high def" of each other, and some actually stood to experience a new perspective. Then, as if in concert, the unplugged Facebookers began to touch each other. Immediately untapped hormones raged as ancient primal urges emerged for the first time. Just as it was about to become an orgy of primal lust, the Cafe manager flipped on a You Tube video of Elmo, http://www.youtube.com/watch?v=UZHSDjtD-dg [youtube.com], and a disaster was avoided.

it's only facebook

[thetoadwarrior]

Who cares if it's down even for a day. Just talk about your pointless activities twice as long the next day.

Re:

[glwtta]

Who cares if it's down even for a day. Just talk about your pointless activities twice as long the next day.

Well, we care because as one of the largest sites, they are expected to have their shit together. So when they don't, it's interesting to see what happened.

It's professional interest, it's not that I'm worried that people couldn't plant their snow peas for two hours, or whatever.

Re:

[thetoadwarrior]

Everyone including NASA makes mistakes. Facebook is not going to be perfect and quite frankly I can live without finding out for 2.5 hours that my friend bought a new shirt. I'd be more concerned about someone like NASA reaching perfection long before I would ever care about Facebook reaching perfection.

Re:

[glwtta]

Facebook is not going to be perfect ...

No one said it is, actually I don't think I said anything about perfection. Facebook is one of the largest apps in the world, you don't think the problems they face can be informative to some of Slashdot's readership?

Re:

[thetoadwarrior]

Yes I do think it can be informative and I wouldn't have wanted the topic to be removed from Slashdot. My original point is that it's not that big of a deal despite the amount of attention this has received across the internet. Being interesting doesn't make it important.

Re:

[davecason]

Thanks. I, too, am waiting for something less-than-useless to show up.

Re:

[davecason]

or is that more-than-useless... i'm still in a new-baby fog.

Re:

[davecason]

Thanks. I, too, am waiting for someone to chime in who knows the caveats of their horizontal data structure.

Re:

[Phroggy]

No.

Somebody made a configuration change, and Facebook's error checking code thought the new configuration was invalid. This code was designed to automatically correct for cache errors by querying the database to get the correct configuration - which it thought was invalid, so it would check again.

It doesn't matter what kind of database you have when you've got a system like this in place.

Obviously, their implementation of this error checking system is fatally flawed. That's why they've turned it off, unti

Re:

[davecason]

I was thinking more along the lines of writing integrity checking from scratch for NoSQL vs. using what a major relational database comes with and has been polished by Oracle or Microsoft

Re:

[account_deleted]

Comment removed based on user account deletion