Facebook, Instagram, WhatsApp, and Oculus Have Been Suffering Global Outage For More Than 3 Hours Now [Update] (arstechnica.com) 252
Facebook -- and all the major services that Facebook owns -- are down today. ArsTechnica: We first noticed the problem at about 11:30 am Eastern time, when some Facebook links stopped working. Investigating a bit further showed major DNS failures at Facebook: "Google anycast DNS returns SERVFAIL for Facebook queries; querying http://a.ns.facebook.com directly times out."
The problem goes deeper than Facebook's obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services -- which are hosted on Amazon rather than being internal to Facebook's own network -- were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services' load balancers were reachable, the application servers that should be feeding the load balancers were not. A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. With no BGP routes into Facebook's network, Facebook's own DNS servers would be unreachable -- as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR. UPDATE 10/4/2021 22:15 UTC: Facebook is coming back online after a six-hour outage due to DNS routing problems.
"Inside Facebook, the outage broke internal systems as well, leaving employees unable to get into offices and communicate easily with each other," reports The Verge. "Some told The Verge they were using work-provided Outlook email accounts, allowing Facebook workers to email each other but unable to send or receive emails from external addresses."
Not only was it a rough day for Facebook and their stockholders, but it was especially hard on CEO Mark Zuckerberg. According to Bloomberg, Zuckerberg's personal wealth has fallen by more than $6 billion in just a few hours.
The problem goes deeper than Facebook's obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services -- which are hosted on Amazon rather than being internal to Facebook's own network -- were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services' load balancers were reachable, the application servers that should be feeding the load balancers were not. A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. With no BGP routes into Facebook's network, Facebook's own DNS servers would be unreachable -- as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR. UPDATE 10/4/2021 22:15 UTC: Facebook is coming back online after a six-hour outage due to DNS routing problems.
"Inside Facebook, the outage broke internal systems as well, leaving employees unable to get into offices and communicate easily with each other," reports The Verge. "Some told The Verge they were using work-provided Outlook email accounts, allowing Facebook workers to email each other but unable to send or receive emails from external addresses."
Not only was it a rough day for Facebook and their stockholders, but it was especially hard on CEO Mark Zuckerberg. According to Bloomberg, Zuckerberg's personal wealth has fallen by more than $6 billion in just a few hours.
Problem? (Score:5, Funny)
"We first noticed the problem at about 11:30 am Eastern time, when some Facebook links stopped working. "
Huh, I thought my coworkers seemed unusually attentive today - albeit a bit high-strung and bad-tempered, like they hadn't had their usual morning cigarette.
Re:Problem? (Score:5, Insightful)
I would expect them to actually be more chatty, and easy going.
The Facebook feed of Hate had been stopped for a day, and have to judge people ideas and views based on their own reaction vs what the algorithm tells them they should believe.
Perhaps we can avoid the Politics trigger point of the day. Shoes, Beans, Cars, Weather, Light Bulbs....
Re:Problem? (Score:5, Insightful)
Re:Problem? (Score:5, Insightful)
Well, I think it shows you might think about not basing your communications on an app from a less that trustworthy (in so many ways) company .
Re:Problem? (Score:5, Funny)
>> countries run all their communication on WhatsApp
don't do that.
Re: (Score:3)
Tell it to Brazil, India, or several countries in Europe. It's not me you need to convince, it's millions of people, businesses, and governments all at once. Good luck.
Re: (Score:2)
Of all days, my wife needs to talk to her brothers in Peru today and WhatsApp pukes.. We had to go to the tienda to pick up phone cards and actually look up their phone numbers.
Oculus (Score:5, Funny)
I'm so glad Facebook bought Oculus so that every Facebook-related problem can just make your hardware unable to be used. So cool that they bought them and then made their shitty servers mandatory.
Re: (Score:2)
Hum, I just tested this with my Rift-S, I can still access my Steam VR games. I can't access the Oculus social features, I think. I am not sure, I have never used them. Maybe it's a bigger issue for Quest users.
Re: (Score:2)
Apparently you can access already downloaded content, but not download anything new or use the social features.
Re:Problem? (Score:5, Funny)
Re: (Score:2)
Glorious! (Score:5, Informative)
4.5 hours now.
Re: (Score:3)
And still nothing lost!
Granted my work has Facebook blocked, for security reasons. However all it is for now is to put people into bad moods, better off than on.
Re: (Score:3)
How do you access useful and relevant technical article and white papers without the world largest social media sight? /s
Re: (Score:2)
Rectal data retrieval systems are independent of Facebook's internal DNS.
Hot on the heels of "Is Facebook the AOL of 2021?" (Score:5, Funny)
Hot on the heels of 9/30's Slashdot article "Is Facebook the AOL of 2021?" we have Facebook doing an AOL impression. Who else remembers the 19-hour global outage of AOL in the 1990s?
Re: (Score:2)
Also caused by a router rule propagation problem back then. I believe that phenomenon was called "flapping."
One can hope (Score:5, Insightful)
Maybe IT can close this issue as WONTFIX, and humanity will be the better for it.
Re: (Score:2)
NOTABUG
Whistleblower was serious! (Score:5, Funny)
I guess the earlier reported whistleblower reports were pretty serious and they decided to just delete facebook
Dontcha know? It's (Score:4, Funny)
"What It Feels Like To Be Trump Day"
Re:Dontcha know? It's (Score:5, Funny)
Dunno about that, I don't feel vengeful, bigoted, and obsessed with conspiracy theories, all while itching to play golf SO BAD.
Re: (Score:2)
Trump is just an actor doing whatever he need to win the election.
He's kinda like DBZ's Mr.Satan.
Re: (Score:2, Informative)
No I have a good track record at succeeding at things, not failing miserably and blaming someone else for my failure.
Re: (Score:3, Insightful)
> No I have a good track record at succeeding at things
You've never owned a hotel nor were President of the USA. True, he may have obtained them via dubious means, but "succeed" by itself doesn't require morality.
Re:Dontcha know? It's (Score:5, Insightful)
But Twitter is still up. Trump never really cared about Facebook himself, he was all about Twitter. Someone would need to do the same thing to Twitter to really duplicate that feeling.
Not that I'm saying anyone should. Just, you know, if it were to happen, I don't think anyone would mind.
Re: (Score:2)
But Facebook fans want Facebook up for similar reasons as Don's Twitter desires. In other words, the brand itself is not very relevant to the emotional association of the comparison.
As an analogy, acholic A may prefer wine and acholic B may prefer beer. But, they can still relate to the problems of alcoholism in general. The beverage difference makes insufficient difference to the emotions and sensations being compared.
Best News Today! (Score:2)
Quick. Lets do an RCA and keep using those techniques to keep that giant dumpster fire down forever.
--
To become truly great, one has to stand with people, not above them. - Montesquieu
Comment removed (Score:5, Funny)
And nothing of value was lost (Score:5, Insightful)
Subject says it all.
Re:And nothing of value was lost (Score:5, Funny)
I really like today's new and improved Facebook.
Re: (Score:2)
Re:And nothing of value was lost (Score:5, Interesting)
Re: (Score:2)
I'm in a science fiction book club that holds its discussion on FB. That's not *worthelss*, but a FB outage isn't an emergency.
That kind of stuff, and things like sharing photos with family members is fine. The real problem is all the garbage FB shovels at you that you didn't ask for, but the algorithm figures will goose your engagement metrics. That stuff is a total waste of time. In your information diet, it's the Cheet-Ohs.
Re:And nothing of value was lost (Score:4, Interesting)
I had similar reasons for using Facebook. Particularly keeping up with my distant friends and family. But this year I went through a particularly bad bout of depression. I don't blame social media for causing it, it was a really rough year for a lot of personal reasons, but I realized that it was making the problem a lot worse. Mostly due to the news feed. If you can avoid the news feed then kudos. It always tended to suck me in and just made exhaustion and depression so much worse.
I made the personal decision to delete all of my social media accounts and ended up feeling a lot better afterwards. I'm not of the opinion that my decision should be other peoples' decision. I just made a lot of observations about the affect that social media was having on my mental health and like to share.
Why I quit social media [youtu.be]
Re: (Score:3)
Re: (Score:3, Informative)
Re: (Score:2)
IRC is forever. Also, phoning someone is still possible.
Re: (Score:2)
Yes but forming a group of people and phoning around is a PITA
Re: (Score:3)
Can you phone someone to convince the gate chick to let you on a plane? There was quite a scramble at the airport today as KLM was unable to send boarding passes to people's phones.
Don't underestimate just what WhatsApp is in some countries. WhatsApp has a pretty feature rich Business API and it's used quite heavily for a shitton more than simply gossiping with your BFF.
Re: (Score:2)
Subject says it all.
With Facebook? No. With WhatsApp yes. You clearly live in a country where it's not popular, or the method of communication used almost exclusively.
Is all the power supplied to the data centres ... (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
Even funnier if they were wiping papers and shredding servers.
did someone ? (Score:2)
Post the truth about something important and secret ?
Did a system update go really bad ?
Senior sysadmin on vacation and CIO told the juniors to do it anyway ?
Durham and the Marines visiting ?
Did a hacker re-enable Trump's account ?
Re: (Score:2)
This is obviously part of Operation Jade Helm. All patriotic citizens are asked to gather in the parking lot of Walmart Supercenter in Waco, TX and await for further instructions from Rudy Giuliani via twitter.
Re: (Score:3)
Post the truth about something important and secret ?
There is speculation [cnn.com].
Finally working as intended (Score:5, Funny)
May be the Internet is finally detected the problem and routing around it, as intended.
Re: (Score:2, Funny)
[wipes coffee from computer screen]
You sir, have just won the entire internet for a day. Enjoy your prize! :-D
Re: (Score:2)
But don't drop it, otherwise the Elders of the Internet will be mad at you.
Eh? What is Facebook? (Score:3, Insightful)
Funny... (Score:5, Insightful)
I didn't even notice!
And the post count on /. (Score:2)
has gone up.
A day without being forcefed bullshit (Score:2)
day is feeling nicer already
Re: (Score:2)
you could try, you know, deleting your FB account/s.
Re:A day without being forcefed bullshit (Score:5, Funny)
not right now you can't
Re: (Score:2)
Re: (Score:2)
Couldn't happen... (Score:2)
Couldn't happen to a more deserving bunch of folks...
And nothing (Score:5, Informative)
Insidious plot... (Score:2)
What if it is an insidious plot to make the world realize they don't care about the whistleblower, they still want their facebook with the damage and degradation they offer...
It's been hours... (Score:2)
It's been nearly 5 hours since I've seen that photo of 9-11 that Facebook's been preventing me from seeing for several years.
And now something at Google? (Score:5, Interesting)
Re:And now something at Google? (Score:4, Funny)
Yeah, Google+ is down for me.
[Rumor] FB messed up their BGP configs so bad... (Score:5, Funny)
...they're literally locked out of their own datacenters right now. [twimg.com]
Re: (Score:3)
That literally had me laughing out loud. Clusterfuck conglomerates getting caught in their own web of stupid is one of the most hilarious things I've seen in weeks.
Alan Sherman (Score:2)
When I heard about today's Facebook outage I was reminded of the song by Alan Sherman: Hello muddah, hello faddah. Or, at least the last part of it:
Wait a minute, it's stopped hailing
Guys are swimming, guys are sailing
Playing baseball, gee that's bettah
Muddah, Faddah kindly disregard this letter
Not just facebook.... (Score:2)
According to downdetector.com many major ISPs and services are having issues. BGP? Major attacks?
Re: Not just facebook.... (Score:3)
Law Enforcement Action / Court-ordered takedown? (Score:2)
Re: (Score:3)
Nah, that would result in less disruption.
Re: (Score:2)
Maybe this is a Trump plot. Get some loyal insiders in the FBI and FCC to take out FaceBook, make sure the world knows it and watch the mob with pitchforks burn down the capitol.
Re: (Score:2)
Re: (Score:2)
Looking around at other news sources and social posts, the people who made the BGP changes are not anywhere near the data centers. They have to coordinate with those with physical access to get it all done.
Why Oculus too? (Score:2)
Re: (Score:2)
Because telemetry. If they can't extract their tenth of a cent of value in data, they'd prefer to deny you service entirely.
Re:Why Oculus too? (Score:5, Informative)
No, once FB bought oculus it became just another way to spy on you and give you ads.
Facebook...took themselves offline? (Score:5, Interesting)
"A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled."
DNS, still ever-present as the Achilles heel of the internet, causing an outage isn't surprising. Couple that with some load balancer and content caching issues, and yeah. Could be a considerable impact like this.
But this...This almost smells like Facebook got hacked in a major way, and they took themselves offline. Who else would make that BGP move and why...
Re:Facebook...took themselves offline? (Score:5, Interesting)
Well, if they pulled the BGP routes for Facebook's network it doesn't matter if facebook's DNS servers are working or not. Nobody can get to anything on Facebook's networks, even if they have the IP addresses.
I just pinged all of the facebook ns addresses listed in DNS, and none of them are reachable.
Re: (Score:2)
Found some posts by supposed employees. They need physical access to the routers to re-announce the correct routes to the Internet again. And those with physical access don't know how to do the actual configuration fix. Sounds fun.
Smart ISPs could be prepared for global disasters or terror attacks by caching and reinstating specific major network routes in the event of a complete BGP cutoff..
Re: (Score:3)
No, that's not what's happening. They use Lenel for their physical security (they're either the 2nd or 3rd largest security system in the world), those panels will be perfectly happy running in offline mode until the end of time. Anyone who had access this morning could still badge in until the expiration time of the access level.
**BUT**
No one who didn't have access to the site before the network outage can be granted access now. If they were accustomed to remove all access when it was not necessary, only
Comment removed (Score:3, Funny)
Bootstrap your systems occasionally (Score:5, Interesting)
I felt a great disturbance in the Force... (Score:5, Funny)
Re:I felt a great disturbance in the Force... (Score:5, Funny)
[insert Meme of Emperor saying "Good, good..." here]
A day without Facebook (Score:5, Interesting)
We need to do this more often. Can we do the entire week between Christmas and New Years?
Internet of Footgun Things (Score:2)
https://twitter.com/leahmcelra... [twitter.com]
Facebook's building access control system relies on the same DNS servers as everything else, so Facebook employees can't get access to buildings to work on this problem.
Perhaps that's why the people with the physical access to fix routing aren't the people who can fix routers with physical access.
I also predict that Internet Of Shit door locks will become less popular, replaced by door locks that may use the Internet but download the access list to the lock, like traditional
Their dns was within Facebook.com (Score:3)
Re: (Score:3)
Everyone keeps wanting to say this is a DNS problem just because their DNS is down. Nevermind the fact that every other Facebook owned IP address was down too. It wouldn't matter if you had a HOSTS file with manually curated entries for Facebook. You weren't going to load anything.
I see that Cloudflare's writeup (that you referenced below) didn't actually dig deep enough to see that Facebook's non-DNS IPs were also non-routable, but there were other better-written articles. I'm sort of surprised they pu
Re: (Score:3)
Re:Around 75% of al GBP hosts are inaccessable (Score:3, Interesting)
ooof, it didn't look like that when I submitted it...
You can watch updates live here easily, lots of activity atm
https://ris-live.ripe.net/ [ripe.net]
Re: (Score:3)
Well for a company the size of Facebook, this may not be too far from the truth.
The guy who knows how to fix the problem, probably isn't being called to help fix the problem. Because it is currently having Execs yelling at kids who just started. To fix the problem, while the older engineer who had encountered the problem, may had moved past the data center, and onto a different department.
They may not volunteer as they don't want to be that guy anymore, or they figure the problem is at hand and if they ge
Re: (Score:2)
or they figure the problem is at hand and if they get involved they will just be a hindrance.
Or if they work at many of the places I have, they want to avoid a combination of 'guilt by association and sitting in Staturday post mortem meetings for something they had nothing to do with or responsibility for other than knowing how to help.."
Re: (Score:3)
FB employees should quickly band together and universally agree it's Zuck's fault.
Re: (Score:3)
Re:Libtard Panic Attack? (Score:5, Informative)
Can you just imagine the libtard snowflake panic attacks happening right now with a huge chunk of the libtard echo chamber down?
Not really cos I can't see the conservative memes that keep claiming that until FB returns.
Re: (Score:3, Interesting)
But Twitter is still running? From what I've seen, most people don't like to express obviously wrong communist views with their real names, and therefore most of the left-wing echo chamber is in places like Twitter and on TikTok where they can talk about how they want to kill the rich and how really antifa are the good guys and cops are enemies of the people all without having to worry about anyone finding them by their actual name.
Re: (Score:2)
But Twitter is still running? From what I've seen, most people don't like to express obviously wrong fascist views with their real names, and therefore most of the right-wing echo chamber is in places like Parler and on Gab where they can talk about how they want to kill the homeless and how really all police are the good guys and antifa are enemies of the people all without having to worry about anyone finding them by their actual name.
I can do that too. It's almost like the extremes on both sides suck.
Re: (Score:3)
Apparently, Twitter is having bandwidth issues from all the people talking about the Facebook outage.