Microsoft's Global Outage May Be Worse Than Amazon's Or Crowdstrike's, Expert Says (arstechnica.com) 46
An anonymous reader quotes a report from Ars Technica: A global outage taking down a wide range of Microsoft 365 services remains ongoing Friday morning. The outage started midday Thursday, impacting users of Microsoft email, cloud, and AI services. Microsoft has documented the full list of services impacted by the error, while moving early Friday morning to protect customers around the world from losing emails during the extended disruptions. "Recovery actions are ongoing," the Microsoft 365 Status account confirmed on X. "To help preserve delayed mail during this event, we've extended email timeout to 48 hours."
Similar to recent world-shaking outages of widely used tech like Amazon Web Services and CrowdStrike, the Microsoft disruption looks like "one of the biggest outages of all time," Ciaran Martin, the former head of the UK's National Cyber Security Centre, told NBC News. Possibly even "worse," he suggested.
Businesses, hospitals, police departments, and schools were among those scrambling to work around the outage. In Japan, it disrupted government IT infrastructure, according to deputy chief cabinet secretary Masanao Ozaki. NBC's report documented various harms, noting that "few public bodies were immune from the outage." In some cases, doctors couldn't send patients "important" updates on medication changes or follow-up care. The financial sector suffered, too, with people blocked from making bank transfers since they couldn't receive verification codes.
Justice systems were also hobbled. The Supreme Court in the Philippines saw its legal work derailed, NBC reported. And one US sheriff's office in Colorado reported that the outage made it difficult for people to post bond for prisoners' immediate release. It's unclear how many users were affected, but Microsoft estimated in January that the total number of paid Microsoft 365 commercial subscriptions exceeded 450 million. Microsoft's most recent updates confirmed that it's seeing "broader improvements in North America, sustained improvements across much of Europe, and encouraging recovery signals in Asia Pacific."
"Some users may still experience intermittent access issues or delayed email delivery as mitigation continues across remaining affected infrastructure," Microsoft said. "Our focus remains on stabilizing recovered infrastructure, completing mitigation in remaining impacted areas, and reducing delayed email queues and backlogs while we validate service consistency across connection methods."
Similar to recent world-shaking outages of widely used tech like Amazon Web Services and CrowdStrike, the Microsoft disruption looks like "one of the biggest outages of all time," Ciaran Martin, the former head of the UK's National Cyber Security Centre, told NBC News. Possibly even "worse," he suggested.
Businesses, hospitals, police departments, and schools were among those scrambling to work around the outage. In Japan, it disrupted government IT infrastructure, according to deputy chief cabinet secretary Masanao Ozaki. NBC's report documented various harms, noting that "few public bodies were immune from the outage." In some cases, doctors couldn't send patients "important" updates on medication changes or follow-up care. The financial sector suffered, too, with people blocked from making bank transfers since they couldn't receive verification codes.
Justice systems were also hobbled. The Supreme Court in the Philippines saw its legal work derailed, NBC reported. And one US sheriff's office in Colorado reported that the outage made it difficult for people to post bond for prisoners' immediate release. It's unclear how many users were affected, but Microsoft estimated in January that the total number of paid Microsoft 365 commercial subscriptions exceeded 450 million. Microsoft's most recent updates confirmed that it's seeing "broader improvements in North America, sustained improvements across much of Europe, and encouraging recovery signals in Asia Pacific."
"Some users may still experience intermittent access issues or delayed email delivery as mitigation continues across remaining affected infrastructure," Microsoft said. "Our focus remains on stabilizing recovered infrastructure, completing mitigation in remaining impacted areas, and reducing delayed email queues and backlogs while we validate service consistency across connection methods."
telemetry (Score:5, Insightful)
Failed Health checks
They say it wasn't AI. Something tells me it was AI code.
Re:telemetry [cries me a river] (Score:5, Interesting)
Me thinks me smells a cyber-attack that Microsoft (and f[r]iends) wants to cover up. In such a case I think it safe to wager an AI helped create the attack. Probably found the vulnerability, too.
<Rubs thumb against first finger> It's the world's smallest fiddle playing a sad song just for Microsoft.
<Rubs both thumbs against both first fingers> It's the world's smallest orchestra playing an especially sad song for Amazon.
Re: (Score:2, Insightful)
Extreme incompetence or them getting hacked again? Well. Usually I would say these people just cannot do things right, but them getting attacked would also be an instance of that, so...
Re: telemetry [cries me a river] (Score:2)
Tuesday this week was patch Tuesday, and if it was a bad patch causing the issues it wouldn't be the first time.
It seems to me that problems created by bad patches have become worse over time. It's one thing that a patch won't install and gives a fault code. But another if it degrades functionality, especially on this level.
Now it's time for corporations to consider if the Azure/Entra train shall be abandoned.
AWS had an opps (Score:2)
https://aws.amazon.com/message... [amazon.com]
Re: (Score:2, Interesting)
The real reason is that these health checks can take down enough servers that the remaining ones get overwhelmed. This can only be called gross incompetence.
Re: (Score:3)
They are claiming some kind of network configuration error. I'm dying to read the RCA.
Re: (Score:2)
You'd think that you'd make your healthcheck systems scale proportionately to the number of systems you've got last reporting active. That seems like a good thing to do.
Personally, I'm not sure this would result in the remaining systems getting healthchecked more often, though. How would that work?
Let's say: you've got 1m servers and (say) 24 healthcheck system. They each send out one service API probe every, oh, 15 seconds, for something on the severely high side of things, with a 3 second timeout. If you
Re: (Score:2)
Failed Health checks
They say it wasn't AI. Something tells me it was AI code.
It was not AI, it was SI !
Re: (Score:3)
Yes, I too blame Sports Illustrated!
Lousy reporting (Score:3)
Re: (Score:2)
To everyone using cloud services (Score:2, Insightful)
May you get all you wish for.... .
Re: (Score:2)
I'd rather get what I pay for, but that's expecting a lot.
You now what ... (Score:1)
I thought that the WordPress issues would have been way worse than this. Microsoft, now touting themselves as an ISP, surely they know how to run the collection of domains they have more than the rabble of content creators running WordPress.
But - WordPress and PHP self-updates much better, and surprisingly, keeps the internet connected better than Microsoft does with all their staff.
Does this mean then, that you're better off self-hosting your domain's email/web?
Re: (Score:2)
But - WordPress and PHP self-updates much better, and surprisingly, keeps the internet connected better than Microsoft does with all their staff.
Does this mean then, that...
I have no idea what your sentence means! PHP self-updates?!? WordPress keeps the internet connected?!?! What???
Does this mean ... you're better off self-hosting your domain's email/web?
No. In most cases, you're better off using a reliable hosting provider if you're just after hosting your own website and email.
I tend to be overly critical of Microsoft. However, it may be worth noting their reliability numbers for email and generic website hosting tend to be quite good. The problem here is that they are so pervasive that even a short outage will cause widespread and far reaching i
Re: (Score:2)
I have no idea what your sentence means! PHP self-updates?!? WordPress keeps the internet connected?!?! What???
You know exactly what I mean, default package updates and "connected" in the sense of publishing and your clients hitting your site with their browsers.
... if you fall into that category, then you already know you're up to the task. Many other hosting providers may have provably worse uptime numbers compared to Microsoft, but the upside is that you won't get caught up in this sort of global mess when it strikes again (assuming you picked a provider wisely).
I don't think it's like it used to be where you needed to know how to partition the disk. There's tons of appliance style installs now.
Arguably, even a click to install style application from Azure could cope better than the outlook situation of today, I'm not sure that you'd get much from that other than a little knowledge to help on the way out.
Re: (Score:2)
The most complicated thing with hosting your own stuff now is domain registration and DNS setup. For email specifically, that can be quite confusing.
Re: (Score:1)
What the bosses and bean counters failed to see coming is the ine
The worst part (Score:3)
The worst part is that we still have no clue what communications or volumes thereof have been missed. I never got NDRs from my tests sent out more than 24 hours ago, but those same tests have not (yet) reached their destination. I have no way of knowing if they are caught in a giant spool somewhere or if they truly went poof into the ether.
Re: The worst part (Score:2)
IME spool. But YMMV
Dave Plummer has his spin. (Score:1)
https://www.youtube.com/watch?... [youtube.com]
Put everything in the cloud they said... (Score:2)
Maybe... just maybe... running your own compute isn't such a bad idea.
Overtly critical (Score:2)
They just used it wrong but they've learnt their lesson now.
It will get worse (Score:2)
Nothing was changed and people did not learn from the previous outages. It will get worse until the mind-set and approaches get fundamentally changed from "cheapest possible" to "reliability first".
Re: It will get worse (Score:2)
No, that's not how this works. People go to azure because they want outlook 365 for whatever reason. They don't think about uptime, they just get comfortable with clicking on the big e to get on the Internet. If the e changes to a stylized circle, they'll get lost, they call IT and ask where it is.
Now think about asking them to switch from Outlook to virtually anything else. Before you even get to the second migration wave, others are already demanding to go back. Then you get political pressure to go back.
Dusting off my login here to name this one... (Score:3, Interesting)
Azur-pocalypse
Re: (Score:3)
Nice. And how about some incident headlines?
1. Microsoft: "I've Fallen and I Can't Get Up"
2. All the CEO's AI and All the CEO's Men / Couldn't Put Microsoft Together Again
Big lesson, maybe some will listen. (Score:2, Insightful)
Microsoft failure should not cripple entire governments or judicial systems.
But no one will learn and the House of Cards will continue to wobble as it grows.
Stop dressing it up (Score:1)
Microsoft has lost all trust, they lost it decades ago, and have only continued the march of embracement, and incompetence. They carry that badge in
Re: Stop dressing it up (Score:3)
They haven't lost all trust, though. Most orgs will learn nothing and change nothing.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Sad as that is, yes. But this is a step to make regulation more likely, even if only in Europe.
Exactly. (Score:2)
The predilection to outsource is something close to a religious edict - there are folks I work with who treat a suggestion we build something in-house when there's something vaguely similarly shaped on the market as if I just asked to fuck their spouse. It is much closer to a moral tenet than anythin
Re: (Score:2)
Aside from some "Office 363" jokes
We had that joke today on Teams.
Other employee: Looks more like Office 364
Me: I think it's more like Office 363 now
Assorted smiley reaccs ensue.
Some entity will need to lose serious money before anything will happen. And when something does, it will be regulated lock-in, not a return to a diverse email system.
Either thing would be better than this. This is fucking chaos.
FAA ground stop ... (Score:2)
... just announced. For all US airspace and possibly Canada as well.
I didn't think Microsoft ever removed license language prohibiting the use of it's products in safety critical applications.
Re: (Score:2)
Canada is apparently not affected at least that is the reporting here in Europe.
But this may well be some indirect dependency on Microslop systems. For a direct dependency (which would be illegal) the shutdown would already have happened a day earlier, hence I guess that is not it. But IT has gotten way too complex and unintended consequences wayyyyy to likely. My bet is we are seeing one here.
Re: (Score:2, Insightful)
Oh, and now the White House claims no connection to Microslop. With the current probability of them lying, I would call that conformation.
Re: (Score:3)
The outage appears to be affecting a telecommunications network overseen by L3Harris Technologies, according to people briefed on the matter.
Re: (Score:1)
Sure. Not a direct dependency. But if it turns out L3Harris is dependent on O364 (with some delay), that would make it an indirect one.
Re: (Score:2)
Something bigger might be going on here. For the past day and a half my AT&T wireless service has been up and down, down mostly today, with AT&T emailing me about the issue and sneaking an update text in now and then only for it to go back down again. Combine this with the Microslop outage and now the FAA outage. Didn't the FAA just recently report they were going all in on AI helping out? Yeah, that's brilliant.
The cloud is not your friend.
Re: (Score:2)
The cloud is not your friend.
Yes. Very much so. And all the large tech enterprises are not either.
Re: (Score:2)
What, too early?