Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix (theregister.com) 28
Microsoft says last week's massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage's worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."
"The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive."
So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.
"The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive."
So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.
So incompetence on top of incompetence? (Score:5, Funny)
Makes sense, good old Microslop.
First, a 14% capacity reduction should never be enough to kill things, even with that Teams failover. They obviously went cheap on infrastructure.
Second, a cascade-failure is what you want to avoid at all cost and that means capacity, monitoring, robustness and tests. Apparently too expensive.
And third, don't they test their stuff under overload? Don't answer that, doing so would cost money.
All in all the usual mode of giving tech details to obscure that they screwed up massively. Again.
Re: So incompetence on top of incompetence? (Score:2)
Re:So incompetence on top of incompetence? (Score:5, Interesting)
And third, don't they test their stuff under overload?
This 6 part article [substack.com] from Axel Rietschin goes into great detail about the problems going at at Azure and why they are happening. It's a clusterfuck.
Re: (Score:2)
S.N.A.F.U. has entered the chat.
Multithreading and race conditions. (Score:5, Funny)
Re: (Score:2)
Understanding and designing concurrency is hard. Very few languages/programming systems (i.e. language plus libraries/frameworks) have a solid approach to that.
(Ada is the notable exception, but you can certainly mis-design concurrent systems in Ada, too. It's just easier to reason about them because of the rigorous language semantics.
Anecdote: Working on the POSIX Ada binding to the C Threads standard, our concurrency expert said, "There's a problem." He described it using Ada83 task semantics, and the
Good news (Score:2)
Microsoft finally load tested the July update on modern high-core hardware. They just used production to do it.
Re: (Score:3)
Turns out, testing in prod is the best testing money can buy!
Most Interesting Man in the World (Score:3)
https://i.redd.it/tjo65x3nres2... [i.redd.it]
Reminds me of the ATT long distance outage 1990 (Score:3)
Thunderbird (Score:1)
But once it's up and running, it works just fine and depends only on the various mail servers to which it connects.
Why use an overcomplicated monstrosity like Outlook? Thunderbird does the job perfectly well. You just need to learn how to tame it a little
Re: (Score:2)
Outlook, the desktop mail client that shipped with MS Office 1997 on, is kind of dead now. You have to fight with your Office configuration files to even install it on new systems. Microsoft intends for people to use the advertiser supported "Outlook" application that ships with Windows now. I've never liked the stand-alone Outlook client. Yes, it has all those scheduling, delegation and note-taking functions that some people need in their professional lives. It's also very delicate thanks to reliance on Je
Re: (Score:3)
Well as you noted, it depends on the various mail servers to which it connects and in this case, Exchange (and OWA) were affected. Client would not have made any difference here. I agree, Thunderbird is sufficient and Outlook is a jar of pickled feces. But who was singing Thunderbird's praises? I didn't see it mentioned in any of the linked articles.
AI? (Score:5, Funny)
why didn't they just use AI to fix it? why have humans spend 15 hours? AI would have done it in one second. I am Bender, Kill All Humans.
Re: (Score:1)
They did use AI in that July patch and in testing it. It said it was OK.
By any means necessary ... (Score:3)
They did use AI in that July patch and in testing it. It said it was OK.
It probably was that one particular test kept failing, the AI did not understand why, so it removed the test.
Seriously, they have done that. It met the goal to ship a patch by any means necessary.
Severely Lacking Root Cause Analysis (Score:2)
Not a peep about whether the changes that caused the failure were the result of LLM generated code/configuration changes, and whether human engineers did proper oversite and review of those changes.
I suppose it's easier to keep selling the random error generator slop bots, if you never look to see if the slop bots were the root cause of the problem.
Careful! (Score:4, Funny)
Re: (Score:3)
Thundering herd problem corollary (Score:3)
Re: (Score:2)
They didn't say why the Teams failover happened, though.
reminds me of a power grid "black start" (Score:5, Interesting)
Those are a pain because the power grid isn't star topology, it's web. This makes it difficult to isolate a few individual nodes to start up first. And once you DO get an "island" working again, you have to slowly add on other isolated segments. If you add too much at once, the whole thing comes crashing down and you have to start all over again.
It sounds like MS's "automated recovery" was set up to fail over to healthy nodes, which overloaded them and took them down too, just like you see when a power grid fails.
Everything is near max capacity when something trips, causing load to fail over to another path, which then overloads it, causing the entire combined larger load to fail over to another segment and quickly trip it, and they go down like dominos. Basically what happens if you fix one gateway and bring it back up, it gets mobbed and immediately "falls over".
Re: (Score:2)
Isolation isn't the problem. The grid can be isolated automatically and remotely down to very fine individual components. The problem is the co-ordination. It's not one company starting up or shutting down their equipment. It's an effort involving detailed planning and steps involving many companies both on the supply and the demand side of the equation, each of which can have their own localised issues to deal with (e.g. generator not synchronising properly).
You did touch on the biggest problem though brin
I Was Hoping For- (Score:2)
- a Send:All Subject:"Anyone want a free puppy" email to have been at the heart of the issue, as wave after wave of people kept telling everyone else to stop replying.
A clever ploy (Score:3)
Lemme get this straight. M$ is blaming M$ for introducing a software defect back in July, that surfaced only now. Then they take credit for rapidly fixing it.
Is this kind of behavior now the norm? Cause a screw-up, and then take credit for fixing it, while sidestepping any responsibility for the screw-up in the first place? It's the same thing that Trump and his ilk have been doing for years. Are we victims ever going to wise up and put a stop to this crap?