Forgot your password?
typodupeerror
Microsoft Network Software The Internet Windows

Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix (theregister.com) 28

Microsoft says last week's massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage's worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

"The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive."

So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.

Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix

Comments Filter:
  • by gweihir ( 88907 ) on Monday November 16, 2026 @11:39AM (#66426986)

    Makes sense, good old Microslop.

    First, a 14% capacity reduction should never be enough to kill things, even with that Teams failover. They obviously went cheap on infrastructure.
    Second, a cascade-failure is what you want to avoid at all cost and that means capacity, monitoring, robustness and tests. Apparently too expensive.
    And third, don't they test their stuff under overload? Don't answer that, doing so would cost money.

    All in all the usual mode of giving tech details to obscure that they screwed up massively. Again.

  • by Fly Swatter ( 30498 ) on Monday November 16, 2026 @11:40AM (#66426988) Homepage
    Rust can't fix that one.
    • Understanding and designing concurrency is hard. Very few languages/programming systems (i.e. language plus libraries/frameworks) have a solid approach to that.

      (Ada is the notable exception, but you can certainly mis-design concurrent systems in Ada, too. It's just easier to reason about them because of the rigorous language semantics.

      Anecdote: Working on the POSIX Ada binding to the C Threads standard, our concurrency expert said, "There's a problem." He described it using Ada83 task semantics, and the

  • Microsoft finally load tested the July update on modern high-core hardware. They just used production to do it.

  • by iamnotx0r ( 7683968 ) on Monday November 16, 2026 @11:46AM (#66426998)
    ATT went down for 9 hours, amazing how long this outage at Micorsoft lasted. ATT outage https://read.engineerscodex.co... [engineerscodex.com]
  • I wouldn't exactly sing Thunderbird's praises. Setting up a new account is still as buggy as ever and puts a lot of people off (and then you're presented with an insistent donation page, when all you really feel like doing is telling it to piss off...).
    But once it's up and running, it works just fine and depends only on the various mail servers to which it connects.
    Why use an overcomplicated monstrosity like Outlook? Thunderbird does the job perfectly well. You just need to learn how to tame it a little
    • by slaker ( 53818 )

      Outlook, the desktop mail client that shipped with MS Office 1997 on, is kind of dead now. You have to fight with your Office configuration files to even install it on new systems. Microsoft intends for people to use the advertiser supported "Outlook" application that ships with Windows now. I've never liked the stand-alone Outlook client. Yes, it has all those scheduling, delegation and note-taking functions that some people need in their professional lives. It's also very delicate thanks to reliance on Je

    • by Tarlus ( 1000874 )

      Well as you noted, it depends on the various mail servers to which it connects and in this case, Exchange (and OWA) were affected. Client would not have made any difference here. I agree, Thunderbird is sufficient and Outlook is a jar of pickled feces. But who was singing Thunderbird's praises? I didn't see it mentioned in any of the linked articles.

  • AI? (Score:5, Funny)

    by zlives ( 2009072 ) on Monday November 16, 2026 @11:59AM (#66427020)

    why didn't they just use AI to fix it? why have humans spend 15 hours? AI would have done it in one second. I am Bender, Kill All Humans.

    • by Anonymous Coward

      They did use AI in that July patch and in testing it. It said it was OK.

      • They did use AI in that July patch and in testing it. It said it was OK.

        It probably was that one particular test kept failing, the AI did not understand why, so it removed the test.

        Seriously, they have done that. It met the goal to ship a patch by any means necessary.

  • Not a peep about whether the changes that caused the failure were the result of LLM generated code/configuration changes, and whether human engineers did proper oversite and review of those changes.

    I suppose it's easier to keep selling the random error generator slop bots, if you never look to see if the slop bots were the root cause of the problem.

  • Careful! (Score:4, Funny)

    by Pig Hogger ( 10379 ) <(pig.hogger) (at) (gmail.com)> on Monday November 16, 2026 @12:05PM (#66427036) Journal
    Be very careful, with that headline, you may very well bring yourself a lawsuite from Monster Drinks....
  • by CommunityMember ( 6662188 ) on Monday November 16, 2026 @12:11PM (#66427052)
    Sometimes hard to see until the load actually hits, and then the health checks take systems out of service, and it only gets worse (at the scale of a Microsoft). It will be interesting to see exactly what they plan to do to avoid a repeat of the failures even as they fix the immediate lock contention issue.
  • by v1 ( 525388 ) on Monday November 16, 2026 @02:12PM (#66427316) Homepage Journal

    Those are a pain because the power grid isn't star topology, it's web. This makes it difficult to isolate a few individual nodes to start up first. And once you DO get an "island" working again, you have to slowly add on other isolated segments. If you add too much at once, the whole thing comes crashing down and you have to start all over again.

    It sounds like MS's "automated recovery" was set up to fail over to healthy nodes, which overloaded them and took them down too, just like you see when a power grid fails.

    Everything is near max capacity when something trips, causing load to fail over to another path, which then overloads it, causing the entire combined larger load to fail over to another segment and quickly trip it, and they go down like dominos. Basically what happens if you fix one gateway and bring it back up, it gets mobbed and immediately "falls over".

    • Isolation isn't the problem. The grid can be isolated automatically and remotely down to very fine individual components. The problem is the co-ordination. It's not one company starting up or shutting down their equipment. It's an effort involving detailed planning and steps involving many companies both on the supply and the demand side of the equation, each of which can have their own localised issues to deal with (e.g. generator not synchronising properly).

      You did touch on the biggest problem though brin

  • - a Send:All Subject:"Anyone want a free puppy" email to have been at the heart of the issue, as wave after wave of people kept telling everyone else to stop replying.

  • by J. L. Tympanum ( 39265 ) on Monday November 16, 2026 @03:14PM (#66427512)

    Lemme get this straight. M$ is blaming M$ for introducing a software defect back in July, that surfaced only now. Then they take credit for rapidly fixing it.

    Is this kind of behavior now the norm? Cause a screw-up, and then take credit for fixing it, while sidestepping any responsibility for the screw-up in the first place? It's the same thing that Trump and his ilk have been doing for years. Are we victims ever going to wise up and put a stop to this crap?

If God had not given us sticky tape, it would have been necessary to invent it.

Working...