Sysadmin Shock As Windows Server 2025 Installs Itself After Update Labeling Error (theregister.com) 86
A security update mislabeling by Microsoft led to Windows Server 2022 systems unexpectedly upgrading to Windows Server 2025, impacting 7 percent of Heimdal customers and leaving administrators scrambling to manage unexpected licensing and configuration challenges. The Register reports: It took Heimdal a while to trace the problem. According to a post on Reddit: "Due to the limited initial footprint, identifying the root cause took some time. By 18:05 UTC, we traced the issue to the Windows Update API, where Microsoft had mistakenly labeled the Windows Server 2025 upgrade as KB5044284." It added: "Our team discovered this discrepancy in our patching repository, as the GUID for the Windows Server 2025 upgrade does not match the usual entries for KB5044284 associated with Windows 11. This appears to be an error on Microsoft's side, affecting both the speed of release and the classification of the update. After cross-checking with Microsoft's KB repository, we confirmed that the KB number indeed references Windows 11, not Windows Server 2025."
As of last night, Heimdal estimated that the unexpected upgrade had affected 7 percent of customers -- it said it had blocked KB5044284 across all server group policies. However, this is of little comfort to administrators finding themselves receiving an unexpected upgrade. Since rolling back to the previous configuration will present a challenge, affected users will be faced with finding out just how effective their backup strategy is or paying for the required license and dealing with all the changes that come with Windows Server 2025.
As of last night, Heimdal estimated that the unexpected upgrade had affected 7 percent of customers -- it said it had blocked KB5044284 across all server group policies. However, this is of little comfort to administrators finding themselves receiving an unexpected upgrade. Since rolling back to the previous configuration will present a challenge, affected users will be faced with finding out just how effective their backup strategy is or paying for the required license and dealing with all the changes that come with Windows Server 2025.
And this is why people are reluctant to update (Score:5, Insightful)
There are far too many cases where fixes do far more than just fix things.
This case appears to be an mistake on Microsoft's part, but it's not at all uncommon for fixes to deliberately change behaviour.
Debian tries very hard not to update versions in a release and backport security fixes to the old version. But almost no other vendor seems to behave like that.
Re:And this is why people are reluctant to update (Score:5, Insightful)
Calm down with the Debian cheerleading - you're describing the entire concept of "Enterprise Linux." All of them do that.
Re: (Score:1)
Redhat also does the backporting thing. IME they have even older versions than Debian in some cases, although it's been a while since I was dumb enough to mess with a rpm-based distribution, and certainly don't want to do business with enemies of the GPL.
Re: (Score:2)
Re: (Score:2)
Re:Typical windows sysadmins! (Score:5, Insightful)
This is even more of a sin after the SolarWinds debacle.
They saw everything that happened, probably were impacted by it, and still learned nothing.
Re: (Score:2)
Re:Typical windows sysadmins! (Score:5, Informative)
Often you don't get a choice. Certain security requirements/ certifications REQUIRE updates to be pushed to all machines within 2 weeks of release to all machines. There's little time for any testing on that, especially server-side. These can be really basic requirements for things that all companies have and that insurers, cybersecurity accreditations, etc. insist upon.
Then you have the difficulty of managing and blocking updates - WSUS is no longer being developed (and goes away in Server 2025). Intune, Autopatch etc. are extra monthly licences and can be a pain to manage en-masse. Intune doesn't even let you block individual updates last time I looked.
There are third-party patch management solutions out there precisely because the Windows ones can be so dire, but even they can't necessarily see this kind of thing and stop it in time.
And rolling back an entire OS upgrade that's mistakenly marked as an update is far bigger a problem than just rolling back a single Windows KB number, and likely requires restoration from snapshot / backup which means downtime and THEN scrambling to stop it updating that update, same as the above, before it decides to do it itself. For every server. In companies that have dozens or hundreds of virtual machines.
Hell, if you've ever managed a network, you'll have seen single individual KBs that blue-screen and put the device into modes where no remote recovery is possible and you have to restore from backup or safe-mode them to remove them. If you haven't seen that, I question what you've been managing and how long.
Windows updating is really awful for modern times. Don't even get me started on CAU and/or deploying non-CAU updates on clustered servers.
Re:Typical windows sysadmins! (Score:5, Informative)
Re: (Score:2)
It's even worse. Certain security requirements/ certifications require updates to be pushed to all machines within 2 weeks of release to all machines. EVEN IF if the update is 100% broken and results in regular bluescreens.
Re:Typical windows sysadmins! (Score:4, Insightful)
It's even worse. Certain security requirements/ certifications require updates to be pushed to all machines within 2 weeks of release to all machines. EVEN IF if the update is 100% broken and results in regular bluescreens.
Looks like who wrote those requirements or mandated that kind of certification needs to review that or be fired
Re: (Score:2)
No one got fired for promoting Microsoft as the blameless company that can do no wrong. More likely those people get promoted so that they can continue screwing things up.
Re: (Score:1)
Lucky you! I work with a set of security controls that sets a 48-hour patching deadline for a lot of computers.
I lock the security weasels in a cage with the aviation safety twinkies, who insist that every change must be rigorously assessed for safety and correctness before being made and that the system can only have unplanned outages once every 23 days (roughly) and planned outages have to be announced at least 72 hours in advance. I listen to whichever one is alive after eight hours if compliance argum
Re:Typical windows sysadmins! (Score:5, Informative)
and goes away in Server 2025
It does not. It is depreciated, not removed. WSUS will keep working for longer than many on Slashdot will remain in the industry which includes the entire Windows 2025 support period.
Here's the relevant quote directly from Microsoft's own website:
Deprecated features continue to work and are fully supported until they are officially removed, and we have no current plans of removing WSUS from in-market versions of Windows Server (including Windows Server 2025).
Re:Typical windows sysadmins! (Score:4, Interesting)
Those are typical windows "sysadmins"! No machine dedicated to test the updates before applying those updates to prod servers? No snapshots on the SAN so they can quickly roll-back? Typical!
Often you don't get a choice. Certain security requirements/ certifications REQUIRE updates to be pushed to all machines within 2 weeks of release to all machines. There's little time for any testing on that, especially server-side. ...
How did you get modded 5, Informative???
You note that you have 2 weeks to push out the updates. Are you trying to tell us that you couldn't update one test system before the two weeks was up? That's all this would take.
And rolling back an entire OS upgrade that's mistakenly marked as an update is far bigger a problem than just rolling back a single Windows KB number, and likely requires restoration from snapshot / backup which means downtime and THEN scrambling to stop it updating that update, same as the above, before it decides to do it itself. For every server. In companies that have dozens or hundreds of virtual machines.
You already have downtime planned because you're applying a set of OS updates. FYI, rolling back a snapshot is quick. While you're doing it, setup that test system you neglected to test on so you can identify what's causing the problem (NOTE: they ended up having to do this anyway), block the update causing the problem, retest then, if ok, deploy to your first subset of servers.
Hell, if you've ever managed a network, you'll have seen single individual KBs that blue-screen and put the device into modes where no remote recovery is possible and you have to restore from backup or safe-mode them to remove them. If you haven't seen that, I question what you've been managing and how long.
So you've lived through that, but you still choose not to test updates before applying them in production?
Am I even on slashdot? Where are the actual sysadmins?
Re: (Score:3)
Why would one test system show you anything at all?
This would have to be one test 2022 system, and I guarantee you people still have 2019 out there en-masse and maybe 2016 (still supported until 2027!).
Then that presumes this update hits immediately and you'll notice the problem instantly (which may be true in this very unusual case) and isn't dependent on what software, services, options, etc. you have installed, whether it was previously in-place upgraded or fresh install, what apps you're using, etc. etc
Re:Typical windows sysadmins! (Score:4, Informative)
Why would one test system show you anything at all?
Read TFS. Installing this one KB update on a test system would have clearly shown the issue - it would get upgraded to server 2025.
Sorry, but I don't think you get that not everyone has 10,000 machines, a staff of 50, and all the time in the world ...
It's 2024. You don't need to own a single server to test this. Use a VM in the cloud: ... it would fail that, so freeze the release, roll back the VM snapshot, and alert someone that this needs more manually reviewed first.
* snapshot the VM
* turn it on
* apply updates
* do whatever reboots and such are needed
* confirm it's ok.
*
You don't need 50 people to do this. Not even 1 FTE. Do this once a week, allowing for the rest of the week for the updates to roll out.
Lastly? This was managed by Heimdal - "One platform. Total Cyber Security". They DO have lots of employees and servers, and managing updates is a core to their role. This isn't a small shop with no full time sysadmins. Is there a reason you're defending these bad practices?
Re: (Score:2, Insightful)
You note that you have 2 weeks to push out the updates. Are you trying to tell us that you couldn't update one test system before the two weeks was up? That's all this would take.
A proper security audit would take far longer than that. Especially for anything where code integrity and behavior actually matters beyond a signature check and validation is required. You can't keep out exploits if you're just randomly installing whatever signed agile garbage the developer's github CI squeezed out five minutes ago.
So you've lived through that, but you still choose not to test updates before applying them in production? Am I even on slashdot? Where are the actual sysadmins?
Not everyone has the money in the budget to maintain a fleet of test systems, or have out-of-band management properly set up and secured so it can be used in these cases. Never
Re: (Score:2)
You note that you have 2 weeks to push out the updates. Are you trying to tell us that you couldn't update one test system before the two weeks was up? That's all this would take.
A proper security audit would take far longer than that. Especially for anything where code integrity and behavior actually matters beyond a signature check and validation is required. You can't keep out exploits if you're just randomly installing whatever signed agile garbage the developer's github CI squeezed out five minutes ago.
What are you saying? That a proper security audit would take too long, so let's not do anything? Just try them on a test system first! You can even use a throw away VM in the cloud - nearly zero cost. And it would have caught this issue, as well as some of the other immediate glaring problem that have happened in the past.
So you've lived through that, but you still choose not to test updates before applying them in production?
Am I even on slashdot? Where are the actual sysadmins?
Not everyone has the money in the budget to maintain a fleet of test systems, or have out-of-band management properly set up and secured so it can be used in these cases.
What year is this? Use a VM. You can even use a throw away one in the cloud.
But let's come back to the actual case here - isn't Heimdal being used so that they can take care of the securit
Re: (Score:2)
should be patching the test bed first, why wasn't this caught predeploy?
Re:Typical windows sysadmins! (Score:5, Insightful)
A lot of what is called security is simply monster creations of bureaucracy by people who can't do anything productive. It's not Jason Bourne defeating the sophisticated powerful players, it's the "everyone take off your shoes" queue at the airport.
Re: (Score:2)
Gotta prove you're important and that you make a difference, and making these sorts of mandates can make that happen!
Re: (Score:2)
, it's the "everyone take off your shoes" queue at the airport.
But we all know that actually works, because the ones with the stinkiest feet will be the terrorists. They're the ones trying to sneak biological weapons on board.
Re: (Score:2)
Often you don't get a choice. Certain security requirements/ certifications REQUIRE updates to be pushed to all machines within 2 weeks of release to all machines. There's little time for any testing on that, especially server-side. These can be really basic requirements for things that all companies have and that insurers, cybersecurity accreditations, etc. insist upon.
Because rushed security practices never produce bad outcomes. Hang on I need to install this latest upda.....3#$#W.s...We've been trying to contact you about your car's extended warranty....
Re: (Score:2)
WSUS still very much works in Server 2025. It's now marked as "deprecated" which means they're no longer changing/updating it (not that they've really done that since 2012R2 anyway) and also "don't be surprised if it's not in the _next_ server release.
Even if they don't keep publishing catalog updates until the 2034 EOL of Server 2025, I would be shocked if they cut them off before the 2031 EOL of Server 2022 (given that under Server 2022 it's still very much supported).
Re: (Score:2)
If it's marked as deprecated, it won't pass a cybersecurity audit.
Re: (Score:2)
My lived experience is that clustering *reduces* availability as often as not. Though perhaps I've just had a bad run.
Re: (Score:2)
It's not so bad if you have the storage.
If you try to cheap out and use Storage Spaces Direct? Or if you try to run it on just a small handful of nodes? Yeah, just throw it in the bin and run VMs individually.
I've seen IT consultants recommend 2-node clusters running S2D to a several $m business. That's just a recipe for disaster.
Re: (Score:2)
Re: (Score:1)
Well, it's almost 2025. I didn't have that flexibility and versatility in 2000 either but now it's almost a standard nowadays even for really small setups where most things are virtualized anyway. Are we going back in time or something?
Re: (Score:2)
I would answer your question, but it's taking a while to check. Apparently, virtualizing my web browser by logging into a virtual machine running off a virtual disk array in a virtual cluster is a literal cluster---- in terms of speed. If I'm ever able to log in and roll back to 2000, I'll let you know whether time travel is possible.
Re: (Score:3)
No one with such budget constraints can afford the budget to deal with all the issues when they don't test things. Do a little work upfront, or a lot of work when things blow up.
If it's your job to manage server updates, then it's your job to do it properly, and to ensure leadership is aware of the situation on the ground.
If you're just a grunt hired to push some buttons, and it's someone else's job to manage the update process, and you've informed them about your concerns and they told you to just click th
Re: (Score:2)
If you get the upgrade for free, why would you need to pay for the license? If it's Microsoft's fault, and Microsoft wants the licensing fees, then they can send staff out on their own dime to do a proper restoration.
Error? (Score:5, Insightful)
paying for the required license
If it's microsoft's error they should provide this for free, otherwise it's basically extortion (we screwed you, now pay up or your data is toast).
Re: (Score:3)
paying for the required license
If it's microsoft's error they should provide this for free, otherwise it's basically extortion (we screwed you, now pay up or your data is toast).
Exactly. This was an upgrade forced by the vendor. Either they can give the license away, or they can pay for all damages incurred.
Re: (Score:1)
Or at the very least microsoft should call and negotiate terms with relevant compensation.
Re: (Score:2)
paying for the required license
If it's microsoft's error they should provide this for free, otherwise it's basically extortion (we screwed you, now pay up or your data is toast).
It's just the way business is done now. Fuck up? Charge the customer. I hope they get their asses sued off for it, but it seems like nothing, no matter how nefarious and evil on the surface, every sticks to them these days.
'Helping IT' (Score:5, Insightful)
So after all the issues at Fortinet, Crowdstrike, SolarWinds, Palo Alto, is it really worthwhile to use this kind of 'helpful' software? Seems to be causing more trouble than it's worth.
Re: (Score:3)
What is more trouble than it's worth? Security? You should see what trouble you get into when you don't have any at all. Big news this year: A couple of incompetent companies lost a lot of money, for everyone else IT admins had a very bad week. That's it. That's completely small irrelevant crap compared to suffering a major data breach or being infected by ransomware, or having internal development documents / IP being sold online.
Security is a lot less cost and effort than not having security. It's like an
Re: (Score:2)
Re: (Score:2)
This is news? (Score:1)
Every week there's another story about Microsoft screwing over its customers and/or user base. At what point do we stop posting these stories since these are a regular part of life, their incompetence something we'll just have to live with?
Re: (Score:2)
Maybe.. (Score:4, Interesting)
Re: (Score:2)
> MS ip address range
How do you keep up with this with all the Azure v4 acquisitions?
Re: (Score:3)
You can update periodically from here [microsoft.com].
Re: (Score:2)
Microsoft has been (Score:4, Funny)
Re: (Score:2)
Same has been said about Windows Vista.
Didn't happen then, probably won't happen now...
Re: (Score:2)
To put this in Linux terms, imagine your favorite distro just dropped a new server release and some third-party script you use for making sure upgrades get installed misread the instructions and helpfully performed a dist-upgrade for you.
Would that be your distro's fault?
Microsoft is always slip-shod and tasteless (Score:2)
Blame the user (Score:2)
I think, it's the third time this year, someone else is responsible for Microsoft's neglect and dirty deeds. When the don't-install-Chrome debacle was reported, the journalist failed to hold Microsoft responsible for their actions. When Recall Snapshots was announced, it was reported as "creepy", without investigating its built-in security and privacy. (Originally, there was none.)
We now have tech reporters playing favourites.
Ads? (Score:1)
Do they at least get Start menu ads as part of the Downtime Enshittification Insecurity
initiative?
Bug in Heimdal Asset Management Module (Score:2)
Re: (Score:3)
Both are to blame. Microsoft for not testing before release and Heimdal for not testing before release.
I sense a pattern forming.....
Re: (Score:2)
Re: (Score:2)
If the information has no value then why is it there at all?
Good code is written on the assumption that users are a bunch of monkeys pounding randomly on the keyboard yet still does something rational in response, not fucking up their servers.
And Jfc why is no one at MS checking this metadata anyway? Jfc, seriously, some intern enters this shit by hand while on the vpn from the bar? The entire process of release should be automated.
How did they not notice? (Score:2)
I think I would have been a bit suspicious of a 5GB+ download (the smallest Windows Server 2022 upgrade I could find) for a "security update". This is roughly 7x larger than the largest Windows Server 2022 update, and 12x larger than any update for about 2 years.
I know disk space is cheap, but even if this was my home computer I would be wary of an update that is large enough to replace the entire OS.
Re: (Score:3)
Do you constantly keep an eye on bandwidth usage? On my 300Mbps connection 5gb would take a few minutes.
Re: (Score:3)
Security update indeed. (Score:4, Funny)
The security update is a whole new operating system?
Somehow that seems appropriate when discussing Microsoft.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Indeed. The real question is "Is it an update or a downgrade?" though and whether it actually fixes any security problems or at least more than it introduces.
License? Nah it's free (Score:2)
Needed another "L" added to the name. (Score:2)
"impacting 7 percent of Heimdal customers."
Heimdall (two L's) would have seen this coming.
why I'm well paid (Score:2)
And people wonder why tech people are well paid. We are there with the right skills at the right time when vendors, and hardware, and people, shit the bed. Always standing by to wipe technical ass.
Re: (Score:2)
Sad but true. To be fair, MS incompetence is a major job creator for IT and IT Security experts. We should be grateful to them. Well, maybe.
KB5044284 was not a stealth upgrade (Score:4, Interesting)
Choosing OPTION-3 was a very bad choice and has little to do with Microsoft's metadata error.
Re: (Score:2)
For decades, people have been telling Microsoft that security updates and feature updates should be separated.
This is the cost of not having them separated. But, I am sure the end result is still acceptable to Microsoft as long as their goals are achieved. Your goals are utterly irrelevant to Microsoft. They are too big to fail.
MS crapware (Score:2)
Always good for surprises. Just never any good surprises. You would think they are a bit more careful with their corporate customers, but apparently not.
Did I get this right? (Score:2)
MS updates system A to system B without user interaction, and then asks money for system B's license? How unscrupulous can they get? At least when they were updating Windows 7/8 to Windows 10, the old licenses remained valid. I guess they need more money now.
Disk space (Score:2)
Testing? (Score:2)
We really need to start hiring and using engineers (Score:1)