Microsoft Releases Recovery Tool for Windows Machines Hit By Crowdstrike Issue (theverge.com) 115
The Verge reports that for machines that aren't automatically receiving Crowdstrike's newly-released software fix, Microsoft has released a recovery tool that creates a bootable USB drive.
Some IT admins have reported rebooting PCs multiple times will get the necessary update, but for others the only route is having to manually boot into Safe Mode and deleting the problematic CrowdStrike update file. Microsoft's recovery tool now makes this recovery process less manual, by booting into its Windows PE environment via USB, accessing the disk of the affected machine, and automatically deleting the problematic CrowdStrike file to allow the machine to boot properly. This avoids having to boot into Safe Mode or a requirement of admin rights on the machine, because the tool is simply accessing the disk without booting into the local copy of Windows. If a disk is protected by BitLocker encryption, the tool will prompt for the BitLocker recovery key and then continue to fix the CrowdStrike update.
Why does all of this look so half-assed? (Score:1, Troll)
Windows is truly the system for amateurs.
Re:Why does all of this look so half-assed? (Score:4, Informative)
https://www.neowin.net/news/cr... [neowin.net]
Re: (Score:2)
It took them 3 days to release a bootable USB?
Well, I already have a bootable USB, a bootable DVD and even a bootable CD for debian and it works on all other distros. Boot, mount root filesystem, chroot, delete file and or fix what's wrong and problem solved.
Re: (Score:2)
Or with an hour or so more of work, all of these steps automatized. Also works for VMs if you can select boot medium.
Re: (Score:3)
Yes and after posting above, I just realized that I could probably a linux rescue usb to fix that windows problem as well and as you said, this could be easily automated:
1) Boot linux rescue
2) Search for windows partitions
3) mount them
4) search for specific file named previously (crowdstrike file)
5) delete file as per instructions
chroot isn't even needed for that fix.
Could be more complex if drive is encrypted with bitlocker or what not, I just checked and linux seems to support it:
https://itsfoss.com/mount [itsfoss.com]
Re: (Score:2)
1) Boot linux rescue
Problem: Secure boot is often deployed on Windows systems and will prevent you from booting that Linux Rescue stack, since it's not signed by Microsoft. Adding a step to turn that off would cause some problems and vulnerability.
Re: (Score:2)
I suspect there should be ways around it but I can't tell for sure, can you? It might involve turning secure boot off temporarily so rootkit would have to be already installed before you boot. As I said, not sure, I'd have to have to do it to investigate more.
https://www.howtogeek.com/1756... [howtogeek.com]
Re: (Score:2)
All major distros are signed and they will boot.
Re: Why does all of this look so half-assed? (Score:2)
Re: (Score:3)
https://phoenixnap.com/kb/moun... [phoenixnap.com]
As well, secure boot doesn't seem to be a problem either as others have mentioned above.
Re: (Score:2)
> Can you mount ntfs shares withe read write capabilities?
Yes, both Linux has this capability. On Linux I've had some trouble getting the filesystem to mount r/w if the dirty bit is set on the ntfs volume, which it would be if it had just had a BSOD. You can force it to mount, but I'm not super comfortable with that.
With WinPE you can just ChkDsk the volume to fix the dirty bit.
Re: (Score:2)
Is that any easier than booting into safe mode in Windows and doing the same thing?
Re: (Score:2)
It's usually much easier to automate things with Linux compared to Windows and that's what the topic of the thread forked into.
Also, some say you need the Windows OS admin password to boot in safe mode in Windows. With Linux, I can access the Windows file system without any Windows OS password. I have done that for friends who forgot their Windows password or similar things like crashed OS, user don't care and just want his data files to put on his new computer, etc.
Re: (Score:2)
Why is it so "difficult" to boot into Safe Mode and delete a file? It's not rocket science.
Re: (Score:2)
admin password
Re: (Score:3)
Re: (Score:2)
Bitlocker recovery key, admin password and doing this manually. Remember this is Windows and hence rather primitive.
Re: (Score:2)
Now do it for a computer 500 miles away.
Re: Why does all of this look so half-assed? (Score:2)
Re: (Score:1)
Yeah, you missed the bit where you have to enter the 48 digit bitlocker recovery key, 100% correctly, assuming you even have access to it.
Re: (Score:2)
Doing it for one computer isn't difficult.
Doing it for 3000 computers also isn't difficult, just very time consuming, and if you can save 1 second on each one, that's 50 minutes across the entire fleet.
Re:Why does all of this look so half-assed? (Score:5, Insightful)
Windows is truly the system for amateurs.
Don't conflate the mistakes of a 3rd-party security vendor with the platform itself.
.conf files using vi or Emacs, and once you're done that you'd have to learn the source code or just blindly trust the half-dozen layers of third-party libraries haven't been silently backdoored.
If this was Linux, there would be four or five competing LiveCDs that may or may not contain the right storage drivers and you'd have to read through a thousand forum posts arguing about if you should edit its
There's a lot to be said for open source, but in this case it's reasonable to assume that the ISO Microsoft has released is trustworthy, functional, and trivial to use. Which really matters when you've got tens of thousands of computers bricked by add-on software.
Re: (Score:1)
"If this was Linux"
That's called whataboutery. It's not, it was and what a fuck up.
Re: (Score:3)
"If this was Linux"
That's called whataboutery. It's not, it was and what a fuck up.
Please. The post I was replying to is clearly meant to be comparative. As in "Windows is truly the system for amateurs as opposed to some other unnamed operating system.." Whataboutery is excusing the faults of one thing by pointing out the flaws in another. As it happens, in this particular instance - as I pointed out and you have not refuted - this incident wasn't about the platform. Kernel is as kernel does. Attributing this fuck-up to Windows is... mis-attribution.
My reply covers both points. F
Re: (Score:3)
That's an interesting claim, but it doesn't match my experience. I used to habitually switch between distributions, until I got bored with it. The only one that ever gave me any trouble was Fedora, and that was because they decided to encrypt the partition...something I never choose to do, so I didn't record the stuff I needed to access the partition. (Not a real problem, just annoying. I think encrypted partitions are a mistake. If you want to keep something secret, put it on an external drive, and hi
Re: (Score:2)
Encrypted partitions only protect a machine that is not running. Hence a very good thing for a laptop you travel with, a problem in most other circumstances.
Re: (Score:2)
Ironically, it would be nice to see more TPM support, easily available as a choice for installing. Ubuntu has it, but you are locked into a snap system and hardware stack, which means you are stuck with a default filesystem (I prefer btrfs for everything due to snapshots, and XFS isn't bad as well.) Encrypted drives, especially a mechanism for preventing an evil maid attack can be very useful. This ensures that someone can't change the initramfs, put in a Trojan to save the LUKS password (or entire encry
Re: (Score:1)
Evil Maid cannot really be prevented and hence most security people do not even try. The TPM primarily serves to lock out the user from things like circumventing DRM, not to protect the user.
Re: (Score:2)
The only one that ever gave me any trouble was Fedora, and that was because they decided to encrypt the partition...something I never choose to do,
Fedora won't encrypt the partition by itself, you didn't pay attention during install and left a box checked. Source: Fedora is on all my machines and partitions aren't encrypted.
Re: (Score:2)
Right. I didn't choose not to do the encryption, but it was the default option.
Re: (Score:2, Insightful)
Bullshit. Microsoft set the tone, the culture and provided the hooking possibilities and made an OS so abysmally bad that you need crap like Cloudstrike. Apologists like you them make the problem far worse.
Re: (Score:3)
To be fair, MS hand was forced by the EU. MS should have taken the route they did with Novell's NDS for NT, refusing to support installation of such software (SAM replacement/access) wouldn't be supported. Bit like when you turn off driver signing.
https://www.neowin.net/news/mi... [neowin.net]
Re: (Score:1)
Bullshit. That is just Microsoft trying to shift blame. That they feel the need to shift blame tells you everything about them.
Re: (Score:2)
It is Microsoft's fault that their OS is so bad that 3rd party kernel modules are even needed.
Re: Why does all of this look so half-assed? (Score:1)
Re: (Score:2)
Don't conflate the mistakes of a 3rd-party security vendor with the platform itself.
If the platform can be taken down so easily bu a 3rd-party tool, the platform is not so solid
If this was Linux, there would be four or five competing LiveCDs that may or may not contain the right storage drivers and you'd have to read through a thousand forum posts arguing about if you should edit its .conf files using vi or Emacs, and once you're done that you'd have to learn the source code or just blindly trust the half-dozen layers of third-party libraries haven't been silently backdoored.
On Linux you would use the Live image linked on your distro's website, no need to use a Fedora image on Debian, nor the other way around. And the said website would provide simple, easy to follow instructions.
Re: (Score:2)
If the issue hit a major Linux disto (a third party driver causing a kernel panic), you'd just need to edit the kernel command line in your boot-loader to not load the problematic kernel driver, run updates and reboot.
Re: (Score:3)
Tomorrow (Monday), I will be wearing my: "Have you thought about turning it off and on again?" T shirt.
Re: (Score:2)
Yep. Of course the massive mountain of irony will be wasted of the idiots behind this disaster.
Re: (Score:2)
Less Manual? (Score:5, Interesting)
You still have to sit at each computer and manually boot it to the USB drive. This may speed up the recovery, but it's not actually saving Windows admins from having to physically touch every computer, one way or the other.
Re: (Score:2)
You don't have to type any commands, maybe just a bitlocker key.
That's a small improvement in that you can mail this to a remote user.
Re: Less Manual? (Score:1)
On the other hand, every sysadmin half worth their salt already did that on Friday morning.
Re: (Score:2)
A lot of organizations are woefully understaffed, there are just not enough people in the department to get the job done.
And then, again, there's the remote staff. This way you can UPS them a flash drive and USPS them the bitlocker key...
Re: (Score:2)
On the other hand, every sysadmin half worth their salt already did that on Friday morning.
No. They did that for their core servers. I know you don't understand what it means for a company to have more than 10 computers, but there are companies out there with thousands of employees many working from home no where near an office, some travelling, some super unlucky to be at airports on Friday.
Even if your IT department is staffed by Santa Clause himself he wouldn't have gifted an even medium sized company a completely working solution by Friday morning. They would have done the bare minimum to get
Re: Less Manual? (Score:1)
Except I have a network of 2k assets to manage with about 10 people. After recovering 200 servers from backup (live restore on Proxmox), I got on Friday morning, after the scope came to light, and asked ChatGPT to help me with a script that launches from a flash drive, PowerShell unlocks BitLocker looking up the code using a dump from the BitLocker database also on the flash drive, then deletes the file and reboots. In about 30s the system is good and all you need to know is the UEFI password to switch the
Re: (Score:2)
Re: (Score:1)
You still have to sit at each computer and manually boot it to the USB drive.
In the past I have setup PXE booting on a network. After booting from the network I was given a prompt. Memtest downloaded and ran memtest, dban downloaded and ran Darik's Boot and Nuke and automatically erased all drives in the system. I would imagine that this tool could be setup to run from a network boot.
In the past I read a bit about Windows Deployment Services, I thought after install if computers were set to PXE boot it would check with the server if it should boot from the hard drive or a netw
Re: (Score:3)
Netboot supports Windows.
Twenty years ago the single admin for our lab full of macs set up network booting so if one of them stopped working he could just hold a couple keys while it booted, select the network image, and have it restored to working.
If the Windows people aren't doing stuff like that how do they recover from ransomware attack... oh, right.
Re: (Score:2)
Gee, if only you could put something in a disk image other than an OS....
Somebody really should invent that.
Re: (Score:3)
Things like this is what Intel vPro and the whole AMT thing is for. Of course, it's all evil and stuff, but the whole "control your PC remotely" is what vPro and such technologies are for.It's like IPMI for desktop PCs.
https://manpages.ubuntu.com/ma... [ubuntu.com]
For remote users, you can probably ge
Re: (Score:2)
The computer needs to be booted up to at least the login screen, possibly the desktop, in order to use a remote access tool, so that isn't going to work.
Re: (Score:2)
vPro doesn't need the machine to be bootable. You plug it into a network, and go to the IP. You an configure the BIOS, power on/off and set the boot image of the PC remotely.
It's IPMI, but for desktop/laptop PCs. It doesn't need a working OS because it's running on the embedded CPU on the processor.
Re: (Score:2)
You still have to sit at each computer and manually boot it to the USB drive.
No *YOU* don't. This is a solution the user can do. That's what I heard my company is doing (I'm on holiday laughing into a mojito at what is going on), getting users to do their own recovery.
Manually walking to the many thousands of machines would be a nightmare. Getting a user to log in as admin and delete a system file would be a nightmare. For me personally, having to get my arse off my work from home and drive the 3 hours to the nearest office where I can get IT support would be a nightmare (were I aff
Re: (Score:2)
Finding the Bitlocker recovery key is probably an even bigger time sink. Assuming they kept proper records for them, of course.
The default setting in Windows 11 is to have Bitlocker enabled on the boot drive, with the key stored in the TPM. They TPM won't release the key to another OS booting from a USB drive, so you must have a copy of the recovery key on hand.
Even with Windows 10, many business laptops came with Bitlocker enabled by default.
I bet there are a lot of admins out there entering serial numbers
Re: Less Manual? (Score:1)
You can query AD over LDAP presuming you have the admin credentials and dump them to CSV.
Re: (Score:2)
It still means finding and typing in long strings of digits and letters on every single machine. I'm seeing posts on Twitter from admins with a few thousand laptops to work through.
Re: (Score:1)
PowerShell can lookup data in either AD or CSV and enter the key for you.
Re: (Score:2)
Actually no. There are tools (iLO from HP) for example that allow you to handle things remotely. I worked on a project with Windows on a little (35,000 PC deployment) and it was on HP hardware mainly because of iLO maturity and we didn't want to send techs to unbrick a system.
So it does not overcome the Bitlocker requirement (Score:2)
Re: (Score:2)
If a machine needed the Bitlocker key, this tool only streamlines the process but people still need to get their Bitlocker keys from IT.
Which is a good thing. If this tool could bypass Bitlocker, that's a huge gaping security issue which would be exploited in a heartbeat.
Re: (Score:3)
Re: (Score:3)
Assuming vaguely competent security, it adds more work overall even if it saves a bit of typing. Typing in the bitlocker recovery key is 3/4 of the characters needed.
You have to make and distribute USB drives (if you can do this, you can easily do the fix free-hand IMO)
Then you have to distribute the BIOS password to change boot order
Then you (might) have to enable USB
Then you have to counter your security training about plugging USB drives in
Then you have to get the Recovery Key ID and/or look up the Reco
Re: So it does not overcome the Bitlocker requirem (Score:1)
all your base are belong to us (Score:1)
You get no benefits, no pay, no aspirin, big huge headaches, and you have no time to do anything but what we tell you.
I thought I'd personally welcome you to the team.
Microsoft bootable USB drive (Score:1)
Re: (Score:2)
Back to the teat (Score:2)
For more sour milk.
God Bless Roy and Moss (Score:3)
Re: (Score:3)
This one isn't that bad, but it *is* a warning that something much worse is plausible. This will, of course, be ignored.
Block root access by third-party non-OS components (Score:3)
Letting third party software run at the root OS protection level may not be such a good idea.
That third party software can be of unknown quality, depending on code libraries/packages of unknown quality and trustworthiness.
Microsoft could go quite a way by, long term here, disallowing non-Microsoft processes from running at the root/OS/Admin privilege level.
How many of the Windows services from third-parties are just there to update the installed copy of that vendor's product?
Consider the many decades of security fixes, from critical to minor, that Adobe Acrobat Reader on Windows has. And then consider that the same company, Adobe, has a root/OS/Admin level Windows service running on nearly every end-user Windows machine for decades.
Re: (Score:2)
Microsoft could go quite a way by, long term here, disallowing non-Microsoft processes from running at the root/OS/Admin privilege level.
Yep, let's lock down the OS and block all antivirus software from working. Slashdot loves it when only one solution is possible. Every computer only running Defender, and with a defence against antitrust suits to boot, all the while removing the user's ownership of how software runs on their machine. *GENIUS*. What could go wrong.
I find it hilarious on Slashdot, home of the "I need to own my device" crowd, the number of people who are currently promoting ideas that their desktop computers should be locked
Dumb Dumb Dumb (Score:3)
You would think for one of the most critical operating systems in the world, the OS could auto recover.
Heres the algorithm.
-> Did I BSOD last boot?
-> Revert last operating system file update.
-> Reboot and repeat regression twice.
I can see a foreign state actor going... Hmmmm patch Tuesday with a one-week time delay.
Re: Dumb Dumb Dumb (Score:4, Informative)
What you and a lot of the other amateurs posting are missing is that this was NOT a Windows update.
None of the Windows update mechanisms were involved so probably half the comments on this story are just plain wrong.
Some method of accessing the file system that bypasses the normal Windows security methods is needed whether that is Safe Mode or booting WinRE.
If a system is encrypted (bitlocker or any other full-disk encryption product) will need a key entered to access the data either way.
The automatic update process that installed the file is Crowdstrike and this is completely separate from Windows updates.
More like an av product auto-updating anti-virus definitions.
As for *nix and *BSD - both do have innately better security but are NOT totally free from this type of issue. It just hasn't happened yet.
And even is it does would not be likely to be as devastating.
Re: (Score:2)
As for *nix and *BSD - both do have innately better security but are NOT totally free from this type of issue. It just hasn't happened yet.
It did happen. To Linux. In May, 2024. Kernel panic on RHEL 9.4. A year prior similar issue on Debian.
Re: (Score:2)
Linux and FreeBSD don't have better security if you have root access.
MacOS does because the operating system is on a read-only volume that even root can't modify. There are some immutable linux distros that do the same, but they are not very popular.
Re: (Score:2)
Re: (Score:2)
I've seen a few speculative tweets and vids on the subject, it seems that maybe Crowstrike was distributing updates w/o going through a WQHL process since the BSOD traces I've seen showed Vulcan downloading a .SYS file and then invoking it. Anyway, it'll be interesting to see what Crowstrike says about the whole thing and explain to all of us why they bypassed WHQL. Speculatively, it makes for a nice attack surface.
Re: (Score:2)
What you and a lot of the other amateurs posting are missing is that this was NOT a Windows update.
The person you are replying to never actually said that this was a Windows update. What they did say is that the operating system keeps track of changes to its running configuration and should use that system to revert any changes that were made if the operating system crashes before it fully reboots again.
This is not a huge ask in this day and age.
Re: (Score:2)
You would think for one of the most critical operating systems in the world, the OS could auto recover.
Ring 0 kernel fault. There is no OS yet. What part of this are you NOT getting?
Clearly a large chunk. Which begs the question - Why do you then imagine you're the arbiter of magical solutions in cases like this?
Re: (Score:2)
Ring 0 kernel fault. There is no OS yet. What part of this are you NOT getting?
No it's not, and Windows does actually detect the BSOD and load the recovery screen next boot. Why is it the people who post with the most aggressive language are those who have no clue about what they are talking about.
Re: (Score:2)
No it's not, and Windows does actually detect the BSOD and load the recovery screen next boot.
Dear chuckledink. Please do try loading last known good config from there and tell us all what happens.
Oh, look. Kernel panic. Wow. I wonder what the problem could be? Is it the OS where we can magically do magical things to bypass this? Why no. It isn't.
Why is it the people who post with the most aggressive language are those who have no clue about what they are talking about.
Really good question. Why do the clueless continue to treat their ignorance like an unassailable fortress? Who knows? It must be genetic.
Re: (Score:3)
Problem is that from the point of the view of the OS nothing changed.
Crowdstrike uses a driver shim that loads external code with improper error handling.
The driver didn't change, only the loaded external code that is pretty much invisible to the OS, so there is no last known good configuration to fall back to.
Re: (Score:3)
No. Auto-recovery is a bad thing for user initiated files. It has all sorts of unintended consequences. Windows already auto-recovers its most critical system level files, but specifically excludes 3rd party software from this.
As it stands you don't end up in a BSOD loop. If you were affected by CrowdStrike windows will reboot into a recovery screen. It already detects the issue *and gives the user the option to do recovery*.
The big problem here is that Crowdstrike didn't save the OS state before applying a
PE should always be available (Score:2)
Every system should have an emergency "Windows PE" built-in by default, so no matter what you can access your personal files with a GUI. I can't tell you how many times I've had systems where even Safe Mode gave me endless BSOD. It's far easier to use Linux to backup everything and then just wipe the machine.
Re: (Score:2)
So your solution would be an unamainted, unpatched, highly vulnerable internal OS on every machine?
Pretty sure we could save the time, effort and agony by just taking a sledghammer to our desktops/laptops/servers.
Re: (Score:2)
Why wouldn't it be patched every time "regular" Windows is patched? Isn't it just a stripped-down version of Windows with most stuff disabled? You know, kind of like Safe Mode?
Get real. The reason why this isn't a thing is because, as usual, Microsoft is paranoid about piracy. Almost all "security" today is about protecting intellectual property, not protecting user data.
Re: (Score:2)
Why wouldn't it be patched every time "regular" Windows is patched?
Because you're now expecting the vendor to maintain 2 distinct versions of OS on the same device?
Because even a "stripped down" OS will come with its own slew of CVEs requiring patching
Because maintaining/managing/patching yet another OS, alongside the existing one, the EUFI BIOS and the firmware is insane
It's almost as if you believe it should be OS's all the way down?
The reason why this isn't a thing is because, as usual, Microsoft is paranoid about piracy.
That's a mighty fine conspiracy theory you have going there. You should probably get that treated before it develops into a full blown case
Re: (Score:2)
It does have a recovery partition by default which does that. The instructions to fix it make use of that.
Re: (Score:2)
The only options when booting from that recovery partition are repair and reset. There is no option to get into a desktop with a GUI. I've never had repair work as it just says, "Windows is unable to repair", and reset will wipe the machine and delete all your applications and data. If Safe Mode doesn't work (and there is no reason why it shouldn't), your only option is Linux.
I remember when you could boot your Amiga, Atari, or Mac from a floppy or CD-ROM and it "just worked". Microsoft stopped allowing
Re: (Score:2)
I think you can get a command prompt though?
That would allow you to navigate to system32 and delete the offending files, just like you would with a linux boot disk.
David Plumber did a good video (Score:2)
https://youtu.be/wAzEJxOo1ts?s... [youtu.be]
No Funny here (Score:2)
Too jaded to be disappointed?
Re: (Score:2)
How do we know this "recovery" process won't make things worse? Maybe it's a way for MS to install more of their sinister malware and spyware.
Probably makes sure to fully enable Recall while it's at it.
Re: (Score:2)
Re: (Score:1)
Maybe it's a way for MS to install more of their sinister malware and spyware.
Yes, because MS would go out of their way to add malware and spyware to a heavily scrutinised one-off recovery tool, when they *already* have full control, access and update capability to your entire system via Windows update at any time.
Fucking moron.
Re: (Score:1)
"Fucking moron." That was uncalled for.
Re: (Score:2)
Re: (Score:2)
I taught myself to program on an Apple ][+. That made me a wimp, because Real Men taught themselves to program on an Apple ][ with an unprotected reset button.
Re:You can not teach someone to program. (Score:1)
poke 1,1