Microsoft Reveals EU Deal Behind Windows Access After Global Outage (wsj.com) 112
A Microsoft spokesman says that a 2009 European Commission agreement prevents the company from restricting third-party access to Windows' core functions, shedding light on factors contributing to Friday's widespread outage that affected millions of computers globally. The disruption, which caused the infamous "blue screen of death" on Windows machines across various industries, originated from a faulty update by cybersecurity firm CrowdStrike. The incident highlighted the vulnerability of Microsoft's open ecosystem, mandated by the EU agreement, which requires the tech giant to provide external security software developers the same level of system access as its own products. This policy stands in stark contrast to more closed systems like Apple's.
Paywalled article (Score:4, Insightful)
I am not now nor ever subscribing to the WSJ. I prefer my bullshit to be free.
The incident highlighted the vulnerability of Microsoft's open ecosystem, mandated by the EU agreement, which requires the tech giant to provide external security software developers the same level of system access as its own products. This policy stands in stark contrast to more closed systems like Apple's.
OK, so how does Crowdstrike function on MacOS? It runs there, too. I'm not going to find out from the "article".
Offtopic (Score:3, Interesting)
My question literally could not be more on topic.
Why does this ass clown who's harassing me keep getting modpoints? Are they an "editor"?
Re: (Score:2)
I think a few folks don't understand the process and others use modpoints to score some payback. Sad actually.
Re: Offtopic (Score:2)
I tried to look up the answer, but the article could not be viewed.
What I want to know is what the article claims the difference is.
You added nothing to the conversation as is typical for a coward, and you're accusing me of the same.
Take the beam out of your own eye, coward.
Re: Offtopic (Score:2)
Ok coward.
Why don't you run along and let people with brain cells have a conversation without your interference?
Re:Paywalled article (Score:5, Informative)
Macos doesn't allow 3rd party access to the kernel, instead they have "System Extensions" (SEXTs) These are further split into 3 categories:
DriverKit - for writing drivers
Network Extensions (NE) - for interacting with network traffic (creating tunnels/filters)
Endpoint Security (ES) - Various system event notifications. (File access, login...)
AV solutions use ES and NE to implement their functionality, limited to what Apple kindly provides. However, Apple seems to have been quite forthcoming with capabilities of SEXTs.
Re: (Score:2)
In the case of software like CrowdStrike, would it be constrained to ES?
Re: (Score:2)
Apple really doesn't allow Kexts. We tried ;)
Crowdstrike team ID as reported by Apple's notarization : X9E956P446 , running as an ES. No non-apple kext running on a Falcon infected Macos.
Re: (Score:2, Interesting)
Windows has a similar set-up. Most drivers run outside the kernel, with the transition starting back with Vista. Winsock (Windows networking) allows user-mode stuff to plug into it with low overhead. There are APIs for endpoint security callbacks in user space.
There isn't any real reason for anything to run in the kernel like Crowdstrike does, except for malware and DRM (but I repeat myself). Most AV software doesn't.
Does anyone know specifically what this broken module was supposed to be doing?
Re: (Score:1)
Re: (Score:2)
File and network interception can be done in user space. There are hooks for it. It's set up for resilience too, so if the AV software crashes it shouldn't take down the whole system.
Re: (Score:2)
In fact, no doubt that's some of the motivation for MS adopting eBPF (https://github.com/microsoft/ebpf-for-windows).
eBPF, which has only been in Windows for two years but has been in Linux for ten; eBPF, which clownstroke uses on Linux but not on Windows. I wonder why they don't use it on Windows?
Re: (Score:2)
No, it doesn't. It needs a way to
- monitor and potentially block file access*. On Linux that is done in userspace using the fanotify API.
- monitor / intercept network traffic. That can be done using local firewall redirects, and then processing and verdict in userspace using the nfqueue target.
* This assumes the "everything is a file" philosophy, and device access can be filtered using the same mechanism. AFAIK This includes named pipes...
Re: (Score:3)
Macos doesn't allow 3rd party access to the kernel
The last thing we want is even more locking down or Appleification of otherwise open OSes. There's a big difference between not allowing something, and making an alternative available.
The user should be free to royally fuck up their OS, but companies should have extensions available so they don't need to do anything dangerous. Windows has plenty of such extensions as well. They are just rarely used.
Re: (Score:2)
Agreed.
The question however was : How is this done on MacOS? Answering that.
Re: Paywalled article (Score:3, Insightful)
Re: Paywalled article (Score:5, Insightful)
Apple gets away with adding more restrictions to the OS, pretty much because macos, seeing little use overall, generally flies under the legislative radar. Basically crowdstrike can only use kernel APIs there but it can't run in the actual kernel space.
The EU regulation with MS was to provide the same level of access to 3rd parties as they use for their own AV solution. They did not mandate a level of kernel memory access.
MS could provide an API for all the operations needed for their AV solution, update their AV solution to use them, and then lock out 3rd parties from direct kernel access, just as MacOS has done. IE: a more secure design that still provides for fair competition.
Re: (Score:2)
I wonder whether the CloudStrike situation would move Microsoft to use a 3 ring architecture, whereby the middle ring would provide the accesses needed by ring 1 drivers, but with a basic crash protection offered in ring 2? Though overall, I am curious what system design or deployment changes will come from this?
Re: Paywalled article (Score:1)
MS could provide an API for all the operations needed for their AV solution, update their AV solution to use them, and then lock out 3rd parties from direct kernel access, just as MacOS has done. IE: a more secure design that still provides for fair competition.
Exploits don't care about locks.
Re: (Score:2)
MS could provide an API for all the operations needed for their AV solution, update their AV solution to use them, and then lock out 3rd parties from direct kernel access, just as MacOS has done. IE: a more secure design that still provides for fair competition.
Exploits don't care about locks.
An exploit didn't take down every Crowdstrike computer running Windows. A regular update did that. Force AV vendors to work out of userspace, enabling that via API calls as needed, and this problem would not have occurred.
Re: Paywalled article (Score:1)
Examples:
https://github.com/0x36/weightBufs
https://github.com/sslab-gatec... [github.com]
That includes altering the functionality of such APIs, aka hooking, which isn't even remotely novel. So how do you detect after said code is already staged? On macos, there's only one thing you can do: Wait on apple, who has zero interest in monitoring your particular deployment, to find out about it. The saving grace here is tha
Re: (Score:2)
Lemme get this straight. Are you basically saying that software such as Crowdstrike is needed to detect when an exploit has gained kernel level access, and also that Crowdsrike MUST have direct access at the kernel level in order to do so? IE: tough shit on preventing software like Crowdstrike from taking down all users when a bad definitions file update goes out and borks their kernel module?
Maybe I missed your point? If that was your point, then what difference does it make if Crowdstrike has kernel level
Re:Paywalled article (Score:5, Interesting)
OK, so how does Crowdstrike function on MacOS? It runs there, too. I'm not going to find out from the "article".
CrowdStrike also runs on Linux, and caused similar problems in Debian and Rocky earlier this year:
https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/
So while I can't comment on MacOS, I can say that the problem isn't limited to Windows.
Here's a very good and detailed explanation of how and why CrowdStrike software took down all those Windows computers:
https://www.youtube.com/watch?v=wAzEJxOo1ts
I can't comment on the necessity or advisability of allowing a kernel space driver to chain a user space file in its execution. It seems like a pretty obvious vulnerability. But the fact that Linux boxes were affected similarly make me wonder if there's a viable alternative way of doing it.
It also seems pretty clear that CrowdStrike has to take most of the blame here. They simply didn't test properly, if they tested at all. Since it's happened at least twice prior to this, I'd say that they're negligent rather than merely incompetent.
Re: Paywalled article (Score:2)
Was it easier for the Linux boxes to recover than it seems the windows boxes on this occasion?
Re: (Score:2)
I don't know how easy it was to recover, but the article I read indicated that after the update the Linux servers failed to boot. So I'm guessing that it was pretty much the same recovery effort as with Windows.
Re: (Score:2)
IIUC &IIRC, the Debian attack was detected on a test server, and no real effort was make to recover. They just diagnosed the problem and reported it to CrowdStrike.
I suspect that if the disk were encrypted, they would run into the same recovery problems as the windows machine did, but since (IIUC) it was a test server they just booted a live CD to diagnose the problem, and then reinstalled...but that IS a guess.
Re: (Score:2)
Re: (Score:2)
Is it possible that the damage caused from the failures on Linux servers was smaller because there is a great variety of Linux distros while Windows is all the same everywhere?
I don't know. But I did dig this up:
https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
If RHEL was also effected, then maybe there's no version of enterprise Linux that isn't similarly vulnerable. That has me wondering if there's some procedure available in Linux that allows admins to prevent CrowdStrike's update push to force a reboot without explicit admin approval. I have an impression - probably biased because I love Linux and hate Windows - that Linux admins tend to be
Re: (Score:3)
Do owners of said Linux systems have any control when their systems will receive CloudStrike updates. I'd be concerned as a system owner that I couldn't delay the update on critical systems for a couple of days.
Re: (Score:2)
Do owners of said Linux systems have any control when their systems will receive CloudStrike updates. I'd be concerned as a system owner that I couldn't delay the update on critical systems for a couple of days.
Good point. Maybe Linux admins have more control over their machines than Windows admins. Whoda thunk it? ;-\
Re: (Score:2)
Re: (Score:2)
You're missing the timeline. The Debian problem was over a month ago, and was then reported to Crowdstrike. So they had reason to know that this kind of problem existed.
Crowdstrike did the same to Linux servers before (Score:5, Informative)
Re: (Score:2)
I remember when that happened and it made the evening news everywhere. IIRC Torvalds pleaded with the EU that he should be allowed to hide APIs from sleazy outfits like Crowdstrike. I just can't find any record of it now.
Re: (Score:2)
Re: Crowdstrike did the same to Linux servers befo (Score:3)
Which distros?
Linux doesn't have one monolithic update channel through which they push garbage to their users. And the various updates are more granularly controlled by the systems' administrators. So Linux based service providers can choose to schedule an update for a small subset of servers. And then pull the plug if something gets borked.
Re: (Score:1)
The file that caused recent issues was Crowdstrike's version of a virus definition file update, it wasn't code, it is distributed directly to the Crowdstrike software many times per day and the Crowdstrike software is dependent on a live connection to their security cloud to function and will bark if you try to block it.
Re: (Score:2)
Linux doesn't have one monolithic update channel through which they push garbage to their users.
What makes you think that these updates are pushed out through the distro rather than through the application itself? There are plenty of applications on Linux that don't sit around waiting for the blessing of a distro maintainer to update themselves. That was one of the core problems with VS Code which we covered here a few months ago: The application self updated in a way that was incompatible with the dependencies on the system.
Crowdstrike doesn't care about what Linux flavour you run or how it is suppos
Absolute nonsense (Score:1, Flamebait)
Re: (Score:1)
Not useless. It allows you to rake in extra cash and give the pretense of doing something for security. That it is essentially worthless is a detail that matters to an engineer but not to a manager.
Re: (Score:2, Informative)
Re: (Score:1)
Well, yes. But too many idiots have made themselves too dependent on Microsoft trash. ecen here you find tons of people claiming this was in no way Microsofts fault, when most of it clearly was. The only good thing I see is that Microsoft apparently feels a strong need to defend itself, so _they_ know they screwed up massively.
Re: (Score:2)
Re: (Score:2)
It's definitely split, that's why I give each side 40%. Crowd Strike should have tested their update with more oversight, but Microsoft should have outright rejected the update.
It seems that while Microsoft does have a strict program for certifying kernel level drivers, the updates from CloudStrike were able to work around this, since they were only "config" updates. The issue would seem is that some of those config updates actually includes p-code.
This video provides more insight into the situation: https://www.youtube.com/watch?... [youtube.com]
Re: (Score:2)
Yes, this is really surprising. Basically a config file is quite often not any less security critical than the code, especially when it can contain code itself. It seems Microsoft either does not understand that or is willfully ignoring it. Obviously, you can split things into "config" (security critical, basically on the same level as the code in many cases) and "database" (less critical, e.g. AV signatures and the like). Seems CloudStrike did not want to do that and Microsoft let them. A real clusterfuck.
Re: (Score:2)
Makes me wonder whether Microsoft needs to add a memory protected kernel ring, outside the core ring?
Re: (Score:2)
Well. Or maybe do the sane thing and hand the actual scanning and related things to user-space by providing an interface. This stuff should not be in the kernel at all IMO, too risky and too complex. If they can do either at this time. My guess would be that cannot, because Windows is a huge mountain of cruft and quite fragile.
Re: (Score:2)
Well. Or maybe do the sane thing and hand the actual scanning and related things to user-space by providing an interface. This stuff should not be in the kernel at all IMO, too risky and too complex. If they can do either at this time. My guess would be that cannot, because Windows is a huge mountain of cruft and quite fragile.
From what I understand, the CrowdStrike module was in the core ring because it needed to have full system visibility, which is something not available to code in the user-space.
Re:Absolute nonsense (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
Except that signing the definition file updates wouldn't have prevented this bug from wreaking destruction, I agree with you. In this case, the driver was signed and it did not execute any code that was not signed.
Re: (Score:2)
Re: (Score:3)
Incorrect. The Crowdstrike driver is not running arbitrary, unsigned code that is separate from the driver itself. Rather the driver itself has a serious bug--not caught in QA testing or by MS's testing when they signed the driver--that leads to a null pointer de-reference in the driver itself while loading definition files from the update file. But at no time is it executing code in the definition file.
Re: (Score:2)
They did not update the product they updated the data file it uses. The product on getting an invalid data file - fell down and went boom.
Which you know can happen. Defensive programming and all but the reality with invalid inputs is that they can cause bad things to happen. Sure we know how to protect against memory / overflow type errors and maybe CS should have done a better job here. You can still get into trouble with login problems. Look at billion laughs attacks, or zip bombs.
Finally we are in ker
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
But the data was PCode, which executable code.
Re: (Score:3)
They did not update the product they updated the data file it uses. The product on getting an invalid data file - fell down and went boom.
Which you know can happen. Defensive programming and all but the reality with invalid inputs is that they can cause bad things to happen. Sure we know how to protect against memory / overflow type errors and maybe CS should have done a better job here. You can still get into trouble with login problems. Look at billion laughs attacks, or zip bombs.
Finally we are in kernel-land here. Assumptions get made, its part of the design. Maybe this wasn't a memory error, maybe they even correctly rejected the file but failed return what the kernel was expecting because they got the error handling itself wrong? - I doubt anyone here knows. Have you hooked IDA pro up to falcon? yeah me neither.
Turns out the config update wasn't change config, but contained p-code that resulted in a null pointer reference:
https://www.youtube.com/watch?... [youtube.com]
Re: (Score:2)
Microsoft willingly let untrusted, unverified code run in kernel space. Why didn't Crowd Strike have to get their signing changed / updated after they made an update on their product? It's useless to have a signing system for trusted kernel access, then completely ignore when someone who interfaces with the kernel makes changes. Essentially the kernel should have detected the update was bad, because it wasn't signed, and therefore should have ignored it.
Because what caused the issue wasn't code to be installed in Windows at kernel level at all, it was "just" Crowdstrike's version of a virus definition file -- Crowdstrike call it a Channel File -- and it gets updated several times per day. Somehow corrupted data in this file triggered a fault in the already installed agent.
Source: https://www.crowdstrike.com/bl... [crowdstrike.com]
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
1. The driver in kernel space has to verify the contents are valid.
2. The driver has to verify the file is safe.
3. The driver should submit the file for an integrity check within the kernel separate from its own, to make sure it's safe.
4. Windows should block the file is any issues are found.
5. Once an issue is found, the service would be disabled.
Could Crowd Strike run in user space instea
always (Score:3)
Re: (Score:2)
Yeah, but they're blaming the wrong other guy. The one they should blame is CrowdStrike. IIUC, this is one case where MS actually is an innocent party.
Until Cloudstrike does it's analysis (Score:2)
So a Kernel Mode Driver causes bad problems. Too bad the vendor of that driver hasn't provided a public disclosure of their deficient code we're all just speculating. The BSODs are the symptom, the cure is to validate through the well-documented processes including best practices to make sure you don't fuck up the planet and make all software engineers look dumb.
Re: (Score:2)
make all software engineers look dumb.
Only those working in the Microsoft space.
Re: (Score:3)
Any OS can be abused if you invoke the right level of dumb and the correctly elevated privilege level.
Re: (Score:2)
Sure. But Microtrash makes it easy to do the wrong thing and very hard to do the right thing. Hence your comment has no merit.
Yeah, suuure. The EU is to blame! (Score:2)
Nothing to do with shoddy software engineering and systems design at Microsoft (which sets the stage) and at CloudStrike (which delivers the kill). All only due to the EU forcing Microsoft to allow competitors in. Obviously, something like that would never have happened otherwise.
Microsoft sleazebags (Score:1)
trying to capitalize on the disaster they helped create. Listen well, Microsoft. If I needed a walled garden, I'd choose Apple.
The bigger elephant in the room (Score:2)
Re: (Score:3)
Precision matters (Score:5, Interesting)
The EU didn't demand that Microsoft grants 3rd Party Software Kernel privileges.
What the EU demanded, that all solutions must have the same access to work with the system as Microsoft does.
So the reason for 3rd party SW having kernel access is because Microsoft is using a shitty security architecture for it's own software.
Apple recognised this problem and went through years of pain and complaints to get security software to use an API and do things in userland territory.
Re: (Score:2)
And yet, it's still very easy to infect a Mac if you wanted to. I've seen all sorts of nasty garbage get pushed on a Mac, including changing the home and search pages for a specific web browser within the OS, but not in a way you could really find. Think of it like a hidden registry setting that says, "Set the home page for Google Chrome to this web site", uninstall Chrome, reinstall, clear out all profiles for Chrome, back and forth, but it's a setting within the OS that can only be gotten to through a
Re: (Score:2)
at least it looks pretty, windows sucks there too
Apple (Score:1)
Study (Score:1)
The blame game (Score:1)
While we're playing the blame game, we have to point out that (in hindsight) the affected companies failed to have sufficient resilience in their systems and architecture. IMHO busines
Blame (Score:2)
MS can't blame the EU, crowd strike exists because the OS has problems that aren't economical to fix.
Re: (Score:2, Interesting)
Re: (Score:2)
1. By providing a proper secure channel for third party developers to access OS internals
2. By requiring that all system-level access that can be disruptive be reviewed by Microsoft and tested
3. By designing the OS such that it recovers gracefully from such nonsense.
That said, this isn't Microsoft's fault. It's just a bad idea in general to give third-parties access to a system like that. I have to say that Linux and MacOS handle this much better.
Re: Buck passer (Score:2)
Re: Buck passer (Score:4, Funny)
I’m sure the systemd team is working on it.
Re: (Score:2)
My idea, which someone shot down by someone with more low level knowledge:
Move third party drivers to a 'rump kernel' over Hyper-V, thereby trapping any illegal operations before they bluescreen.
Or maybe I've just reading too many articles about the theoretical benefits of Genode and/or Fuchsia...
Re: (Score:2)
'someone shot down by someone'
sorry, 2 o'clock in the morning and I guess my idea is still worse than my grammar! :)
Re: (Score:2)
You have just implemented an inefficient microKernel. The correct answer for a uKernel is to turn the kernel into a very small amount of auditable code - 150-200kLoc give or take. Everything that isn't needed for scheduling and memory management and message passing is moved into a resource driver in user mode. A resource driver in user mode takes over the hardware and uses it. A user mode driver instead of making a system call to get access to the driver - passes a message to the resource driver. When
Re: (Score:2)
Re: (Score:2)
The CrowdStrike software specifically needs to know what every other process in the system is doing. That's the whole point. If it's in an isolated environment, it wouldn't be able to determine if some other process was doing something it shouldn't.
Re:Buck passer (Score:5, Insightful)
So if I convince you to run a linux kernel mode driver that shits all over memory - that is the fault of Linus and Linux engineering?
Trust me I could have that driver written in about 15 minutes, and it would take down any system it is deployed on.
Re: (Score:2)
That was my first reaction, but I don't have any proof that it's correct.
The closest I have that is that the version for Debian, which it is claimed was detected and reported to Cloudstrike over a month ago, was quickly detected and removed.
Re: (Score:2)
match made in heaven
Re: (Score:3)
Disagree
It's the fault of IT admins who lazily allowed automatic updates instead of testing before widespread deployment
Re: (Score:2, Informative)
Untrue. The update that caused this was automatic even if automated updates were turned off. Get some clue?
Re: (Score:2)
Company saves big $$$, their employees get the problems. Believe me, that's how it goes, I have been seeing this decline my (multinational NSADAQ listed tech) company for years. And all the managers hail the SaaS because their
Re: (Score:1)
I see some cretins deep in denial cannot stand the truth and have to go "shoot the messenger" by downvoting. Pathetic.
Re: (Score:2)
liquidation of the corporation and executives
Harsh. Perhaps give them a second chance as indentured servants?
Re: (Score:2)
And risk another world wide disaster?
Re: (Score:2)
That's kind of funny.... you say that as if you're totally ignorant to all the popular anti-malware apps on Linux and other non Microsoft operating systems.
Re: (Score:2)
You can install any crap you want on Linux, but few people do. And if they do, that is often only because their companies require an anti-virus software as a precondition to giving network access. Guess which flea bag OS caused that requirement.