Malaysia Asks Microsoft, CrowdStrike To Consider Covering Losses From Global Outage (channelnewsasia.com) 93
Malaysia's digital minister said today he has asked global tech firms Microsoft and CrowdStrike to consider compensating companies that suffered losses during last week's global tech outage. From a report: Five government agencies and nine companies operating in aviation, banking and healthcare were among those affected in Malaysia, minister Gobind Singh Deo told reporters. "If there are any damages or losses, where there have been any parties that have made such claims, I've asked them to consider those claims and see to what extent they are able to help resolve the issue," Gobind said, adding that the government would also assist on the claims where possible. The total amount of losses incurred has not yet been determined, he said. The outage will cost Fortune 500 companies $5.4 billion, according to estimates from insurers. The projected financial losses exclude Microsoft.
Some people are so needy (Score:5, Funny)
The 10 bucks in Uber Eats [slashdot.org] oughta be enough compensation for anybody.
Re: WE NEED LIABILITY (Score:2)
Hardware manufacturers have product responsibility, but software makers don't. Time to change that.
Re: (Score:2)
Re: (Score:1)
bahahahaha
oh man.. if I was one of the affected companies, I would set aside $10M to sue these cucks.
Re: (Score:2)
Depends on how the contract is worded. They might just get comped service. Like 1 month free service. You would have to prove negligence, like proving they did not properly test during quality check.
Microsoft had nothing to do with this... (Score:5, Informative)
If a drunk girl runs over a grandma in her Honda, Honda isn't responsible.
Re: (Score:1, Informative)
In your analogy, Microsoft carefully evaluated the drunk girl and determined she was fit to drive.
Make no mistake, Microsoft dropped the ball here as well.
Re: (Score:1)
Re: (Score:3)
Microsoft has its own endpoint protection products and is a competitor to Crowdstrike. Are you absolutely sure you want to say that allowing third-party software onto Windows is a legal liability for Microsoft?
Re: (Score:2, Troll)
You do know that Microsoft put that driver through their own stringent testing process, right?
Re: Microsoft had nothing to do with this... (Score:2)
After they were forced to by the EU, no? Their ability to say "this is really dangerous, even though we can't point to a specific exploitation at this time" was essentially gutted.
Re: (Score:2)
How? My laptop booted just fine since I dont use crowdstrike.
Re: (Score:1)
This is a kernel mode driver. Microsoft did their own aggressive testing (WHQL) and issued a certificate.
Re: (Score:2)
Every iteration? Or just the initial build? If its every iteration is this a case of the Challenger O-ring debacle?
Re: (Score:3)
In what way did Microsoft "evaluate" CrowdStrike software? Do you imagine that they did QA on it? Apparently, even CrowdStrike didn't do proper QA on it, so why would you imagine that Microsoft would do it on somebody else's software?
Re: (Score:2)
They're probably referring to driver signing. That doesn't mean MS tested the software, it just attests that the the person holding the certificate signed the binary. Microsoft only revokes those certificates if it becomes apparent the private keys have leaked (allowing other people to sign drivers) or the certificate is being used to sign outright malware (e.g. ransomware). They don't revoke certificates for signing poor quality software.
WHQL hardware drivers are a bit different. Those are supposedly t
Re: (Score:2)
But that's only for hardware drivers, so it isn't even applicable to this CrowdStrike crap.
False. The driver in question, being a kernel mode driver, went through the WHQL process.
Re: (Score:2)
WHQL is not a QA test of the driver. It's just a test of the signing process, to confirm that the driver properly loads with the secure boot process.
https://learn.microsoft.com/en... [microsoft.com]
Re: (Score:3, Insightful)
In what way did Microsoft "evaluate" CrowdStrike software?
WHQL
Do you imagine that they did QA on it?
That's what happened.
why would you imagine that Microsoft would do it on somebody else's software?
Because that's how WHQL works.
You clearly know nothing about this, so why are you so certain that Microsoft did nothing?
Re:Microsoft had nothing to do with this... (Score:4, Informative)
What caused the Crowdstrike outage wasn't a driver update, it was their version of a virus definition file (Crowdstrike call it channel files). This gets updated many times per day (can be hourly), is a normal data file pushed directly to the Crowdstrike data directory and read by Crowdstrike software, which requires a live connection to their "security cloud" to function. So everyone saying IT admins should have tested the update first don't really understand how this work, it isn't possible.
Unfortunately this particular definition file contained malformed data that provoked a malfunction in the already installed kernel level agent. And as anyone who has read Crowdstrike's PIR know it was entirely due to Crowdstrike''s test and release process being shockingly lacking in normal safeguards.
Btw. Crowdstrike did the exact same thing to Linux servers earlier this year (Debian and RH), so if this is a Microsoft issue it is a Linux issue as well.
Re: (Score:2)
From what I understand the issue is that the Crowdstrike driver was downloading some bytecode as part of the definition updates. It was the bytecode that was flawed, and the lack of proper checks on it in the driver.
From what I've seen of WHQL they don't demand source code, they just do stability tests.
Re: (Score:3)
WHQL doesn't involve Microsoft performing a QA process on the driver itself. Rather, it's just a test of whether the driver signature properly allows it to load during the secure boot process. https://learn.microsoft.com/en... [microsoft.com]
Even if Microsoft had tested the driver itself (which they didn't), the CrowsStrike update that caused all the mayhem, was a *content* file, essentially a configuration file that provided updates to rules that were used to detect malware threats. The driver itself wasn't updated, just
Re: (Score:2)
it's just a test of whether the driver signature properly allows it to load during the secure boot process
That's obviously not true. Are you addicted to being wrong?
Re: (Score:2)
I provided a link to the Microsoft Learn WHQL page. What better authority is there than that?
Re: (Score:2)
The link isn't the problem. It's your inability to read and understand the contents.
You're wasting everyone's time with your endless nonsense. Go away.
Re: (Score:2)
As part of WHQL, Microsoft does test for "compatibility and reliability" on Windows. It does not test the actual driver functionality.
However, THIS update--the one that caused the blue screens--was a "content data file" which was not tested as part of WHQL. It was deployed outside of the install process, and loaded in real time by the pre-installed drivers. Thus, WHQL did NOT provide any testing of the faulty update.
Re: (Score:1)
Re: Microsoft had nothing to do with this... (Score:2)
No. MS validated the car. They had no idea who you were going to put behind the wheel.
Re: Microsoft had nothing to do with this... (Score:2)
We donâ(TM)t sue the DMV for every drunk driver.
Re: (Score:2)
I think that if a drunk girl takes her Honda to a non-Honda mechanic who then damages the engine, causing severe failure that leads to her running over grandma, it ain't Honda's fault.
Re: (Score:2, Informative)
To reuse your analogy, if you needs to install aftermarket brakes on your Hone because Honda designed the car so poorly the standard brakes are plain dangerous, and the manufacturer of the aftermarket brakes fucks up and cause you to run over Grandma, both the aftermarket part maker and Honda are at fault, not you.
Re: (Score:1)
Re: (Score:2)
Yes, unless the Honda is the only option if you need to drive on certain roads, or your employer mandates that you need to drive a company Honda, otherwise you're out of a job.
Re: Microsoft had nothing to do with this... (Score:2)
What are the "standard brakes" in this strained analogy?
Re: (Score:1)
The analogy would be first Honda deploying faulty system upgrade causing all Honda cars to simultaneously experience an OS panic and come to a halt mid road, causing as well the largest traffic jam in history, with an impact of billions in missed revenue.
Re: (Score:2)
The update was for the Crowdstrike software, not nothing Microsoft published.
Shockingly, Microsoft is not at fault for anything about this incident.
=Smidge=
Re: (Score:2)
That's right, thanks for the correction.
Re: (Score:1)
Thank you for leaving the democratic party. Tell those kooks to stop trying to sue rifle manufacturers. Its literally the same analogy.
Re: (Score:2)
Microsoft doesn't advertise how awesome their software is with Crowdstrike, but rifle companies often advertise right wing warrioring as a cool thing with their rifles. Also, Microsoft puts a lot of effort into security for their products, but companies specifically leave out safeties and other security features that could reduce deaths.
Re: (Score:2)
talk out your ass much? the manufacturers arent the ones doing the advertising. The FFL dealers are the ones advertising on their websites. Go to Kimber, Colt, Heckler & Koch, Smith & Wesson, etc. They just show products. And at no time are the FFL websites advertising shooting PEOPLE. Your example is just another made-up lie told by left-wing loonies. But auto companies, the point of the post, often DO run commercials of their vehicles driving at excess speeds or inches away from cliffs. In fact th
Re: (Score:2)
Re: (Score:2)
"A little is good, more must be better!" - Beverly Hofstadter
Re: (Score:2)
Before you speak on the matter you should educate yourself. Educate yourself on the 1986 hughes amendment. Educate yourself on where h&k is located (hint, not the US). Educate yourself on the fact that the video you are illustrating is a military and law enforcement contractor video. No regular citizen in the US is allowed to own an automatic weapon or select fire weapon manufactured after 1986. The NFA registry is closed to new submissions. You should ask yourself why this type of marketing works so w
Re:Microsoft had nothing to do with this... (Score:4, Informative)
It gets better. Microsoft was apparently developing APIs which would allow apps to more safely hook into OS functionality related to AV stuff, rather than needing to use kernel code in a driver. If this API had come to fruition apparently CrowdStrike could have leveraged it and potentially we'd just be hearing a news blip about CrowdStrike's app breaking worldwide, but systems remaining running and working, instead of bluescreens everywhere..
But the EU courts said MS couldn't incorporate those APIs into Windows for anti-competitive reasons.
That's bullshit (Score:1)
The only thing the EU said was that MS couldn't use APIs in Windows Defender that they don't allow third party security software to use. They can make a more isolated, safer API available, they just can't make it impossible for third parties to install kernel modules.
Re: That's bullshit (Score:2)
Sure but then if they rapidly iterated on these APIs, they'd be told that was unfair because other companies couldn't keep up. There's a reason the APIs are private. They're unstable and not something MS wants to provide customer support for.
Government regulation often has a negative impact on security, even if that's not the intention.
Re: (Score:1)
That doesn't sound right. The EU's issue was hidden APIs that only Microsoft could access. If Microsoft introduced new APIs for this functionality, and their AV offering (Windows Defender) used them, there would be no problem.
Re: (Score:2)
Microsoft never does "the right thing" as normal people would understand it. The EU doesn't take kindly to such actions.
Had Microsoft done what you say, in the way you describe it, they'd have been absolutely fine. Instead, they tried to do it in such a way that their own Windows Defender would get an advantage over everyone else - that's why the EU said no.
Re: Microsoft had nothing to do with this... (Score:2)
Re: (Score:2)
But the EU courts said MS couldn't incorporate those APIs into Windows for anti-competitive reasons.
Which would not have happened had Microsoft not been acting like a greedy little dictator that wants to take over the world.
Microsoft needs to be completely and utterly dissolved. Their only future should be in the history books of how unregulated capitalism leads to garbage monopolies that will refuse to produce quality products.
Re: Microsoft had nothing to do with this... (Score:2)
Exactly, MS is not at all fault here. And I don't blame CS either. When you offload testing and security to an outside firm, this is the risk you take. Customers could have done staged and vetted rollouts. Customers could have diversified their exposures. Customers could have forced CS to split read only definitions from software code changes. Customers could have audited CS' change management processes.
Nope, people homogenized, got their bonuses, washed their hands, and moved on.
If CS owns people money,
Re: (Score:2)
I thought this as well until I learned Crowdstrike Falcon has two types of updates. The client which you can hold back so it's not bleeding edge, and then content (think of it like virus definitions) updates which the crowdstrike customer has NO control over.
I was the business owner of Crowdstrike Falcon across 3 companies and up until last year. This is something my many sales reps never told me about.
I do however fault businesses (customers of crowdstrike) for not having a quick path to disaster recovery.
Re: (Score:2)
If a drunk girl runs over a grandma in her Honda, Honda isn't responsible.
I agree with your sentiment but there are always more than one factor in any incident.
If that drunk person runs over their grand parent in a Honda and it was found that there was a software issue that could have caused unintended acceleration, Honda certainly would shoulder a portion of the blame.
If you've ever read a serious incident report (I.E an aircraft or industrial accident) then you'll know that they look at every possible cause and the cause is rarely down to a single party. It's never as bla
Re: Microsoft had nothing to do with this... (Score:2)
The Honda will protect the grandma driving it, but there will be damages to the car.
Shared responsibility. (Score:1)
Re:Shared responsibility. (Score:5, Informative)
You know that's not possible with Crowdstrike, right?
They say they're adding these 'features' in an upcoming release, in today's email update.
Now, I know of a hospital in Boston that used another vendor because they had that proper level of control. They stayed open.
It's totally legit to fire the people who signed off on buying Crowdstrike despite this defect.
There were better options.
But companies like Delta Airlines might just go out of business instead.
Re:Shared responsibility. (Score:4, Informative)
Any IT manager who uses crowdstrike should be fired. It's clearly a shamefully fucked up product. The threat of a bad update is far greater than that of a zero day.
Re: (Score:2)
I was on PTO that day and enjoyed it, because I don't pay for CrowdStrike. Still enjoying the uptime.
Re: (Score:2)
IT Managers do not own the tool, it's a Security tool.
Re: (Score:1)
Re: (Score:3)
Isnt CrowdStrike automatically rolled out to all the managed workstations? I thought that was the whole point of protection from zero-day exploits. This isn't like a windows service pack where as admin you manually setup the automated update, is it?
Re: (Score:3)
As you can see from CrowdStrike's incident report https://www.crowdstrike.com/fa... [crowdstrike.com], the content update (not a software update) that caused the blue screens was rolled out to everyone at once. Customers had no option to roll out the update to selected systems to control risk.
With that said, customers *did* have the option to choose better software.
Huh? (Score:2, Insightful)
Exactly how were these Malaysian companies FORCED to use those products? None of our systems use Crowdstrike (there are, you know, other EDR products) and we had ZERO problems. How about taking responsibility for your upstream decisions instead of blaming everyone else.
Re: Huh? (Score:2)
Sic semper outsourced security
Re: Huh? (Score:2)
That is a very weak argument, if you buy food and it ends up being poisoned, should you also not complain?
Yes, but... (Score:4, Informative)
The EU should also be in the mix. Or rather should be there instead of Microsoft.
Back in the day, when Windows was making a move the the 64-bit kernel (x86_64/amd64 branch) they introduced a feature called Kernel Guard. That was basically a hardened kernel which could not be patched after boot. This would prevent malware from injecting rootkits, and yes, also third party drivers from breaking the kernel (at least in this path).
Of course, the "so called" security software would not work with it, as Symantec, McAfee and others were essentially malware themselves. And they went crying to EU.
Long story short, EU agreed with the malware, sorry security software, authors, and force Microsoft to weaken kernel security:
https://web.archive.org/web/20... [archive.org]
If anyone is to blame, excessive, and non-intelligent government intervention should be top of the list.
(I'm not saying Microsoft are saints, or did not actually do anti-competitive stuff. But in this particular instance they were right to lock the kernel down)
Re: (Score:1)
It's a sovereign state, it makes a difference (Score:2)
If they wanted to be extra polite they could work through some international tribunal or file suit outside of Malaysia. A much more likely path is to use their court system. Note that criminal charges are on the table and that could include individuals.
Microsoft and CrowdStrike are in a precarious situation. They need to make some restitution, but if they do
Re: It's a sovereign state, it makes a difference (Score:2)
MS should simply respond with "we recommend Windows Defender for a reason".
Re: (Score:2)
In this case "asking" is not a request that can be flatly ignored. A government has a whole lot of options that are not available to other entities.
If they wanted to be extra polite they could work through some international tribunal or file suit outside of Malaysia. A much more likely path is to use their court system. Note that criminal charges are on the table and that could include individuals.
Microsoft and CrowdStrike are in a precarious situation. They need to make some restitution, but if they do so then a whole lot of other players will pile on. A negotiated deal with no explicit damage amount is likely their best option. BTW, splitting hairs over who is really at fault is not that relevant. That discussion is not very meaningful outside the US when state actors are involved.
Malaysia has it's problems with government, same as all countries but it's still a functioning democracy, in fact one of the better functioning ones in SE Asia with less corruption than most of it's peers (I know that is not a high bar). Like the US, UK, et al. the government doesn't just get to demand compensation. Even if they change laws, it can still be challenged in court and ultimately even if they lose they can still opt not to do business in that country (Google pulled out of China because they beca
All legal action has a first step (Score:1)
All legal action has a first step, I am sure the offer step-up and assist will be followed by strong legal action if their request is denied, and I am sure any actions taken here will be considered in subsequent legal action.
Re: (Score:2)
Do you have Hillary's email?
Uh, no (Score:2)
If my business depends on something to work correctly and the vendor does not guarantee correct operation in the contract, it is my responsibility to get an insurance that will cover the cost when that thing fails.
Re: Uh, no (Score:2)
I am curious about what CrowdStrike's EULA says about this stuff. They are presented as a managed service. Usually managed services have specific wording around service levels and penalties that would limit liability.
Re: (Score:1)
The EULA limits their liability to the software fees paid. They're never going to have an agreement that makes them liable to consequential losses if they can't control what they might be and charge accordingly.
It was in the fine print (Score:2)
Not going to happen for a standard license fee (Score:1)
Cover for consequential loss is never going to be included in a standard software license fee.
The same PC/Windows OS/Crowdstrike based system could be used for an office worker or critical infrastructure.
No single license fee would be suitable for the very different consequential losses in those two situations.
This is on the sysadmins too (Score:2)
Crowdstrike; "We considered it, no thanks" (Score:1)
Not a chance! (Score:1)
Motherfucking consequences (Score:2)