Who Will Pay For the Costs of Crowdstrike's Outage? (cnn.com) 196
8.5 million Windows devices were ultimately affected by the Crowdstrike outage, according to figures from Microsoft cited by CNN.
And now an anonymous Slashdot reader shares CNN's report on the ramifications: What one cybersecurity expert said appears to be the "largest IT outage in history" led to the cancellation of more than 5,000 commercial airline flights worldwide and disrupted businesses from retail sales to package deliveries to procedures at hospitals, costing revenue and staff time and productivity... While CrowdStrike has apologized, it has not mentioned whether or not it intends to provide compensation to affected customers. And when asked by CNN about whether it plans to provide compensation, its response did not address that question. Experts say they expect that there will be demands for remuneration and very possibly lawsuits.
"If you're a lawyer for CrowdStrike, you're probably not going to enjoy the rest of your summer," said Dan Ives, a tech analyst for Wedbush Securities....
But there could be legal protections for CrowdStrike in its customer contracts to shield it from liability, according to one expert. "I would guess that the contracts protect them," said James Lewis, researcher at the Center for Strategic and International Studies...
It's also not clear how many customers CrowdStrike might lose because of Friday. Wedbush Securities' Ives estimates less than 5% of its customers might go elsewhere. "They're such an entrenched player, to move away from CrowdStrike would be a gamble," he said. It will be difficult, and not without additional costs, for many customers to switch from CrowdStrike to a competitor. But the real hit to CrowdStrike could be reputational damage that will make it difficult to win new customers... [E]ven if customers are understanding, it's likely that CrowdStrike's rivals will be seeking to use Friday's events to try to lure them away.
One final note from CNN. Patrick Anderson, CEO of a Michigan research firm called the Anderson Economic Group, "added that the costs could be particularly significant for airlines, due to lost revenue from canceled flights and excess labor and fuel costs for the planes that did fly but faced significant delays."
See also: Third Day of 1,000+ Cancelled Flights, Just in the US, After Crowdstrike Outage .
And now an anonymous Slashdot reader shares CNN's report on the ramifications: What one cybersecurity expert said appears to be the "largest IT outage in history" led to the cancellation of more than 5,000 commercial airline flights worldwide and disrupted businesses from retail sales to package deliveries to procedures at hospitals, costing revenue and staff time and productivity... While CrowdStrike has apologized, it has not mentioned whether or not it intends to provide compensation to affected customers. And when asked by CNN about whether it plans to provide compensation, its response did not address that question. Experts say they expect that there will be demands for remuneration and very possibly lawsuits.
"If you're a lawyer for CrowdStrike, you're probably not going to enjoy the rest of your summer," said Dan Ives, a tech analyst for Wedbush Securities....
But there could be legal protections for CrowdStrike in its customer contracts to shield it from liability, according to one expert. "I would guess that the contracts protect them," said James Lewis, researcher at the Center for Strategic and International Studies...
It's also not clear how many customers CrowdStrike might lose because of Friday. Wedbush Securities' Ives estimates less than 5% of its customers might go elsewhere. "They're such an entrenched player, to move away from CrowdStrike would be a gamble," he said. It will be difficult, and not without additional costs, for many customers to switch from CrowdStrike to a competitor. But the real hit to CrowdStrike could be reputational damage that will make it difficult to win new customers... [E]ven if customers are understanding, it's likely that CrowdStrike's rivals will be seeking to use Friday's events to try to lure them away.
One final note from CNN. Patrick Anderson, CEO of a Michigan research firm called the Anderson Economic Group, "added that the costs could be particularly significant for airlines, due to lost revenue from canceled flights and excess labor and fuel costs for the planes that did fly but faced significant delays."
See also: Third Day of 1,000+ Cancelled Flights, Just in the US, After Crowdstrike Outage .
No one (Score:4, Interesting)
Crowfstrije's stock will temporarily drop. Maybe the release manager and a mid level QA person are reprimanded and that's it. Move along, nothing to see here. The post mortem and lessons learned is scheduled for Thursday. Please be sure to attend and bring your notes on the unfortunate event.
Crowd-Stroken (Score:2)
>> mid level QA person
There is no QA person.
Let's call them "Crowd-Stroken"
or "CrowdNuk'em"
Re:Crowd-Stroken (Score:5, Funny)
Re: (Score:2)
Wouldn't it have been cheaper to just use Linux? :-)
Re: (Score:2)
Hindsight is 20/20. Again and again.
During dotcom I had one startup job that was pure windows in 24/7 public facing production. What a fucking nightmare. I swore never again and did pure Linux jobs after that where at worst there was some stray internal windows service no one cared about. Life was good after that.
Re: No one (Score:2)
Re: (Score:3)
Geez, are the Russian *paying* you to post idiocy like this, or are you just that STOOOPID?
There is *nothing* here for the feds to pay on. On the other hand, private businesses have this thing - I don't think you've ever heard of it - called "insurance", and they'll pay.
Everyone (Score:5, Insightful)
Re: Everyone (Score:2)
Re: Everyone (Score:5, Insightful)
Business 101:
Socialize the risk/cost, privatize the gains.
In this case, the public will pay and C-suite/attorneys will get paid.
EULA Monster bites Main-street, not Wall-street. (Score:4, Informative)
EULA: "We are NOT responsible if this product eats your pets, crashes your car, falsely sets off your burglar alarm, sets your toaster on fire, or melts your computer."
Re: (Score:2)
or melts your computer
Posted to remind you of intel's hyper-threading vulnerability, I do not remember getting a new CPU from Intel. So we paid for that :)
Re: (Score:2)
But that's only applies to the direct customers.
The customers of the customer might have a case. Especially if the airline passengers don't get paid by the airline on some technicality then they might have a case against Crowdstrike. If each of them bring a case against Crowdstrike then it's going to be a busy time for lawyers.
Re:Everyone (Score:5, Interesting)
Just like any other outage, all that overtime that was needed to get systems back online will get passed to the customer by raising prices. Again.
Meanwhile, Crowdstrike will use their EULA as a defense to pay a little in damages in possible. The only people who are likely to make any money from that are the lawyers.
EULA ? Stating: nobody? (Score:5, Informative)
Re:EULA ? Stating: nobody? (Score:5, Informative)
You don't need a huge EULA for that. The MIT license is brief and has a clear no-warranty clause.
Re: (Score:3)
Sure for basic downloaded software something like the MIT's disclaimer works.
A complication here is software like Crowdstrike here deploys and runs an update from an online service that is separate from the software and managed by the vendor. There's a potential Liability other than warranty, and that's a duty of care to put Software on that automatic update service that doesn't harm your users.
Finallly, a type of tort called "Gross Negligence" cannot be waivered away; not even by a Warranty disclaim
Re:EULA ? Stating: nobody? (Score:5, Interesting)
I remember exactly one client at my last role, and I deeply respect them for actually carefully reviewing our terms and settling on agreeable terms for them. Most people aren't paid to care. I can only hope that people abandon the vendor, so other vendors learn they can't get away with this.
Re: (Score:3)
Enterprise contracts use a combination of EULA, SLA, project agreements and possibly other contracts. Just thinking they have a line in a EULA is too simplistic for this type of product.
Re: (Score:2)
Re: (Score:3)
and in update in version 1.3.45 we added the line saying this over rides any old Enterprise contracts or SLA's
Re: (Score:2)
Then someone figures out that an EULA can't be updated one-sided and declining to accept the new version of an EULA isn't ground for ending service.
Have fun with a lot of various grandfathered EULAs.
Isn't it quite obvious? (Score:3, Insightful)
Those who made the decision to use it should - nobody forced it on them.
Re:Isn't it quite obvious? (Score:5, Insightful)
Re:Isn't it quite obvious? (Score:5, Insightful)
There was a better option: nothing.
This kind of "all eggs in one basket (that someone else is carrying)" security model is worse than the risk it claims to mitigate.
Re:Isn't it quite obvious? (Score:5, Insightful)
You are assuming that there was a better option.
There is, the way everyone else roll out patches -- in-house IT teams prepare and test the package, then roll them out by phases.
Yeah, it may take longer, but it won't BSOD all your machines overnight.
Re: (Score:3)
.... yeah ... .goood except.
Every damn package requires an update. And the updates never stop.
How many software packages might even a small or medium enterprise (say, revenue about 100 Million $US / year). Hundreds and possibly thousands. every update, possibly multiple per day. Not every package has an insane cadence, but others ... or nucking futs. And remember testing isn't just "it loads, doesn't crash the computer ... go". It's checking each and every possible interaction with all other software.
Resour
Re:Isn't it quite obvious? (Score:5, Interesting)
Re: (Score:3)
They said that they have a policy in place where new software updates first roll out to a group of test machines, and then in the next round they roll out to a group of machines that are in production but non-critical, and then in the final round it rolls out to all computers.
Seems like a sensible approach in most cases, but the idea loses its appeal when you begin to contemplate being hauled in front of Congress to explain why you've allowed a data breach by failing to have the latest security updates on your most critical pieces of infrastructure.
Re: (Score:2)
Re: (Score:3)
Simple answer: "We tested these updates first, because we didn't want another Crowdstrike meltdown"
That answer holds a lot more water today than it did a week ago.
But regardless, it all comes down to risk vs. reward. Presumably CrowdStrike has the data on this and could say how often their auto-updates have blocked execution of a zero-day vs. how often they've broken a production system. If I were them I'd be getting ready to dust off those stats. Assuming the stats are favorable, of course, and with nothing other than a gut feel I contend that they probably are, even with the spectacular failure las
Re:Isn't it quite obvious? (Score:4, Insightful)
What most people don't get is there is a difference between normal updates and updates to security software. Security software is updated constantly because of zero-day vulnerabilities. Security wants these updated ASAP to get those vulnerabilities patched. It is the constant fight between infrastructure and security, security wants things updated immediately because they don't want the possibility of a breach while infra wants to slow roll things because they care about uptime and possible issues that may come out of an update. In this case, infra and support for stuck remediating all these systems because security wants immediate updates.
Re: (Score:2)
Re: (Score:2)
Indeed. "Design by cretin" is what I would call this.
Re: (Score:3)
This is changing across the board. I work for a pretty boring company that followed this practice. Our biggest problem before was being stuck with old software. We still have it today.
However some pieces of software are being allowed to just update automatically. For example a lot of the Microsoft ecosystem is just being allowed to self-update. Teams just updates on it's own for example.
I don't know the solution, but it has to be more than just our EULA means we're not responsible for anything. My personal
Re: (Score:2)
That only applies though if your airline is preventing other airlines from doing their jobs and landing and taking off planes. This af
Ass-U-Ming (Score:2)
>> You are assuming that there was a better option.
You are assuming that Linux does not exist.
Re: (Score:2)
You are assuming that Linux does not exist.
Crowdstrike's software runs in the Windows kernel, and has a type of virtual machine there since it requires access to resources that are not allowed to be accessed by user-mode programs. I think (someone correct me if I'm wrong) that Linux would have the same restrictions, and Crowdstrike's software would have therefore been architected the same way and with the same results.
Re: (Score:2)
Nobody needs this snakeoil on Linux, especially not a glorified attack-logger in that deep in the kernel. ....
It seems this SW was masqueraded as a "kernel driver" because windows restrictions on precedence over network interactions, and it integrated an interpreter or similar deferred execution because WHQL process takes too much time for them to validate an update, it seems
Re: Ass-U-Ming (Score:2)
Re: (Score:2)
snakeoil glorified attack-logger functionality ? :)
Re: Ass-U-Ming (Score:2)
Re: (Score:2)
There was. Automatic updates didn't happen overnight so there was time to move your production servers off the platform.
Re: Isn't it quite obvious? (Score:2)
Re: (Score:2)
I'm not assuming anything at all.
Any decision to install software on critical infrastructure, especially software that needs to run in kernel space, requires serious assessment by a competent professional. Doubly so if the software is installed widely across the IT assets of the institution that chooses it. If the person who is making the decision to deploy such software doesn't know how to produce such an assessment, they have no business making the decision. The massive effect is the result of a bunch of
Re:Isn't it quite obvious? (Score:4, Insightful)
Yeah. I gotta say that just letting an external tool automatically replace whatever software it liked, no testing, in production gear does seem very irresponsible of all those affected businesses.
Re: (Score:3)
Problem for security software is they will pitch that they will keep ahead of attackers. Attackers do not honor maintenance windows or update testing, so to deliver on the promise, they can't delay their updates either.
Plus this was a "content" update. Imagine having to try to stage/test security suite content updates, which are commonly multiple times a day...
To the extent you buy their promise of value at all, you'd have to be willing to accept content updates super quickly.
Re: (Score:2)
Plus this was a "content" update. Imagine having to try to stage/test security suite content updates, which are commonly multiple times a day...
If your "content" update can brick the target systems, you better make sure they are well-tested.
Re: (Score:2)
In general I think a lot of the security companies are full of it. However I'd think either:
-You recognize the security company isn't everything it's cracked up to be and don't buy
-You think the security company is critical and so you have to let them update.
Re: (Score:2)
It is a recipe for a catastrophe. Just think what happens when a competent attacker hacks their update process...
Re: (Score:2)
They were probably obliged to by their insurance company, who they will now be contacting for a business continuity claim.
Re: (Score:2)
Really, now? Can we have examples of an insurance company forcing a customer to use the services of this crowdstrike thing, please?
The opposite seems more likely - insurance companies to receive claims from their customers for the outages by crowdstrike - from those who have bought such coverage at least.
But, quite obviously, such an insurance is an offset of the liability those companies have incurred by choosing to use the spyware outfit.
Re: (Score:2)
Every insurance policy includes wording about taking reasonable care to avoid claims. For businesses looking for IT cover specifically, it usually specifies that AV software must be installed, and when your coverage is tens or hundreds of millions there is probably a list of ones you should choose from.
If you examine your bank's terms and conditions it probably says something about the consequences of not running AV software and your account getting hacked too.
Re: (Score:2)
So, no evidence any insurance company has forced people to subscribe to whatever snake oil crowdsource is selling?
Thanks.
Re: Isn't it quite obvious? (Score:3)
Those who made the decision to use it should - nobody forced it on them.
Not fully true.
You might notice that the industries heavily hit are all highly-regulated. Using Crowdstrike is part of maintaining their required security compliance. It is the standard solution so while in theory they could use something else, whatever that else is would be seen as assuming far more risk to them and their customers.
Note also that the reason Crowdstrike can even cause a problem like this is partly because of an agreement MS made with the EU in 2009 requiring them to provide other security s
Chain of responsibility seems obvious. (Score:4, Informative)
Re: (Score:2, Troll)
Blame CISA
https://www.cisa.gov/topics/cy... [cisa.gov]
They are the ones push EDR into places that should decidedly not have anything but first party software running kernel mode. Critical systems should have user-space - event monitoring, and high degrees of isolation; they should not have AV/EDR products from separate vendors on them. To big a risk anywhere availability is a concern; because these things break availability like disk access by design. Its foolish!
But the fundamental character of people who chose to
Re: (Score:2)
Re: (Score:2)
These kind of companies have a long history of lobbying the government.
Re: (Score:2)
Re: (Score:2)
Crowdstrike is definitively not fucked. What makes you think so ?
Boeing had planes literally falling off the sky and a few years later nothing has changed *at all* inside the company. And they're still not fucked.
Re: (Score:2)
Re:Chain of responsibility seems obvious. (Score:5, Insightful)
Crowdstrike made something that could break on improper data. Microsoft had the weakening due to an EU directive requiring them to allow competing anti-virus solutions.
Some pretty interesting info can be found in a video by Dave Plummer: https://www.youtube.com/watch?... [youtube.com]
Re: (Score:2)
Insurance Companies (Score:2)
The Insurance Companies who forced all the companies to use crowdstrike's product to "protect" their infrastructure.
thats who needs to be sued.
CrowdStrike Terms And Conditions (Score:5, Informative)
Is CrowdStrike good? (Score:2)
I'm totally out of the corporate type IT loop. Is CrowdStrike normally a good product (obviously excluding the recent issue)? Exactly what is it needed for?
Re: Is CrowdStrike good? (Score:3)
Re: (Score:3)
Crowdstrike is a malware detection system whose main selling point is that it reacts to stuff with crowdsourced detection. If some malicious software is detected on someone else's systems then it is supposed to be detected and stopped on yours too. In order to accomplish that timely they have to be able to push updates out on short notice.
No comments on quality, not enough experience with it personally.
Re: (Score:3)
In my opinion it's overhyped and overmarketed to business management.
The intent of the product is to be able to catch malware of various kind. If it really do something is another issue.
Holding Up To Its Name (Score:2)
At least they hold up to its branding.
I mean, they did strike a crowd of services didn't they?
As to who will pay? Well, CrowdStrike won't pay anything.
The best thing the affected clients could do is to leave CrowdStrike.
This is not about a novel vector attack that no Information-Security type of company knew about; and then they recover from it victorious and with more knowledge.
No. That is not the case. This is CrowdStrike shooting itself in its foot and making all of its clients experience "ransomware
What about consequential losses ? (Score:3)
You had a flight booked to go to see a concert and hotel for afterwards. Your flight was cancelled. The airline might give you the cost of the ticket back, but what about the other things that you paid for - who will pay you for your losses ? You are not a ClownStrike customer so your ability to sue them is limited - that is assuming that you want to risk engaging lawyers for an action with little chance of success.
Re: (Score:3)
As far as the end customer is concerned, the airline holds ultimate responsibility for providing what you paid for. And their choice of software vendors is part of that responsibility.
Re: (Score:3)
Your travel insurance (and the airlines etc) will most likely call this a Force majeure and you will get nothing.
business 101 (Score:2)
Once you have the customer's money... (Score:3)
Never Give It Back!
- Ferengi Rules of Acquisition [fandom.com].
I can speculate without information, too (Score:5, Informative)
"I would guess that the contracts protect them,"
Guess? So you have nothing of value to say but they quoted you anyway? Holy fucking shitballs.
Want to have a good time, Search for cloudstrike instead of crowdstrike and find out how often people get this wrong [state.ny.us].
Anyway any dickhole can look up the terms of use [crowdstrike.com] for Crowdstrike software. I especially like section 5 where they have the right to exfiltrate your data and show it to anyone they want. But the really important part is section 7:
As always, they indemnify themselves for destroying your business through negligence.
Re: (Score:3)
As always, they indemnify themselves for destroying your business through negligence.
IANAL, but IIRC you can be found to be liable if you are negligent, despite what you state in a contract. I suppose a lot will depend on the laws of the jurisdictionYMMV
Re:I can speculate without information, too (Score:4, Informative)
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW is what the lawyers will pick at. You can't indemnify yourself against gross negligence.
Re: (Score:3)
Indeed. And an invalid pointer that prevents booting certainly qualifies as gross negligence. Even simple (but complete) testing reliably finds such a thing. That testing was obviously not done.
Re: (Score:3)
Note that the online terms are what govern CloudStrike's relationships with small businesses where the size of the business doesn't justify CloudStrike's negotiating a separate agreement. Bigger companies negotiate their own agreements. If you're in the legal department at, say, American Airlines, you don't say "Oh, your extremely one-sided terms of service are perfectly ok with us." Instead, you hand CloudStrike your 40-page agreement that is very one-sided in your favor. And then you spend a couple o
Re: (Score:2)
I especially like section 5 where they have the right to exfiltrate your data and show it to anyone they want.
Kind of hard to have a crowdsourced malware engine if you can't do this. I get why it's problematic, but I don't know how you would word the terms differently.
But the really important part is section 7:
Not sure this will hold up in court. There are limits to what you can make people agree to in a contract. And nearly every contract has a severability clause so this can disappear in court with the rest of the contract still in effect. The maximum extent permitted by applicable law will also severely limit some of this, depending on the state.
Re: I can speculate without information, too (Score:2)
Re: (Score:2)
Good point, but let's also note: "...TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW..."
They can only indemnify themselves to the extent the legal system allows them to do so.
If the LAW says that those harmed are due some restitution, there's going to be a case made.
Terms and Conditions (Score:2)
DDS (Score:2)
Re: (Score:2)
Doctor of dental surgery?
Crowdstrike will pay, just not compensatory (Score:2)
Senior managers to IT security software people "How do we ensure this doesn't happen again?"
IT people: "we can't, until Crowdstrike changes how it deploys binaries." ie. customers can test updates first and control the rollout schedule.
Senior managers: "is this coming soon?"
If the answer is Yes, then you stick with crowdstrike. If No, many senior managers will say get rid of it. It's not worth the risk.
This is also going to put the spotlight on all apps where the user doesn't control the deployment of upda
CrowdStrike: Statement from our CEO (Score:2)
The outage was caused by a defect found in a Falcon content update for Windows hosts. Mac and Linux hosts are not impacted. This was not a cyberattack.
Maybe it was a cyberattack.
Either CrowdStrike failed in such a simple and trivial QA step as to check an update in its own computers or they suffered from an internal intrusion in order to provoke an intentional cyberattack.
To rule out the possibility of being a cyberattack, CrowdStrike must say something more than "a defect found in a Falcon content update for Window hosts."
In other words, how secure is the QA process in this company tha
Re: (Score:2)
The outage was caused by a defect found in a Falcon content update for Windows hosts. Mac and Linux hosts are not impacted. This was not a cyberattack.
This does send a message to $BogeyMan that they need to get one of their guys employed at CloudStrike. Just before they perform $EvilAct they release an "untested" update that hampers detection & recovery from $EvilAct.
OK: this might sound like paranoia but I do not regard it as impossible. There are other companies other than CloudStrike that would be worth $BogeyMan's attention.
One bit of me has been wondering if what happened on Friday was a dry run ?
The consumer (Score:2)
Society (Score:2)
The costs will be borne by society at large, AKA all of us.
What happened to redundancy? (Score:2)
I'd say if your systems are that critical, you'd have at least two independant instances running. If not on different OS, at least on different hardware and software.
Anti-malware or database or whatever screws up after an update, the other instance(s) with other vendor software keeps running.
Well, I know the usual answer. Plenty of years in the trenches behind me.
Isn't everyone affected guilt to some degree? (Score:2)
When it comes to who bares legal responsibility for the issue, why did Microsoft let Crowd Strike carelessly engage with the Kernel with untested buggy code? If the signing means a
Re: Isn't everyone affected guilt to some degree? (Score:2)
Large customers will get refunds (Score:2)
That's how this game is played. Nobody actually fires off lawsuits for real because they're too expensive. But it is going to cost the company that made the software millions in lost revenue from refunds and discounts.
Still going bankrupt (Score:2)
EULA (Score:2)
You can be sure that the EULA says, sorry you use this at your own risk. Which is why the idiots who used it shouldn't be made whole.
who will pay ? (Score:3)
3rd party kernel push updates stupid (Score:3)
CrowdStrike does not seem to allow their clients to adopt a strategy such as a "canary deployment". CrowsStrike decides when their clients get updates, not the clients. There have been warnings that went unheeded in this regard, such as: https://www.neowin.net/news/cr... [neowin.net] . The news seems to be downplaying the risks of CrowdStrike defections, but if CrowdStrike does not adjust their deployment methodology before companies figure out what is fundamentally wrong with the current approach, they are going to lose a lot more customers than is currently being predicted.