Delta CEO Says CrowdStrike-Microsoft Outage Cost the Airline $500 Million (cnbc.com) 90
Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million. From a report: Bastian said the figure is representative of not just the lost revenue, but "the tens of millions of dollars per day in compensation and hotels" over a period of five days. The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline. The company had to manually reset 40,000 servers, Bastian said. Further reading: Delta Seeks Damages From CrowdStrike, Microsoft After Outage.
Stating the obvious (Score:5, Insightful)
Re:Stating the obvious (Score:4, Informative)
Is there a precedent that says a big company can ignore the Binding Arbitration and No Class Action clauses in the Windows EULA and still sue in court anyways?
Also, I do believe the Windows Volume licensing program and the IoT Windows contain a Contractual clause that states you will Indemnify Microsoft from any Liability and Legal expenses including reasonable attorneys' fees that arise from your usage/distribution of the software. Including in High-Risk applications, which there is another clause that states that they are At your own risk, and the software is not guaranteed to be reliable. CrowdStrike's EULA has a similar clause about the Software not guaranteed to be fault-tolerant and is not designed for and should Not be used in (Long list of high-risk applications) or applications which are required to be fault tolerant
Re: (Score:2)
First, these are disputes between two parties so a class-action clause wouldn't apply. Second, volume license agreements and end-user license agreements are two different types of contracts. The airline must have a volume license agreement, and it's possible Delta was able to negotiate a corporate contract that does not include a binding arbitration clause. The FAA has to approve software used in aircraft or flight management. At least for those systems the airline simply cannot agree to boiler-plate lang
Re: (Score:1)
Re: (Score:2)
Does the FAA approved software include computers that operate booking systems and point of sales terminals?
I would think they would need to approve everything that affects air traffic controlling or the aircraft itself. Tickets don't keep planes in the air and they don't affect accidents on the runways.
I could see something pertaining to background checks of all passengers so a requirement to obtain appropriate passenger details that have not been doctored, but that's possibly outside of the FAA and more of
Re: (Score:2)
Re: Stating the obvious (Score:1)
Re:Stating the obvious (Score:4, Informative)
>The FAA has to approve software used in aircraft or flight management.
This is untrue on the flight management side.
Source: Worked at SITA for several years building flight management / cargo management systems. We used random FOSS software all the time. I brought in fail2ban at one point and there was no review other than does it work and is there proper documentation.
Re: (Score:2)
Re: (Score:2)
The applications in bold above sound like the type of applications that require security the most, and I imagine CrowdStrike's marketing material reflects that. If that's the case, then it would be another classic example of sales and marketing making statements that dir
Re: Stating the obvious (Score:1)
Re: (Score:2)
Leaving aside the fact that Microsoft is not a party to this, yes there's precedent that contractual clauses of all kinds get invalidated in the case of gross negligence. But the key word is "gross" negligence. That's a high bar to prove and simple negligence often doesn't negate any contractual clauses.
It's why a sign that says "Enter at own risk" can cover a house that is untidy, but not cover someone leaving an active beartrap right in the doorway.
Re: (Score:2)
Big companies like Delta don't just buy their software off-the-shelf subject to the same shrink-wrapped EULAs as the rest of mere peasants do. Their lawyers get together with the vendor's lawyers and hammer out a contract specific to that business relationship. And, of course, those agreements are confidential unless some court action forces them out into the public spotlight. So those "we accept no blame" and mandatory binding arbitration clauses may or may not apply here. And the most likely outcome i
Re: (Score:2)
Yeah, the precedent is that if you can spend enough money on lawyers, you can find loopholes in your "binding arbitration agreements" that regular people can't.
Re: (Score:2)
EULA's can't protect them from negligence claims.
Re: (Score:2)
Clearly something didn't work in their recovery process, however I don't think we should be pre-judging that. What if their recovery computer also had Crowdstrike on it? What if they had followed some Crowdstike advised process for recovery that other airlines didn't and that was what broke? Let's wait to find out the reason and then laugh at the right people.
Yes, you should have a recovery process from offline data that you have tested and if they didn't have one they should not win completely. No, you can
Re: (Score:2)
Fact is, Delta did not have recovery processes in place, nor did they have catch-up practices in place.
They also used illegal methods to make sure they didnt have to pay out to customers due to their failures.
Now they are looking to blame anyone else to try and get money back for their failures.
40,000 servers (Score:2)
Re: (Score:2)
Something tells me their IT system is extremely inefficient.
Don't most companies have a separate server for every 2.5 employees? :-D
Delta Airlines.... 4000 flights per day, probably 100 people per flight on average, and you figure that people book six months out at most, so basically one server for every 1800 customers over that six-month period, or one server per 18 flights.
If we assume a modest 16 GB of RAM and 4 CPUs per server (with zero disk space) at Azure prices ($121.18 per month), that's $29,083,200 over six months. If those assumptions are even remotely
Re: (Score:2)
Big opportunity in Bug Insurance (Score:2)
Why isn't bug and hacking insurance a bigger market? That seems a better way to handle these things than suing software companies, because software co's are often too small to pay out big settlements. And MS would rather eat live toads than accept all the blame for their bugs/leaks.
And insurers will get nosier about how software they insure works, encouraging better practices.
Re: (Score:2)
Re: (Score:2)
Certainly. You can insure anything, against anything. The rates may not be favorable... meaning that if the insurer feels the likelihood of a payout is too great, the premium may exceed the payout. For infrequent risks with potentially large payouts, the insurers insure each other (re-insurance) to spread the liability.
If my small business had been affected by this, I would be claiming under my "business interruption" clause. I am too small to need specific insurance for software/bugs/hacking -although
Re: (Score:2)
On the other hand, Delta has hired David Bois to represent them in this matter. Also, I doubt they licensed CrowdStrike one at a time just accepting the standard EULA when they already have a fair sized legal department.
Re: (Score:2)
Or maybe other airlines didn't use CrowdStrike to the degree Delta did?
Poor DR processes and false claims (Score:4, Insightful)
Not a Microsoft issue. A Delta Airlines, first and foremost, and CrowdStrike issue. Quit spewing nonsense you spineless, ignorant, buffoon
Re:Poor DR processes and false claims (Score:5, Insightful)
A CrowdStrike issue first, because they were negligent. Delta Airlines should have had a reasonable recovery procedure, but there will be some losses that were unavoidable by them.
The real question Is.. CrowdStrike's negligence so extremely severe that the Liability waivers in their Software EULA cannot bind Delta?
In other words: Does CrowdStrike's negligence count as gross negligence [cornell.edu], or is it just ordinary negligence?
Crowdstrike's actions would have to meet all 3 criteria at the same time: (1) Willful, (2) Wanton, and (3) Reckless conduct disregarding another person's life or property. Even the slightest care taken to avoid such thing happening would mean Ordinary, and their warranty disclaimer should be valid if so.
full image rollbacks with live data on the DB serv (Score:2)
full image rollbacks with live data on the DB server can be an mess. And this an outage that kills all of your HA at the same time.
Now maybe the DR process does not cover an bad update being auto pushed (with no control) to all systems at the same time?
Is manual editing of disk images not allowed as part of the process?
Is doing an mass remarriage and re-domain join going to over load the system so they had to do it slowly in groups?
Does reimageing of some system need an local tech to touch the system if jus
Re: (Score:2)
full image rollbacks with live data on the DB server can be an mess. And this an outage that kills all of your HA at the same time.
Sure, but this is just a Windows issue; it doesn't call for rolling back anything other than possibly the Windows OS Drive. They are still down long after the repair procedure is known which takes about a minute to do and does not require rolling back anything.
no update control or poor default settings? (Score:2)
no update control or poor default settings?
Now even having update control may not save CrowdStrike? even more so if they had some update control but it was only in some deeply hidden registry setting.
Re: (Score:2)
Re: (Score:2)
I don't think you can reasonably call out bitlocker here. It's providing a function for which there are inherent tradeoffs. Besides, bitlocker did not add much overhead to the recovery process. You should already have a system in place for looking up the bitlocker recovery passwords in a timely fashion.
I've heard CrowdStrike is claiming that a Windows bug caused the files to be written incorrectly, though I haven't seen that claim supported anywhere yet (or even the existence of the claim!) Besides that, it
Re: (Score:2)
I don't think you can reasonably call out bitlocker here. It's providing a function for which there are inherent tradeoffs. Besides, bitlocker did not add much overhead to the recovery process. You should already have a system in place for looking up the bitlocker recovery passwords in a timely fashion.
Typing a BitLocker key takes maybe a minute, yes. But it was a part that was manual. Finding the BitLocker key was the hard part. While MS in the cloud will store them for consumers, that process is a bit more complicated for Enterprise users. Also I am not certain but accessing those keys generally requires MFA which requires the company's authentication servers to be working. Guess which servers were also down? Windows authentication servers.
Re: (Score:2)
Typing a BitLocker key takes maybe a minute, yes. But it was a part that was manual. Finding the BitLocker key was the hard part.
If finding the bitlocker key is hard for you, you have already fucked up.
If you are a fuckup, nobody can save you from yourself.
Machines should have asset tags, there should be a database with the asset numbers associated with the bitlocker keys, and it should take you no time to find them. Ideally this resource is protected, so you might need to print out the keys specific to a site and take them with you, but if you don't know where your assets are located, see the part above about being a fuckup.
Re: (Score:2)
The real question Is.. CrowdStrike's negligence so extremely severe that the Liability waivers in their Software EULA cannot bind Delta?
It is highly unlikely that Delta is bound by the stock EULA. They most likely negotiated a sales contract with Crowdstrike with custom terms, which may or may not have included an SLA and possibly a removal of the liability waiver. The point is we do not know the license terms that bind Delta.
Re: (Score:2)
They most likely negotiated a sales contract with Crowdstrike with custom terms, which may or may not have included an SLA and possibly a removal of the liability waiver.
They probably Will have negotiated custom purchasing with Microsoft. And they could have negotiated special support contracts and volume license agreement addendums that subject Microsoft to additional commitments.
However, we do know that Microsoft does not bend when it comes to the terms that apply to the Windows SKU you get with your pu
Re: (Score:2)
Crowdstrike's actions would have to meet all 3 criteria at the same time: (1) Willful, (2) Wanton, and (3) Reckless conduct disregarding another person's life or property. Even the slightest care taken to avoid such thing happening would mean Ordinary, and their warranty disclaimer should be valid if so.
Sending (obviously) untested changes out clearly shows willful, wanton, and reckless disregard. It isn't even an argument.
Re: (Score:2)
Sending (obviously) untested changes out clearly shows willful, wanton, and reckless disregard. It isn't even an argument.
The rapid response update in question is a channel config file that is restricted to predefined Templates which are validated on the client, therefore, they may say that no testing is necessary. Delaying updates for manual testing can be extremely detrimental -- we are discussing fast updates designed to contain emergent rapidly-spreading ransomware; for example every 5 minutes upda
Re: (Score:2)
Obviously a Microsoft issue. They forced Crowdstrike to use a risky kernel module instead of providing a good API. That Crowdstrike messed that up is a separate issue. They are both at fault.
Incidentally, credibly accusing Microsoft of incompetence just got a bit easier. Apparently they cannot even do competent DDoS protection for their core services. Yes, different issue and different area, but "demonstrates character" as they say.
Re: (Score:3)
Something tells me that if your Linux kernel panics due to dumb things you are doing in kernel space you would not blame Linux for that.
To say that anyone is "forced" to meddle in the kernel is utterly ridiculous.
Like, I am *entiteled* to make money off of your product! If you don't allow me to have the same access that you, the developer of the software, then I will be *forced* to go around you! /eyeroll
Re: (Score:2)
You seem to be pretty clueless about the technological reality. Please stay out of the discussion.
Re: (Score:2)
Something tells me that if your Linux kernel panics due to dumb things you are doing in kernel space you would not blame Linux for that.
Your own fevered imagination.
To say that anyone is "forced" to meddle in the kernel is utterly ridiculous.
They must do some things through kernel interfaces on Windows which can be done through user interfaces on Linux. To accomplish the same things, you are "forced" to do them that way. This is perfectly acceptable English which you should really understand if you want to have arguments in this language. The person you're arguing with knows this, and it's not even his first.
Like, I am *entiteled* to make money off of your product!
It's "their" product in that they own the copyrights and collect the profits. But they also sell licenses for
Re: (Score:2)
They did nothing of the sort. Microsoft provides the same APIs as all OSes except MacOS. No one forced Crowdstrike to do anything. Get your head out of your arse.
Re: (Score:2)
Lying, lying and some more lying. There is an EU regulatory decision that directly contradicts you and that Microsoft had to accept. Do you have any shame?
Re: (Score:2)
As much as it pains me to have to defend the beast of Redmond, this one isn't on Microsoft. It is another European Union meddling with things they don't understand issue. [tomshardware.com] The EU forced MS, in 2009, to open up direct kernel-level access to random 3rd-parties like Crowdstrike. Under the law, they literally cannot fix this flaw and secure the windows kernel unless they exit the European Union market. So, as long as other companies are daft enough to go on using windows in critical systems, this is going
Argos 51% blinded: we're essentially in the clear. (Score:3)
"But isn't that a single point of failure?"
"What? No, absolutely not!"
"Why not?"
"Well, for one thing, there's like a million of them."
That's OK (Score:1)
Will Delta keep using it? (Score:2)
Probably.
So what next? (Score:4, Insightful)
Smart CEO moves away from Microsoft and orders tech department to look into migration to open-source solution? Yes?
Yeah... I didn't think so.
add more ways to provides customers with more cont (Score:2)
adding more ways to provides customers with more control over update delivery.
So customers had little to no controls over the definitions updates rollout timing?
Re: (Score:2)
There's a difference: when a Linux kernel goes haywire, you can downgrade it. Or patch it. Or figure out what the problem is yourself if it's that important and your company is big enough to have a sizeable tech team with knowledgeable people in it.
When a Windows install goes wrong, you pray that Microsoft fixes it. Or whoever fucked it up fixes it. Because it's a black box.
Re: So what next? (Score:2)
Yes, and as was pointed out to me the timestamps on the redhate solutions article show that the Kernel panics were occurring several minutes after boot, plenty of time to log in remotely and apply a fix. The Microsoft/Clownstroke failure prevented booting so a fix had to be applied manually.
These two things are not the same. The failure with Windows was much worse. It's arguable that the differing designs of the systems led to different types of problem.
Re: (Score:2)
These two things are not the same. The failure with Windows was much worse. It's arguable that the differing designs of the systems led to different types of problem.
Indeed. These two things are _not_ the same. On Linux this was an easy fix and one easy to automate. Also, on Linux, it would very likely be possible to do all this outside of the kernel. On Windows, not so much.
But the Microsoft apologists will use any and all lies to defend the indefensible crap Microsoft is selling. Kind of like climate-chage deniers, anti-vaxxers, flar-earthers. Too many people are just too fucking stupid.
Re: (Score:2)
the Microsoft apologists will use any and all lies to defend the indefensible crap Microsoft is selling. Kind of like climate-chage deniers, anti-vaxxers, flar-earthers. Too many people are just too fucking stupid.
Yes, all of that is easily explained by cognitive dissonance. They think they can't possibly be idiots, so they can't believe only idiots would believe what they believe.
Re: (Score:2)
Stop trying to shift blame. You just look incompetent. On Linux, there is no need to use a kernel module for this. This is fully on Crowdstrike. On Windows, you must use a kernel module, because Microsoft does not have the APIs to do things differently. And that is on Microsoft.
Re: (Score:2)
Crowdstrike had a kernel module that was also crashing linux boxes. It seems to be there thing.
Re: (Score:2)
Yes. It seems they are exceptionally lazy where actual risk-management and solid engineering is concerned. I don't think they even need a kernel module on Linux for what they do. On Windows they need it and that is on Microsoft, who set up this disaster.
Re: (Score:2)
It was, though it was crashing them well after booting so that you could fix them remotely.
eBPF (which CrowdStrike uses on Linux but not on Windows) might have stopped this particular crash from happening, if used correctly.
My employer just announced that they're dropping CrowdStrike (the contract apparently just expired) and moving to another solution... it's a fair number of seats, too. I presume this is going to happen, you know, a whole lot.
Re: (Score:2)
Smart CEO moves away from Microsoft and orders tech department to look into migration to open-source solution? Yes?
Yeah... I didn't think so.
To achieve what? The same thing has happened on both Debian and RedHat in recent memory. As usual the people who have no idea on computer security and reliability think the answer is to simply change systems while leaving flawed processes intact.
Re: (Score:2)
Smart CEO moves away from Microsoft and orders tech department to look into migration to open-source solution? Yes?
Yeah... I didn't think so.
I've got a friend I met years ago at another job and he, like me, is a long time IT worker. He told me that Delta outsourced the majority of their IT and that's the main problem they had issues coming back. If so, they aren't going to spend the money to replace stuff. They'll just look to blame others because the CEO won't be able to admit that outsourcing can be bad.
Movie Scene (Score:1)
One imagines the scene when the bad guy disappears into the secret door in Temple of Doom. While laughing of course.
Actually numbers are too small... (Score:3)
Doesn't appear that they have accounted for likely permanent business loss due to irate customers changing airlines permanently due to getting royally screwed over by the Delta multiple-day outage and either missing trips or getting stuck somewhere...
Re: Actually numbers are too small... (Score:3)
Re: (Score:2)
Doesn't appear that they have accounted for likely permanent business loss due to irate customers changing airlines permanently due to getting royally screwed over by the Delta multiple-day outage and either missing trips or getting stuck somewhere...
Sadly airlines are pretty much like telco's, you see that line of people over there, they've just sworn they'll never use our competitor ever again.
Michael O'leary, head of Ryanair famously quoted as saying "our booking system is full of people who said they'll never fly Ryanair again".
Al Capone and Tax Evasion (Score:1)
After producing an insecure, unreliable operating system for more than three decades, Microsoft is finally getting sued.
And the irony is that the reason for the suit is an outage caused by a third-party vendor. Which says more about our legal system and the nature of Microsoft's customers than anything else.
Al Capone wasn't prosecuted for being a gangster, but for tax evasion.
This doesn't bode well either way... (Score:2)
If there is a court case from this, this doesn't bode well for the population at large no matter what the verdict due to precedents set:
If the plaintiffs win, and it is found that EULAs mean nothing, then there will be no more F/OSS, because people can sue some guy who wrote an obscure program and GPL-ed it because they made it a cornerstone of their business, and it had a bug in it. In this world, Chinese and Russian software developers would survive, but anyone writing any code in the US would be like so
Re: (Score:2)
Ages ago (early 1990s as an intern), I worked for a startup where the CTO was wanting a dongle to guard their software. They didn't just want a parallel port dongle (USB wasn't around then), but something that went into an internal slot. If the software detected piracy, the dongle would dump the capacitors on it into the machine, frying it. Because of the EULA, the lawyers said the company would easily get away with it. However, thankfully this never took off, because other C-levels said that they don't want that bad PR.
Which of course wouldn't harm pirates, who wouldn't have the dongle card. It would only harm their real customers, when something went wrong and it fried their computer due to a bug. Age old story of anti-piracy harming/inconveniencing customers and doing nothing to prevent piracy.
Re: (Score:2)
Some things never change. The CTO was all about anti-piracy measures, but wound up getting ixnayed by the other C-levels, who just to spite the CTO, released the product with zero anti-piracy measures, because they believed that if someone wanted their product, they might as well have a good experience with it, so they would eventually buy it, and because the software was mainly for businesses, the threat of the SPA/BSA was enough to keep businesses off the high seas for the most part.
Re: (Score:2)
> the threat of the SPA/BSA was enough to keep
> businesses off the high seas for the most part.
Well, that's a toss-up. Anyone who's had to directly experience the headaches of a specious BSA audit because a disgruntled ex-employee fired off some fake anonymous tips would probably be very astute with their licenses and documentation. But the MBA types who infest many companies, and far too many C-Level cohorts, who know nothing about anything besides the right person to go golfing with in order to ge
Re: (Score:2)
USB killer for ISA and PCI?
Re: This doesn't bode well either way... (Score:5, Interesting)
"If the plaintiffs win, and it is found that EULAs mean nothing, then there will be no more F/OSS, because people can sue some guy who wrote an obscure program and GPL-ed it because they made it a cornerstone of their business, and it had a bug in it."
Not necessarily. The situation will still vary between things you paid for and things you didn't. Providing something for free with no warranty is different from providing something at a cost and then trying to put limits on liability.
One way to address this is to make publishers liable for their advertising claims. If they claim that their software will protect you and then it harms you instead, you should be able to sue them for fraud and win.
The fact that the masses simply accept false advertising as part of the landscape is pathetic. It is fraud by any reasonable OR legal definition.
Re: (Score:2)
Exactly. This will be about negligence on a paid-for service with a paid-for product. Entirely different than FOSS. Sure, if Red Hat sells a broken distro, they may get sued the same. But not the FOSS coder that may have broken things.
Re: (Score:2)
That is nonsense. Negligence works differently depending of whether you paid for a product and had a contract or not.
Re: (Score:2)
If there is a court case from this, this doesn't bode well for the population at large no matter what the verdict due to precedents set:
If the plaintiffs win, and it is found that EULAs mean nothing, [snip] If the plaintiffs lose, and EULAs are found to be watertight, then it means we will see more Draconian EULAs coming our way
I think there's a slim possibility of a positive ruling for all parties involved (well, not Crowdstrike): "The indemnity clauses in the EULA do not apply here because the update ignored the fact that Delta specifically opted for internally managed update cycles, rather than auto update".
If the case rests on this distinction, then neither possibility will be as egregious. FOSS can continue because the lawsuit rests on Crowdstrike ignoring customer policies, which is demonstrably the case. If a FOSS project i
Re: (Score:2)
If the plaintiffs win, and it is found that EULAs mean nothing, [snip] If the plaintiffs lose, and EULAs are found to be watertight, then it means we will see more Draconian EULAs coming our way
As I said upstream, it is highly unlikely that Delta is bound by the stock EULA we can see. Megacorps generally negotiate a contract specific to their needs. Given Delta's business, I would assume that they included a SLA in the purchase contract.
They should sue the ethernet cable manufacturer (Score:2)
Re: (Score:2)
I actually see merit in this lawsuit.
The ethernet cables weren't designed to selectively carry only "good" content.
CloudStrike was specifically sold as a tool that would block harmful content. Instead, it *became* the harmful content.
Centralization Has Its Advantages (Score:1)
How Microsoft could look at this... (Score:2)
Re: (Score:2)
Yes, probably.
Hmmm (Score:1)
I wonder what 50M UberEats gift cards would look like physically.
NOPE... choice of software and bad processes did (Score:2)
Install Linux (Score:2)
Problem Solved