Third Day of 1,000+ Cancelled Flights, Just in the US, After Crowdstrike Outage (cnn.com) 92
For the third straight day, "More than 1,000 US flights have been," reports CNN, "as airlines struggle to recover from a global tech outage that left thousands of passengers stranded at airports."
More than 1,200 flights into, within or out of the United States were canceled by early Sunday afternoon, while more than 5,000 U.S. flights were delayed, according to the tracking website FlightAware.com... On Saturday, 2,136 US flights were canceled, and more than 21,300 flights were delayed...
USA Today notes that Friday several U.S. airlines issued ground stops (according to America's Federal Aviation Administration) "which caused a domino effect into Sunday."
They note that "most of the cancellations and delays Sunday are likely to be caused by airline crews and equipment being out of place."
USA Today notes that Friday several U.S. airlines issued ground stops (according to America's Federal Aviation Administration) "which caused a domino effect into Sunday."
They note that "most of the cancellations and delays Sunday are likely to be caused by airline crews and equipment being out of place."
Good thing it's private industry (Score:5, Interesting)
Can you imagine if this was the result of something the government did? Can you imagine the uproar, the Congressional investigations, the finger pointing, the firings? People would want heads to roll.
Good thing this only happened because of private industry.
Re: Good thing it's private industry (Score:2)
Re: (Score:2, Troll)
We knew and know what it is, but it has been destroyed in the past 20 years... regulation. FTC/FAA and the Federal Government have had rules in the past that are more customer/consumer centric. If airlines fail to help the customer, there were fines and fees related to it as well as a requirement by the airline to make it right for the customer. Wasnt always 100%, but it was something to make airlines come up with contingency plan
Deteriorating IT infrastructure (Score:3)
It's rusting
- Putting systems "on other peoples computers" (cloud) is a risk increasing proposition as another /, user recently commented
- Since 2014+ computer and software vendors 'sharing by default' and 'user behavior is the monetized product'
- Rotation out of older hyped technological saviors (cloud, cryptocurrency) for new hyped technologies (AI everything)
- Expectation that "yet another level of indirection will solve things" https://en.wikipedia.org/wiki/... [wikipedia.org]
- Expectation that always adding complexity
Re: (Score:2)
Luddites are hoping for progress, are they?
Well, that's new.
Re: (Score:3)
In order for the benefits of capitalism to be realized, there must be competition. But there can never be a winner.
As soon as one company dominates the industry, all the benefits go right out the window. Prices go up, quality goes down, and barriers-to-entry are erected to ensure that no competitor can ever get a foothold.
So, monopoly-busting is just a necessary form of "economic hygiene." It's hard to do, since the ones who suffer from it are the most wealthy and politically powerful people in the count
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Insightful)
Can you imagine the uproar, the Congressional investigations, the finger pointing, the firings? People would want heads to roll.
Do you seriously think that's not going to happen after this?
Re: (Score:3)
Can you imagine the uproar, the Congressional investigations, the finger pointing, the firings? People would want heads to roll.
Do you seriously think that's not going to happen after this?
Still waiting on the uproar from the plebs, not a peep out of Congress so far, and no firings that we are aware of. So yeah, let me know when this happens because the chances are slim to none.
Re: (Score:2)
not a peep out of Congress so far,
This literally just happened a couple days ago. How fast do you think Congress moves?
Regardless, even if you don't think they'll investigate for the right reasons - they'll probably call hearings once they learn it's cost them money [msn.com].
Re: (Score:2)
Indeed, it will. The stock-market losses Crowdstrike itself already had will be peanuts compared to the overall losses.
Re: (Score:3)
There was not enough time. The damage to customers will probably go into the two-digit billions, maybe three digit. This is probably not enough to finally get real vendor liability for when they screw up (and the code analyses I have seen say they likely screwed up massively by not even doing reasonable testing), but it could and should drive things into that direction.
In all other engineering disciplines, actual liability, requirement that qualified engineers are used, certification requirements, etc. all
Re: (Score:3)
Do you think congress has any understanding at all about technology?! They do what their lobbyists tell them to do.
What I want to happen is for the Cyber Security Czar or whatever to push for regulation on the core things that made this event possible, across all agencies. FTC is an obvious start, but GSA creating policies that require vendors to have diversified systems and procedures to limit the impact that any single vendor can have on operations. Then let it funnel down to the FAA to build on those
Re: (Score:2)
not a peep out of Congress so far
Ahem...
https://yro.slashdot.org/story... [slashdot.org]
Re: (Score:2)
Can you imagine the uproar, the Congressional investigations, the finger pointing, the firings? People would want heads to roll.
Do you seriously think that's not going to happen after this?
Do you seriously think anyone's head would roll because of this?
Just look at history, how many heads rolled after the 2008 financial crisis? Or any major crisis in the last 30 years?
Re: (Score:2)
Hundreds of people went to prison for their roles in the Savings and Loan scandals.
No, nowadays accountability is something that no businessperson should ever have to face in the U.S.
Re: Good thing it's private industry (Score:2)
Re: (Score:2)
I agree. The head of the Secret Service was accountable within 24 hrs of an assassination attempt.
Re: Good thing it's private industry (Score:1)
Re: Good thing it's private industry (Score:1)
Not quite (Score:5, Informative)
They note that "most of the cancellations and delays Sunday are likely to be caused by airline crews and equipment being out of place."
Delta is incredibly hard hit. The story seems to be that just about every single windows computer in their organization was impacted. Servers, customer service desktops, operatiosn desktops, check-in kiosks, airport information terminals, etc.
The solution is booting each computer in safe mode, typing a 40 character Bitlocker password -- that's written down in a physical binder -- running a handful of commands to delete the bad update files and then rebooting. (So I've read)
Many pilots, flight crew, etc., are saying that 48 hours in, they have not heard from anyone at Delta, and the wait time for the CREW phone numbers is 8-10 hours. There are pilots and crew waiting to fly, but who can't get approval. The CEO even released a statement blaming a vendor update that impacted their crew tracking software. So, for Delta at least, it's not just repositoning equipment, they don't even know where their people and equipment are!
What a fuck up. Where is the testing before rolling out updates to tens of thousands of computers?!
Re:Not quite (Score:5, Informative)
Delta is incredibly hard hit. The story seems to be that just about every single windows computer in their organization was impacted. Servers, customer service desktops, operatiosn desktops, check-in kiosks, airport information terminals, etc.
Back in the 90s, I had a job working for a company that provided software services to the world airline industry, so Delta was among our US customers. Remember how in this decade it seems like every year Southwest Airlines had some major hardware malfunction and it screwed things up for them for days? Back in the 90s, that was Delta. Every year or two they had a horrific meltdown of their computers that brought everything to a halt. Either Delta's current CEO or the guy before him did a huge IT upgrade for Delta and I figured, hey they probably finally got off those ancient mainframes they were using. Well, looks like instead of using Linux, they went 100% in for Windows and well, that's what you get. Honestly, that's the kind of IT decision I expected out of American Airlines, not Delta. And American had major problems too with this Crowdstrike issue.
Re: (Score:2)
Bring the ancient mainframes back into service.
Re: (Score:3)
I'm not sure Linux really solves the problem though. Airlines clearly need an independent recovery system for when there is a problem.
I remember a bank I did some work for needing to make a decision to migrate the hot standby to Iron Mountain after an electrical failure forced them to run off diesel generators with no UPS for 2 weeks. Their concern was that it was a one-way migration; it could be done in 5 minutes, but recovering after the decision would take years. It would also cost them about 50% of t
Re: (Score:2)
Linux can be configured to create a volume-level snapshot before every software update is applied. Those snapshots can then be easily selected from a menu during the boot process. If this was in place, anyone who knows how to use the arrow keys to navigate down the menu and press the Enter key could have gotten the machines to boot. With Windows, a person may have to find and manually enter a 48-character BitLocker key, start safe mode, log in as an adm
Re:Not quite (Score:4, Informative)
Crowdstrike had a linux incident a few months ago. They wouldn't be safe from crowdstrike on linux either.
Snake oil (Score:2)
Yeah. The huge difference is that this kind of snake oil is not needed on Linux.
Re: (Score:2)
Crowdstrike had a linux incident a few months ago. They wouldn't be safe from crowdstrike on linux either.
OK, but could that problem be fixed remotely or not? I've seen the problem description (a kernel panic on a specific 5.x kernel on redhat) but not the solution, because redhate hides it from non-customers.
Re: (Score:2)
They broke Debian Stable with an update that caused boot failures, too. That needed local access to the machine to fix (well, unless it's a server with lights-out management, or a managed workstation with vPro boot redirection or whatever AMD's equivalent is called).
Re: (Score:2)
What I'm getting here is that their software should not be trusted near anyone's business... but thank you for specifics.
Re: (Score:3)
Well, looks like instead of using Linux, they went 100% in for Windows and well, that's what you get.
If you think computer security and stability simply comes from not using Windows I really hope you're not still providing software services, especially given that this was a 3rd party software related issue.
Here's a hint: Most of the world's airlines are 100% windows shops. Most were operational again by the afternoon. So no, clearly "100% Windows" isn't anywhere near as important as having a competent IT department along side a well thought out business continuity and disaster recovery plan.
I did an audit
Re: (Score:2)
>> The idea that simply not using Windows makes someone safe from disaster is one of the dumbest and most dangerous ideas in the IT world.
Probably in second place, just after the idea of using windows.
Re: (Score:2)
They were operational by the afternoon but this is the 3rd straight day.
Understood. Your guy from the linux story sounds like a tool but I dunno man usually when I run into a Windows shop, they didn't travel that road via competent IT.
They got there by believing salesmen and trying to save money on IT labor. etc. When I heat recruiters say "all windows shop" it's a red flag to me.
Re: (Score:2)
Your guy from the linux story sounds like a tool but I dunno man usually when I run into a Windows shop, they didn't travel that road via competent IT.
The opposite of what I said was not be a windows shop. The opposite of what I said was to have competent security. Simply running Linux is not competent security. The people who push that are often the ones who *don't* understand what security actually is.
It's like that ol' trope of "I run Linux and therefore can't get viruses" bullshit. Simply running Linux isn't the answer, user competence, network security, and antivirus systems are.
When I heat recruiters say "all windows shop" it's a red flag to me.
Which is a dangerous view. Security is not about what OS you run. If it
Re: (Score:2)
Well yeah "I run linux/mac so I can't get viruses" is dumb yeah.
Which is a dangerous view. Security is not about what OS you run. If it is, you don't understand security.
The OSCP is mostly windows nowdays and pentesters love windows machines for a reason. But security isn't the reason all windows is a red flag. It's a red flag because they had a choice between paying more for staff or paying more for software and they chose software.
Re: (Score:3)
What a fuck up. Where is the testing before rolling out updates to tens of thousands of computers?!
The roll-out was automatic and, bonus!, apparently even applied if you had automatic updates disabled. Such a great design! It apparently then caused a segmentation fault on boot due to a defective jump-vector. So really bad or no testing, because that should not even be possible in a well-tested system. Certainly not in a part of the system that blocks booting up.
Re: (Score:2)
Re: (Score:2)
Sure, the testing by Cloudstrike was either crap or missing. That was their first gross mistake. But testing on the customer side was apparently not even possible because the update got pushed to every installation, regardless of whether automatic updates were disabled. I would call that "gross negligence" or "criminal negligence" and Cloudstrike should be liable for any and all damage caused and tripple damages on top of that.
It is time we take IT engineering out of the kindergarden it currently is in and
Re: Not quite (Score:1)
Re: (Score:2)
Indeed. Licensing, qualifications and liability. This crap-show has to stop. It is far, far too expensive doing things this cheaply.
Re: (Score:2)
You are an idiot if your production environment is on automatic that you don't control. Yes that means it should NOT be running on an operating system that you don't control the updates. Including the automatic ones.
What a shit show.
This has not affected any production system that I manage but I'm expecting to get calls for office systems on Monday.
Re: (Score:2)
I agree. In this case this means you are an idiot if you use Cloudsource.
Re: (Score:3)
The roll-out was automatic and, bonus!, apparently even applied if you had automatic updates disabled.
This was NOT a Windows update. This was the result of a bad configuration file, pushed to the Crowdstrike security software, which runs in Ring 0 and loads during the device driver initialization phase of the boot process.
If you want a detailed overview of the problem and why it happened, watch this video from Dave Plumber, a retired Microsoft software developer. Dave's Garage -- Crowdstrike [youtube.com]
Re: (Score:2)
So what? Windows provided the setting, culture and abysmally insecure OS that needs things like that. Stop trying to protect them. They are 80% at fault here.
Re: (Score:2)
Yeah, it's funny, all the non-techies I know just call it "the Microsoft crash". Most of them don't even know that a third party was involved and have never heard of Crowdstrike.
Re: (Score:2)
But they're not wrong. After all, Microsoft's decades of insecurity begat Crowdstrike, and Microsoft's shitty kernel APIs provided the context for this accident waiting to happen.
Re: (Score:2)
Blaming Microsoft for Crowdstrike's failure is like blaming Ford because the aftermarket turbocharger you installed ruined your car's engine.
Yeah, Windows has some security issues but Crowdstrike is trying to protect against them. If you install a better garage door opener in your home and it fails and locks you out, you don't blame the home builder for designing the house with a garage door.
Re: (Score:2)
Hmm... there's also a sticky note here that says "agile" means "move fast and break things", and that the person who wrote this trite was a Microsoft spokesperson.....oh well, that's not important.
Re: (Score:3)
I don't want any software that won't let me decide when updates happen and allow me to first deploy to a test dummy, then small batches so I can stop the rollout if something like this happens.
Re: (Score:2)
What a fuck up. Where is the testing before rolling out updates to tens of thousands of computers?!
What a fuck up. Where is the testing before rolling out updates to tens of thousands of computers?!
What a fuck up. Where is the testing before rolling out updates to tens of thousands of computers?!
It bears repeating almost infinitely. Well said.
We do testing with operating system patches, but we didn't do it here.
Why?
Partially because we abdicated our responsibility as professionals and partially because Crowdstrike pushed each and every customer HARD to accept bleeding edge updates. They did it by saying, "N is the current version, you can only choose N-1 or N-2, so you are defended against cutting edge mishaps."
What they failed to tell you was that N-1 was actually N. Good job Crowdstrike.
When the Keystone Kops learn to code... (Score:5, Insightful)
Re:When the Keystone Kops learn to code... (Score:5, Insightful)
For the individual company your counter will fall on deaf ears because the issue was actually quite minor in comparison to what security software is designed to avoid.
What is worse: An individual company having to call in IT staff over a weekend to fix some computers after a day or two of outage, or an individual company dealing with the fallout of a major data breach or worse, ransomware lockout of all their IT systems.
Yeah it was a huge fuckup, but it is hardly worse than the alternative.
Re:When the Keystone Kops learn to code... (Score:5, Insightful)
One alternative is simply, practicing best security practices, keeping your systems up to date on security patches, not exposing unnecessary services and hardening your systems manually, proper admin training. Rather than relying on some bolted on top software to keep the computers safe. Those people have neither of these problems.
Re:When the Keystone Kops learn to code... (Score:5, Insightful)
keeping your systems up to date on security patches
But that's what happened here. Everybody updated to the latest version all at once, and the latest version had a problem.
The only fix for this kind of problem involves testing patches for a few weeks before deployment and/or staged rollouts. Nobody does that for security updates because... we need to push out security updates NOW NOW NOW.
Doesn't matter if it's the OS or an application. Doesn't matter if it's security related. Deploying a single update to every computer in the world instantly is just a bad idea.
Re: (Score:2)
The only fix for this kind of problem involves testing patches for a few weeks before deployment and/or staged rollouts.
The fix for this problem is for Crowdstrike to do meaningful testing before distributing an update, which they obviously didn't do. They clearly a) didn't test, or b) don't have enough diversity in the test pool.
After this, many Crowdstrike customers (any smart ones, that is) are going to disable automatic updates and manage those updates manually. This means a lot more work for them. It also means less security, because of delaying of updates, but Crowdstrike can't be trusted to do their job correctly so i
Re: (Score:2)
One alternative is simply, practicing best security practices
Yes, but that means not running windows, and most organizations are dependent on it.
Re: (Score:2, Insightful)
Why is Windows so insecure by design in the first place?
The fact that you need some external software installed deep within the bowels of the operating system to protect you from nasties is well, ludicrous.
I would be suspicious of using any Microsoft operating system that required patching with special 'trust us' vendor software to plug the holes.
Re: (Score:2)
"Why is Windows so insecure by design in the first place?" - Because Microsoft has no idea how to properly implement it. Gates years ago said it was "top priority" after a bunch of fuck ups, then just a few years ago Nadella said the same thing after another series of fuck ups. People need to learn that Microsoft Windows is a huge security hole in itself. Trying to keep hackers out of a Windows network if futile, not matter how much third party security bullshit you add to it. I ditched Windows when version
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
No. This brings me business because it didn't affect me or the systems I manage.
This happens when you don't want to understand your business and expect some stranger's magic potion will fix it for you. Why would a company be dealing with a data breach? There are ways to prevent the breach in the first place.
FUD is what sells just like your post. I haven't been affected by the alternative either. Not because I believe it will never happen to me but because I need to understand how my business works and what
Re: (Score:2)
This happens when you don't want to understand your business and expect some stranger's magic potion will fix it for you. Why would a company be dealing with a data breach? There are ways to prevent the breach in the first place.
Maybe. So this stuff is mandatory according to a lot of agreements and frameworks. It's not a bad idea either, the problem is that through the history of these products there's always been an incentive to be shitty.
To make matters worse, you might think, yeah man I'll use clamscan, tripwire, app armor and so on and piece together something that meets requirements.
Well you could but that's more work and no executive in the world is going to understand "I built our endpoint protection myself because I didn'
Re: (Score:2)
I've been through many security audits and basically it's "cover your ass" rather than evaluating security. As long as you show that you went through the process you pass. Showing that your process has a black box, like antivirus software, meets the bill. What that black box does is not important. Just that it's there.
Re: (Score:2)
FUD is what sells just like your post.
Okay you're right, my post was FUD. I'll stop applying security patches in a timely fashion. I appreciate your advice on good security practices.
Re: (Score:2)
FUD is what sells just like your post.
Okay you're right, my post was FUD. I'll stop applying security patches in a timely fashion. I appreciate your advice on good security practices.
That's not what your post said. We can't read your mind.
Re: (Score:2)
The onion analogy has a place here. Different layers have different risks, and the glue between layers is a part of the equation.
It would not be a complete failure if all desktops in a department needed to be rebooted 15 times to resolve this. It would not be a complete failure if routers were down for a couple hours to do a similar procedure. The same even goes for your external-facing servers, as it does for your internal servers. The problem is that all of the layers are treated the same and use the
Re: (Score:2)
Technically this amounts to a ransomware lockout, just with no ransom... Recovery being dependent on the existence of recovery procedures. So not that much of a difference.
Re: (Score:2)
An old school way of discussing Security is "CIA".
Confidentiality
Integrity
Availability
Everything except Confidentiality was violated here. You can "minimize" the events here; however, without availability, you have no business to conduct, making the rest of the triad pointless.
This was a serious blow and minimizing the events will lead us back to the same place again, but a few years later.
Re: (Score:2)
"We'll secure your computer.
Just give us unfettered kernel-level access to all computer functions, whitelist our software, allow us to download arbitrary, unknown and encrypted updates for the Internet, any time we like, apply them to all your computers without any manual intervention, and give us complete access to your network access, filtering, filesystems, network storage, permissions, etc. so we can do that."
It was always a dumb idea and that it still propagates as "the accepted norm" is dumber still.
W
Coverup Theory (Score:1)
There's a theory [x.com] that somebody needed to run some deletion commands in a recovery environment with no auditing active.
Particularly not a great thing for certain people (Score:5, Insightful)
Re:Particularly not a great thing for certain peop (Score:5, Insightful)
I should add that the fucked up quality of work and lack of QA/QC in this matter is very likely the result of DEI hiring. What a stupid idea!
This is what Fox does to your brain.
Here's a more likely scenario. The suits notice that the QA team is something that doesn't generate revenue. Since all that matters is short term profits they either axed the team or severely cut back. Here we are.
Re: (Score:3, Funny)
Yes but was the QA/QC team brown? Fox viewers are dying to know!
Re: (Score:2)
There is no QA, QC team. None.
Re: (Score:3)
But there must be a brown guy we can blame this on somewhere. What will our viewers think if we can't peddle "the message".
Re: (Score:2)
It seems unlikely that Crowdstrike will survive as a business entity, once the legal actions begin.
Re: (Score:2)
When the dust settles on this, a bunch of people are gonna be fired from Crowdstrike and they may face serious legal consequences.
Trust was broken. There are going to be FAR more consequences than a few people being fired and some money changing hands.
Crowdstrike: We protect your computer systems... (Score:1)
Re: (Score:3)
in their defense, they made all those computers safe against attacks.
Job done.
Mission accomplished.
Why they don't leawe that Windows for the past? (Score:4, Interesting)
Re: (Score:1)
Having systems that always work doesn't make news or get promotions. In fact people forget that the reliable systems, developers, and admins even exist.
Re: (Score:3)
Re: (Score:3)
Ding! Ding! Ding!
I worked in a windows shop and it was a weird place, nobody knew any unix, nobody had a college degree, basic scripting was considered a god tier skill, and absolutely anything that could be outsourced was. I asked about linux several times and eventually it was revealed that IT management didn't know you could get a support contract. (and probably the unspoken fact that I would have been the only one there who could work on it)
Re: (Score:2)
First of all, taking the security record of Windows, why such big companies still have Windows on their employees PCs? But most of all, servers? Just why?
Why not take a look at the security record. The vast majority of big companies have no security issues with Windows. Many people hear about a bug or security exploit aaaannnnndd it's gone, turns out an update got pushed but our "trusted partner Microsoft" *pats self on back for good choice in trusted partners*.
That's how the world functions. Not by news cycle but by impact. Most businesses do not feel security impacts from Windows - especially those with functioning IT departments who have their computers w
Re: (Score:2)
>Why not take a look at the security record. The vast majority of big companies have no security issues with Windows
Why not ask penetration testers if they'd rather deal with windows or linux? It's gotten so much that the OSCP certification is mostly windows now.
Most likely all businesses have security problems with windows and never notice.
Or... 2% of flights. (Score:2)
I mean, could be worse.
I had to re-read the opening (Score:2)
Like that classic I had to the network