CrowdStrike Unhappy With 'Shady Commentary' From Competitors After Outage (arstechnica.com) 107
CrowdStrike's president hit out at "shady" efforts by its cyber security rivals to scare its customers and steal market share in the month since its botched software update sparked a global IT outage. From a report: Michael Sentonas told the Financial Times that attempts by competitors to use the July 19 disruption to promote their own products were "misguided." After criticism from rivals including SentinelOne and Trellix, the CrowdStrike executive said no vendor could "technically" guarantee that their own software would never cause a similar incident.
"Our industry is built on trust," Sentonas said. For rivals to take advantage of the meltdown to push their own products "lets themselves down because, ultimately, people know really quickly fact from, possibly, some shady commentary." Texas-based CrowdStrike had a reputation as many major companies' first line of defense against cyber attacks, but the high-profile nature of its clients exacerbated the impact of July's global disruption that shut down 8.5 million Windows devices. Insurers have estimated that losses from the disruption, which grounded flights and shut down hospital systems, could run into billions of dollars. Delta Air Lines, which canceled more than 6,000 flights, has estimated that the outages will cost it $500 million and has threatened litigation.
"Our industry is built on trust," Sentonas said. For rivals to take advantage of the meltdown to push their own products "lets themselves down because, ultimately, people know really quickly fact from, possibly, some shady commentary." Texas-based CrowdStrike had a reputation as many major companies' first line of defense against cyber attacks, but the high-profile nature of its clients exacerbated the impact of July's global disruption that shut down 8.5 million Windows devices. Insurers have estimated that losses from the disruption, which grounded flights and shut down hospital systems, could run into billions of dollars. Delta Air Lines, which canceled more than 6,000 flights, has estimated that the outages will cost it $500 million and has threatened litigation.
There's an easy way to check... (Score:5, Insightful)
end users need update control (crowd strike had no (Score:5, Insightful)
end users need update control (crowd strike had no control over role outs)
Re:end users need update control (crowd strike had (Score:5, Insightful)
crowd strike had no control over role outs
What does this mean? Did their developers not write the code that rolls out updates to customers? Do their administrators not have any control over whether an update is released for production and, if so, to whom? If what you say is true, then they deserve to be obliterated from orbit.
Re: (Score:3)
You're both right but you're each talking about a different phase of the roll-out process.
CrowdStrike rolls out updates to its customers.
Customers roll out those updates to their systems.
In a consumer / "non-enterprise" context those tend to be one step. Microsoft, for example, makes a patch available and Windows Update automatically installs it.
But in mission critical systems it is common to treat any system changes as inherently risky. It's not at all uncommon to see air-gapped legacy systems that are run
crowd strike had no way to manage updates on the e (Score:2)
crowd strike had no way to manage updates on the end user side other then blocking it at the firewall level.
No way to have test groups, or set update windows.
Re:end users need update control (crowd strike had (Score:4, Insightful)
It's worse than that. Most Crowdstrike environments actually do have ways to test roll-outs. You can have one group that gets the updates as soon as they are available, then you have the majority of your install base on N-1 meaning they are always one version behind.
This however doesn't apply to definition updates and that's the rub as I believe it was the definition update that borked the whole process just like the Linux version did a few months prior.
That is the reason a lot of customers are looking to jump ship, they clearly had a problem, knew about it in April, and fell into the same trap in July. For a security company who's main currency is trust, that's simply not acceptable.
Re: (Score:2)
But then the user will keep hitting "postpone" until they get infected with a zero-day, and the CTO will cover their arse by blaming Crowdstrike.
They can't win. Timely updates occasionally brick your computer, delayed updates leave you vulnerable to malware (and their corporate customers are juicy targets for phishing and ransomware).
Re: (Score:2)
having end users hit postpone is not the same as domain level WSUS role outs or even an away to set only update at X time.
Re:There's an easy way to check... (Score:5, Insightful)
Re: (Score:3)
This narcissist salesperson is creating doubt in minds. It'll be marginally efffective, and repeated over time Crowdstrike will be able to overcome what should have been a business-ending incident.
Re:There's an easy way to check... (Score:4, Insightful)
The narcissist salesperson isn't doing shiat compared to what Crowdstrike did to their customers.
I had to help with my org's response. Thankfully we're small enough that the site visits for those few devices that required site visits were regional, not national in scope, and we were able to deal with most non-end-PC endpoints without having to physically touch hardware, but it was still a PITA and affected mobile end-user-endpoints in addition to servers and normal on-prem enterprise PCs. Real egg-on-face even with getting priority backend systems working quickly.
If Crowdstrike wants to avoid competitor's sales staff belittling them then they need to stop giving legitimate reasons to be belittled.
Re: (Score:3)
I don't disagree but these kind of leeches in sales are able to keep even the worst products in circulation for far longer than seems appropriate based on their efficacy.
This is just how the game is played.
Re: (Score:2)
I am well aware of that, I've been to Cisco Executive Briefings at their Tasman Way "Customer Experience Center" twice before they sold off that property when they moved their headquarters.
Plying the executives that went along with drink, food, and swag is stupidly effective.
Re: There's an easy way to check... (Score:2)
They're not wrong. But they're also guilty of it. As a security practitioner myself, I well know just how much the sales side of the industry loves to use FUD to peddle their crap. And most of what they sell is exactly that. Even worse, they resort to spamming, something the industry itself is supposed to be preventing, not participating in. I'd put security salesmen right below dog shit, and right below security salesmen are car dealerships.
Re: (Score:3)
There have been multiple surveys in which people have said they're prefer living next door to a child molester over living next door to a spammer.
Re: (Score:2)
Re: (Score:2)
No mention of why you shouldn't do file parsing in ring 0?
Re:There's an easy way to check... (Score:4, Informative)
And if you do proper input validation, like _all_ competently written code should do and even more so anything security critical, then that partial rollout has a pretty good chance of not failing either. These people messed up on an unbelievable level. Gross incompetence is to weak a term for it.
Re: (Score:2)
This message keeps getting lost. Too many complete idiots out there.
Not just input validation.... but WHERE you do the validation, let alone the loading/parsing itself. Certainly not in ring0, and also a file that is remotely installed.
Beyond stupid.
Re: (Score:2)
Indeed. Of course, you need to validate all input in ring0 as well, if you have input. But that should always be an additional barrier, not the first one.
These people do not even know the very basics of writing secure and dependable coding.
hahahahahaha (Score:5, Funny)
That's what they get for firing me from my internship last month just cause I made one lousy mistake.
Re: (Score:2)
Re: hahahahahaha (Score:2)
They're making a joke that an intern caused the whole thing. But saying they actually worked there.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
mid-life career change?
Re: (Score:2)
This lower UID was able to recognize the joke [youtube.com]
Re: (Score:2)
He's a *career* intern. :-)
Re: (Score:2)
Hey career intern is a noble profession right up there with doctor, engineer, and career student.
Re: (Score:2)
I never thought I'd face age discrimination on slashdot of all places -- the safe space for nerds. After decades on slashdot, I finally climb out of my bed (or is it a sofa?) of Cheetos wrappers and Red Bull cans, get a hair cut and shave my neck beard, to get myself a job but I still can't get no respect.
Re: (Score:2)
but I still can't get no respect.
You have all my respect for being able to dwell in your parents' basement for this long!
:-)
All joking aside, I get the joke now
Re: hahahahahaha (Score:2)
Re: (Score:2)
Hot take that everyone is going to hate: Phased rollouts excuse poor design decisions. It's a crutch. It allows devs to do stupid things with the excuse "oh, any problems will get caught in staging".
Please, stop with this idiocy as a "solution". It's a shit mitigation. Necessary, yes. The actual solution? Fuck no.
Re: (Score:2)
Re: (Score:2)
Hot take that everyone is going to hate: Phased rollouts excuse poor design decisions. It's a crutch. It allows devs to do stupid things with the excuse "oh, any problems will get caught in staging".
Please, stop with this idiocy as a "solution". It's a shit mitigation. Necessary, yes. The actual solution? Fuck no.
And besides, if the point of the service is to be able to block malicious activity that's happening live in the field, then you really don't want to be waiting to implement those responses. A firewall has to be reliable enough to block live packets, without being crashed by "surprises".
Re: hahahahahaha (Score:1)
As someone who was CrowdStriked (Score:5, Insightful)
Shady? Really? Don't make my life hell and you can start talk.
A good product sells itself! (Score:3)
Zero sympathy. (Score:5, Insightful)
A (horrible) mistake was made. Own it. Don't complain that your competitors are pointing out how they're different. Complaining just makes you look shady, and like you don't really understand your mistake so... why would we trust you really understand the scale of change you need to make?
Re: (Score:2)
Re: (Score:2)
The fact that they aren't owning the fuck up only makes the matters worse.
What they should be doing is having a "good will" campaign, and being very public about the changes they're making to make sure this doesn't happen again. They're not doing that - they're doubling down.
Re: (Score:2)
I think their point is that the competition is talking out of it's ass. As if a botched updated from any of the other vendors or a false positive on a critical system process or file couldn't happen... None of them should be talking right now, and instead looking at how to strengthen their internal processes.
Anyone that has worked with any of these vendors knows that it's a razor edge we play on and any small f'up can blow up in your face. And sometimes, the f'up isn't even their own fault (the security
Re: (Score:2)
I think their point is that the competition is talking out of it's ass. As if a botched updated from any of the other vendors or a false positive on a critical system process or file couldn't happen... None of them should be talking right now, and instead looking at how to strengthen their internal processes.
Granted, partially. But I can offer that the EDR/MDR vendor my company recommends absolutely does staged rollouts of updates, starting with their own systems. Anyone can screw up, yes. But the magnitude of screwup was much, much larger than it should have been.
Their PR rep should have told them to just shut up and take the hits, focus on pushing the message that they are putting in processes and systems to ensure this won't happen again, are working closely with OS vendors on how to limit impacts in the future, etc... not going to rebuild the reputation any other way.
Wholeheartedly agreed here. This is Crowdstrike being tone-deaf.
Re: (Score:2)
Exactly.
Nevermind the fact that they're gaslighting about "trust". Really, bro? You just crippled the global economy for a week. All travel was down for days, and a great deal of finance, was down for the better part of a day. It was a full week before things were 'back to normal'.
Definitely a company to avoid. I'd short the shit out of their stock if I wasn't so poor.
Re: (Score:2)
Of course they'll say that. But the other restaurants probably didn't check up on their own chefs before saying that.
Meanwhile, the poop restaurant is checking constantly now.
Which one is less likely to become the next fecal point?
Re: (Score:2)
Of course they'll say that. But the other restaurants probably didn't check up on their own chefs before saying that.
Meanwhile, the poop restaurant is checking constantly now.
Which one is less likely to become the next fecal point?
Probably. The other restaurants probably put twice as much poop on their food. And probably send teams to rob their customers' houses while they dine. And probably the profits all go to fund research to create new diseases that only impact children.
Which one is less likely to become the next fecal point? The restaurant that has demonstrated they don't have an anti-fecal-fetishist-cook policy. Meanwhile, (at least some of) the other restaurants have very carefully explained that they DO have such poli
"no vendor could "technically" guarantee" (Score:3)
Re: (Score:2)
Yeah, his major point is correct, that nobody can guarantee they won't have a problem just as bad, but to expect that the competition won't point out that they not only *could* have a problem that bad, but the actually *did* is unreasonable, and just reminds me that they did.
Re: (Score:2)
They can't GUARANTEE it but than can take a lot more precautions to reduce the probability than CrowdStrike did.
Re:"no vendor could "technically" guarantee" (Score:5, Insightful)
They can't GUARANTEE it but than can take a lot more precautions to reduce the probability than CrowdStrike did.
Gross incompetence is probably being too kind. Let's see:
About the only critical mistakes that they missed were "rollout on Friday at 5 P.M. before a holiday weekend" and "allow a single approver to commit to production", and I can't be certain that they didn't do the latter of those, because I don't work there.
This is a screw up on the level that in a just world should pretty much bury a company. If they still have customers, they should be figuring out how to appease them and talking very publicly as quickly as possible about the fundamental architectural and procedural changes they are making to ensure that similar things never happen again to stem the bleeding.
Whining that their competitors are saying mean things about them leads me to instead assume that they still don't fully comprehend how badly they screwed up and aren't making any fundamental architectural and procedural changes to prevent similar mistakes in the future, which gives me very little faith that this won't happen again. And in any world — even a relatively unjust one — a repeat of this disaster absolutely will be enough to bury the company.
Glad I don't own any of their stock.
Re: (Score:2)
Back in the mid 80s, I was working at JPL as a "seeing eye person" for a senior programmer who'd lost his vision to diabetic retinopathy. Mostly, we were working in FORTRAN, because that's what was required. Much of the time, we were working with nested IF statements, and I saw that every time he came to the last possible case, he still tested for it. In the unlikely event that the test failed, he saved the data, printed a distinctive error message -- 1
Re: (Score:2)
> Parsing configuration at ring 0. Check.
Thank you for pointing this out. It constantly gets overlooked.
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
Perhaps, by not ignoring decades of best practices, the competition has made themselves much less likely to cause a major outage. It's fair enough for the competition to point that out and they have some (albeit very limited) evidence behind the claim.
Meanwhile, CrowdStrike's president seems happy enough FUDding the competition by claiming, with no evidence whatsoever, that they are just as likely to have this problem.
Re: (Score:2)
As if SentinelOne has never caused an outage? They simply haven't done it YET on as a massive scale as CS because they don't have the market penetration.
None of these vendors are perfect with a 100% track record. They should all shut the f--- up.
I wanted to be the first to say "hahahahahaha..." (Score:2)
Playing sympathy songs on the tiny violins again.. (Score:5, Interesting)
It's not "shady" to attack a competitor when its product utterly and completely failed, causing the same type of downtime and disruption they bought it to try to prevent!
Obviously, yes - software errors can happen to any vendor. That's not the point. It's about how a company handles this eventuality. Testing before a mass push to production is one way, and rolling out in stages is another. You can even do what Microsoft often does where they let people opt in to receiving "early updates". If they really want to get every cutting edge patch and update ASAP, great. They're electing to be part of your test group....
Re: (Score:3, Interesting)
Gi
Re: (Score:2)
Re:Playing sympathy songs on the tiny violins agai (Score:4, Insightful)
Crowdstrike is absolutely a downtime avoidance solution -- that comes by default with being an antivirus/intrusion detection system. If Crowdstrike detected an virus and solved it by shutting down the entire company's computers, that would be effective, but obviously would cause a major problem.
It's actually easy to say what Crowdstrike could have done better -- they reported the details themselves:
https://www.crowdstrike.com/wp... [crowdstrike.com]
This includes array checks, content validation, and staged deployment.
I think a lot of criticism is deserved in this case due to the severity of the mess-up. The company brought down a decent percentage of the world's business computers and caused billions (perhaps trillions) of dollars in lost revenue, productivity, lost data, etc. It deserves (I'd even say *requires*) a lot of scrutiny, openness (on behalf of Crowdstrike) and some consternation from everyone.
Re:Playing sympathy songs on the tiny violins agai (Score:4, Insightful)
These two things are not mutually exclusive. Most IDS solutions are capable of performing just fine without making their host systems unbootable.
You must be an executive or work in PR.
So as long as their user experience is just slightly better than that of a ransomware attack, then everything is alright. That might even be true - if it was the only product on the market.
It seems to have affected almost every system it touched. I guess they could have tried it on actual hardware before rolling it out. They also could have rolled it out in waves so that it wouldn't have affected almost all of their customers at once.
Basic testing and upgrade waves are not hindsight - they've been considered best practices for many years.
I think the poor bastards that had to fix hundreds, or even thousands, of unbootable computers due to CrowdStrike's avoidable mistake would say that the criticism is more than "rhetoric".
Re: Playing sympathy songs on the tiny violins aga (Score:3)
Re: (Score:1)
Feelings over Facts (Score:2)
Imagine a crowd striking a company called "CrowdStrike" and said company getting upset.
Re: (Score:2)
Every time I read Crowdstrike I'm reminded of this movie quote, "I said the crowd is unarmed! There are a lot of women and children down there, all they want is food, for god's sake!"
Re: (Score:2)
Soylent Green is people.
It takes 1000 "attaboys" to get promoted. (Score:2)
Re: (Score:2)
Sorry, I don't trust that. I barely trust myself.
Whaaah (Score:2)
We f*cked up and are now losing customers left and right and are not being forgiven for the massive financial damage we did, this just isn't fair.
Come back, everyone, come back, otherwise my normally huge CEO bonus is going to suck this year!
Re: (Score:2)
> this just isn't fair
Well ... that might be true.
There's a proffered theory that somebody had to delete data in a way to avoid auditing (winpe or whatever it's called now) .
Crowdstrike is the company that was just somehow allowed to tell FBI to pound sand when they subpoenaed the DNC servers.
That only means one thing and imagine if the CEO does care but knew his company was being sacrificed.
What a terrible situation to be in.
(y'all can line up dates on the calendar to see the theory that fits the data -
Right (Score:2)
They can't technically guarantee it, but any security company that raises this big of a red flag about their testing processes deserves to be laughed out of the industry.
Re: (Score:2)
Any software security enterprise that cannot do proper input validation in their code is a joke, nothing else. That Crowdstrike cannot do proper testing either is just the icing on the cake.
CI & QA? (Score:3)
Do SentinelOne and Trellix do CI and QA?
That might be a competitive market advantage.
Re: CI & QA? (Score:1)
The problem is that CrowdStrike also claimed to do those things. They claimed to be using industry standards and quality control and to an extent that was true, the problem is that you can do the wrong type of quality control and CI/CD. Part of this problem was that shoddy QA was part of their CI/CD so they continuously integrated and deployed shoddy updates. The crux of the problem was a dev pushed a code update after the tests completed and because the code was already considered tested it thus went throu
The elephant in the room... (Score:2)
I guess we're all going to go on ignoring the fact that the only reason this was even possible in the first place is the EU's enforcement of Microsoft's Swiss-cheese security by the former forcing the latter to give random dirtbag companies like CrowdStrike unrestricted ring-0 access to the kernel, eh? I guess not, though, the current /. groupthink being that the EU is a saint and every US tech company is the devil*.
> no vendor could "technically" guarantee that their
> own software would never cause
Re: (Score:2)
Stop pushing that bullshit lie. The EU mandated a level playing field as anti-trust law requires. MS made a disingenuous offer. The competitors complained. Hence MS (!) decided to open up kernel access.
Re: (Score:3)
The EU mandated that Microsoft provide competitors with the same level of access that their own (competing) products enjoyed. Microsoft had at least three options on how to respond and stay in compliance:
1. eat their own dogfood -- make their own products use the same APIs that they were trying to get other companies to use
2. get out of the business of making these security products and let the ISVs figure it out
3. let the ISVs into ring 0
Microsoft could have gone with options 1 or 2, and then Crowdstrike
Re: (Score:2)
No, its on Crowdstrike for putting shit code in ring 0, not MS.
Re: (Score:2)
No. It's my machine. I should be able to put whatever I want in ring 0.
But if I do, that's MY responsibility.
I shit the bed (Score:1)
but I resent everybody in the dorm room whispering that I can't clench my anus.
There's no crying in capitalism! (Score:1)
Crowstrike (Score:3)
They did it to themselves (Score:2)
To be more exact, they messed up massively because they cannot even get simple software engineering and testing right. Anybody sane of their customers will go to a competitor, because they cannot really be worse, but will likely be better.
Boo Fuckin Hoo (Score:2)
What I want is instructions on how to remove their crapware.
They're not playing this game right. (Score:2)
FAKE NEWS! SHADY COMMENTARY! (Score:3)
Someone's updated their glossary!
I'm sorry, at some point you "in charge" types need to accept that when someone points at a real thing that actually happened, that anybody with a moment of time and an internet connection can check for themselves, it's not "fake" it's not "shady" it's just reality smacking you in the face, just like it does everybody else that sometimes makes mistakes. Most of us normal, non-in-charge folks have to own it when we make a mistake, because there's no one else to point the finger at. When you, or your company, make a mistake, you are *NOT* absolved of it just because it's a corporation. You are *NOT* entitled to tell others they aren't allowed to comment on your mistakes. You may try to pretend that the issue isn't yours to own, but others are perfectly within their rights to point out that you're only pretending. And frankly, your fantasy doesn't mesh with reality.
Crowdstrike fucked up bad. Competitors rightly said, "Try our shit instead. We haven't caused a global issue." Crowdstrike's president is coming off like a toddler that's angry they had to take their timeout. "But, Billy doesn't have to sit in the corner!"
"Billy didn't shit his pants then take them off and toss them on the picnic table."
"That's some really shady commentary!"
What is it about today's C-suite that makes them think they are just *ENTITLED* to whatever they want, whenever they want, and then they feel about whining publicly like a bunch of children when the rest of the world doesn't bow down to kiss their ass when they drop the ball completely? We already don't like you folks. How about you pretend to join the rest of humanity and act a bit humble when you drop the ball so thoroughly? This comes off as the corporate equivalent of throwing yourself on the ground and screaming, "I WON'T ADMIT IT! I WON'T I WON'T I WON'T!" Take responsibility. Take a moment to reflect on your failure and the failure of your company. Assess. Maybe reassess several times. Tell the public what you'll do to mitigate the potential in the future. Then maybe we can move on.
This public denial just reminds everybody that you're awful. And it makes all of us want you to fail again just so we can laugh at you and anybody that accepts your half-assed processed going forward.
Re: (Score:2)
What is it about today's C-suite that makes them think they are just *ENTITLED* to whatever they want
Aristocrats will be aristocrats. I am uncertain how that developed in the USA, but it is here and it is very bad.
Re: (Score:2)
What is it about today's C-suite that makes them think they are just *ENTITLED* to whatever they want
Aristocrats will be aristocrats. I am uncertain how that developed in the USA, but it is here and it is very bad.
They were always here. But the original aristocracy in America were very good about putting on a false front and being "men of the people." Or at least it appears that way looking back at the founding of our country, or even before. I just think our current crop are too stupid, or too arrogant, to bother keeping up the facade. They want people to *KNOW* they see themselves as better people, and everyone else as lesser.
Criticism is justified (Score:2)
Pretty much everyone I know was effected, even my mom and uncle were complaining about it. If I were a CIO I would be evaluating replacement options. This won't kill Crowdstrike, but adopting them as a vendor in the future will be extremely questionable going forward.
Re: (Score:2)
As it looks like right now I have realized that it doesn't matter if you have a security add-on like Crowdstrike or any other third party supplier because when there's a strike it goes faster than anyone can handle - and it circumvents most security solutions.
I have realized that the only way is to compartmentalize the network as much as it's feasible to do. Keep your printers on a separate subnet for example. Modern multifunction printers are smart enough to allow installation of programs. So are TV:s.
In a
Re: (Score:2)
I messed up (Score:2)
The Germans have a saying for it (Score:2)
Wer den Schaden hat, braucht für den Spott nicht zu sorgen.
Would someone please pass a cup of Streisand? (Score:1)
Take your lumps. (Score:2)
If you do dumb shit and knock half of corporate america into the jackpot, you deserve what you get.
Don't want to get dragged for your colossal fuckups? Don't colossally fuck up.
Fire the CTOs (Score:1)
Crowdstrike unhappy? (Score:2)