Sophos Anti-Virus Update Identifies Sophos Code As Malware 245
An anonymous reader writes "Yesterday afternoon anti-virus company Sophos Inc. released a normal anti-virus definition update that managed to detect parts of their own software as malicious code and disabled / deleted sections of their Endpoint security suite, including its ability to auto-update and thus repair itself. For many hours on the 19th, Sophos technical call centers were so busy customers were unable to even get through to wait on hold for assistance. Today thousands of enterprise customers remain crippled and unable to update their security software."
Sophos points out that not everyone will be affected: "Please note this issue only affects Windows computers."
99.999% (Score:5, Insightful)
how many of Sophos customers are not on the Windows platform? that makes me laugh.
Re:99.999% (Score:5, Insightful)
Re:99.999% (Score:5, Insightful)
Re:99.999% (Score:5, Funny)
What's impressive is that this got out of Sophos' testing lab and into production. I guess they must not test signatures in house at all. Congratulations, Sophos customers, you've been promoted to alpha testers.
Actually, it's an incredible show of honesty on the part of Sophos. Perhaps Symantec and McAfee will follow suit and flag their own software as malicious as well.
Re: (Score:2)
Re: (Score:3)
Certainly it makes it one of the easiest to remove antiviruses, which is a pretty major AV feature in my book.
Re:99.999% (Score:4, Funny)
What's impressive is that this got out of Sophos' testing lab and into production.
What's really impressive is that is that it also orchestrated a DDOS attack on the Sophos tech support helpline...
Re:99.999% (Score:5)
The trouble, in this case, is that it detects its own signature update componenets as viruses...
Not only should this have been caught in testing(Since it would have cropped up more or less the moment the new signatures were loaded onto a live system with Sophos installed; but they hit files about which sophos presumably has intimate knowledge, this isn't some 'obscure packing/compression scheme used by legacy CAD program that seemed like a good idea in the 80's looks like a suspicious obfuscated payload' kind of thing.
I am not impressed, though thankfully it only took me a little over half a day to fix it here...
Re: (Score:2)
I think it basically detects all files on your system that include "updater" in the path. It also kept doing it over and over again.
Re: (Score:2)
>>>All it takes is one mistake and you have a loose cannon and a front page news article like this one.
This is why my virus update is off. I update about once a month, and I only accept OLD updates not newer ones. So if I had Sophos on my computer I would be having zero problems right now.
Re: (Score:3)
Re: (Score:2)
Speaking of percentages, I wonder what percentage of anti-virus updates go terribly wrong like this. 0.00001%?
It's got to be more than that. I remember a few years back that several people in my company who were foolish enough to have anti-virus on their Windows PCs configured to auto-fix problems came in in the morning to find it had deleted some essential Windows DLL files.
That software probably only updated once a week, so you're talking more like 0.1%.
Re:99.999% (Score:4, Funny)
I'm just glad I didn't have a mouthful of coffee when I read:
or I would still be cleaning coffee off of monitors, laptop, papers, etc.
I have a couple of old Windows XP installations I can still get to when some idiot creates a web site that only works right in IE (e.g., I live in Colorado and the state has a site for doing your state income tax that doesn't work when accessed with Firefox). Ditto for software like most income tax programs. I don't otherwise use Windows. Even my work laptop is running Linux (Fedora 16).
Cheers,
Dave
Re: (Score:2, Troll)
Re: (Score:2)
They also have a mac client, if I recall. If you need A/V for the Windows boxes anyway, plus something on the mail server to snip some of the crap out on the way in, it becomes a fairly easy sell for the vendor to shove a few mac or linux licenses out the door if some of their customers have a paranoic 'zOMG all computers must have antivirus to protect our megahertz!!!" policy. If you have to implement that, it's easier to at least implement it all in one place, with one console, and maybe a volume discount
Re: (Score:2)
Re:99.999% (Score:4, Funny)
Re: (Score:3)
My work requires av to be installed. No mention that it's files can't be chmod 000ed though :)
Re: (Score:3)
I got hit by malware on Redhat years ago (the L10n worm) so it does happen.
Anyway, I have a corporate Win 7 desktop with Sophos now and got this bug. Every few minutes it popped up a warning that I had been infected with malware. Very annoying. By the end of today it had stopped, so either IT had fixed it or it had managed to commit suicide. The one time I did get infected with malware on this PC Sophos didn't catch it and I had to download Malwarebytes and fix the registry myself.
Re:99.999% (Score:5, Informative)
So far, there have only been a couple 'proof of concept' viri for Linux. Nobody's figured out a way to pry any money away from us yet. :D
but linux antivirus aren't used to protect linux, they are useful if you run a mail server or a proxy so you can clean mails and webpage before they infect a windows user, or to clean an infected windows installation, for example the kaspersky live cd is based on linux
Re: (Score:2)
They can be. The first ever virus was written for UNIX.
Re: (Score:3)
They can be. The first ever virus was written for UNIX.
Unless counting a self-replicating failure on an early Manchester machine, the first virus we know of was from 1971, and ran on TENEX on a modified PDP/10. No UNIX (or Unics).
The first virus outside arpanet or labs infected Apple systems, by the way.
Re: (Score:2)
Is this some type of lesbian virus??
Can We Say Test our Code, anyone??? (Score:5, Insightful)
This is a classic case of not thoroughly testing code and making sure you have enough variations of test machines to ensure as little pain to clients as possible.
If I were a customer, I would be shopping for a better company.
Re:Can We Say Test our Code, anyone??? (Score:4, Insightful)
Is there a better company, though? Seems like all the major antivirus vendors have had embarassing false positives like this in the past.
Re: (Score:2)
Yes, but getting a false positive on your own software takes it to an entirely new level.
Re: (Score:2)
Hello QA department your fired.
Re: (Score:2)
Hello QA department your fired.
nah, more like: Hello $computerguy, you're hired. we need a QA dept.
Re:Can We Say Test our Code, anyone??? (Score:4, Informative)
This is a classic case of not thoroughly testing code and making sure you have enough variations of test machines to ensure as little pain to clients as possible.
Antivirus engines and definitions change daily, weekly at the most. Where do you suppose this "thorough testing" of code is supposed to happen? It costs time and money, and while you're busy doing that testing, the support lines are being flooded with "We've been infected by something your software doesn't protect against! What are we paying you for, anyway?" As a bonus, your competitors, who didn't decide to setup a massive lab with dozens of employees in it, testing all the typical configurations of a half dozen operating systems and the couple hundred most popular software packages of each... they already released a patch.
Now, a software patch that causes the application to stomp on its own dick is amusing (and difficult to forgive), but demanding a massive expenditure of time and money is almost as unforgiveable. It's easy to demand best practices and ample safety margins: It's quite another thing to deliver it in a business environment. Most people in the industry, including the people at Sophos I'm sure, do the best they can with what they're given. It's pretty much the work creed of anyone in this industry -- few have the time and resources to do it right, they have to settle for 'good enough'.
And sometimes, good enough breaks.
Re: (Score:3)
Re:Can We Say Test our Code, anyone??? (Score:5, Insightful)
That's pocket change compared to how much the company can lose over a screw up like this.
Emphasis mine. Look, every major antivirus producer has made a similar mistake to this. Sometimes, it takes the whole operating system down with it (Symantec anyone?). Whether you agree or disagree, it's clear there are business incentives for a fast workflow process -- and as the old saying goes "Do it fast, do it right, do it cheap -- pick any two." It's obvious which ones the antivirus industry as a whole has chosen. Rather than argue over whether or not they're right, I'm pointing out why they're making those choices. Businesses aren't willing to pay a premium to avoid mistakes like this. The cost of the occasional screwup like this is less than the cost required to do all the testing and lab work that many here on slashdot seem to support.
It's a business decision they've made, right or wrong.
Re: (Score:2)
Not sure. This issue hit my workplace (state university), and it only affected 2 computers in my office, and I never heard about it from outside the office. I think there were other factors that triggered this.
Re: (Score:2)
Re: (Score:2)
You're defending mediocrity?
Re: (Score:2)
How interesting... (Score:2)
... the chicken ate the egg, after all...
QA? (Score:2)
So, how much testing do they perform on their own product. I suppose they do not even know how their own "dogfood" tastes.
Re: (Score:3)
they're running Avast free version like everyone else.
Re: (Score:2)
So, how much testing do they perform on their own product. I suppose they do not even know how their own "dogfood" tastes.
Are you kidding, the bitch killed and ate her own pups! How do you test for the software equivalent of zombie Apocalypse?
Which just goes to show... (Score:2)
"test by eyeballing the code" has its drawbacks.
In a perfect world, the QA manager would be updating his resume.
Re:Which just goes to show... (Score:5, Funny)
"test by eyeballing the code" has its drawbacks.
Exactly. Sometimes code that looks useless is really pretty important. The article follow up said they removed this test from an iteration loop, since there weren't comments about what it did. Apparently the original programmers thought it obvious...
if ( asimov_3rd_violation())
{
continue;
}
else
{
remove_file(filename);
}
Re: (Score:2)
Oh, that's brilliant. The thing is, any geek would get the significance immediately. What kind of dunderhead would delete it?
Re: (Score:3)
This should be obvious to any geek! What is Asimov's 3rd law? All together now: "A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws."
I've never seen the code in question, and it's obvious to me that this means "don't delete myself".
Re:Which just goes to show... (Score:4, Insightful)
Re: (Score:2)
I think this was an in-development definition that wasn't meant to be deployed at all. It referenced a virus that didn't exist "shh/updater-b" and Sophos didn't even have a page for that name on their web site when it hit. It flagged anything on the system with "updater" in the path.
Only Windows? (Score:2)
Blasted it off as quickly as I could. No harm done that I can find.
FINALLY (Score:2)
An honest scan report from a major anti-virus vendor. Was it flagged as spyware/advertising trojan?
Tautologies are fun (Score:5, Funny)
Obviously, once this change had gone in, Sophos was correct to identify itself as malicious.
software leukemia! (Score:3)
Let's see this isn't a virus, it's kinda like software leukemia or a software autoimmune disease.
Re: (Score:3, Funny)
Re: (Score:2)
+1 House reference
Re: (Score:2)
-1 Explaining a joke
In other news... (Score:4, Funny)
The detection rate for Sophos's malware engine inched closer to 100%.
Own Goal (Score:2)
A definite Own Goal. This gaffe is one that will be repeated for years to come, if not decades.
McAfee (Score:2)
As memory serves McAfee did this about 8-10 years ago with an update. It's a sign of poor release management and a failure to follow best practices. If they fail to follow best practices for something like this that is high visibility and customer facing, imagine what they look inside the company.
Time to start bringing your business elsewhere.
There needs to be an award for this (Score:5, Interesting)
Every year, we need to go down the list of software makers who have managed to totally Bork their users. The Meltdown awards. Just to distinguish between the companies that handle it well and the companies that are incompetent.
Re: (Score:2)
Operationsystemic lupus sophosus (Score:2)
Nobody expects the Spanish Inquisition (Score:2)
Perfect attack vector for a real infection - as part of the AV suite. Talk about stealthy.
Malware makers take note! (Score:5, Interesting)
Wanna cause problems? Add code from the various AV vendors...
Re: (Score:2)
Stooge? How so? He should be commended for pointing out yet another possible threat.
They're not the first AV vendor to do this. (Score:2)
Re: (Score:2)
So it failed twice... (Score:2)
First for calling itself out. And then again for NOTcalling Windows out.
So it goes...
Windows AV programs are malware (Score:4, Interesting)
Just think about it. The average Windows AV program runs with sufficient privilege to wreck your system by altering or removing arbitrary files. And it gets fed multiple updates per day created by teams of workers working in a hugely stressful situation: When a new virus appears, you've got to get those signatures out NOW.
I'm amazed people don't see this risks in this.
Re: (Score:3)
Neither. I don't run Windows AV software and I don't run Windows.
Ah. You take the other risks that I missed. Gotcha.
We were affected (Score:2)
We are currently considering switching AV vendors from Kaspersky (our license renewal is coming soon). So the boss contacted Sophos and they sent a guy yesterday to install a demo and got hit with this bug.
Needless to say the guy was pretty embarrassed.
I like ESET nod32 myself, but it seems that the administrative console is not as good as Kaspersky (K's allows to deploy software, turn off machines, send messages to users and lots of other non-AV stuff we actually need)
Could be worse. (Score:2)
BitDefender once did the awesome feat of quarantining every. single. file. They even rolled out the update to all x64 Vista and 7 machines (possibly XP, too).
Thanks goodness for backups.
Notes from an effected enterprise (Score:5, Informative)
Yes, this was bad. The virus signature in question appears to match any software that does auto-updates (possibly trying to spot phone-home malware?) so it's flagged dozens of software packages and according to what policy you've set, quarantined or deleted the files. This includes the auto-update part of the sophos client. The flood of emails from the sophos enterprise manager package as machines were switched on this morning quickly alerted us that this wasn't good, and just looking at names of the files it was flagging was enough to see that this was a false positive. Cleanup continues.
We've been very happy with sophos enterprise, and I'm staggered that this signature made it out the door - they should have numerous controls in place to ensure this can never happen and I await an explanation for how they failed.
I'm not too impressed by some of the advice given in their cleanup procedure [sophos.com] - they advise setting the policy to not scan certain sophos directories - guess where viruses may try to hide in future.
This is an embarassing fubar which will have had a high impact on thousands of enterprises. It'll be interesting to see if Sophos come clean about the circumstances and can be convincing enough about how it's never going to happen again.
Re: (Score:2)
malware from whom's perspective. Adobe absolutely things keygens are malware.
Re: (Score:2)
It makes me LOL that people still have keygens for Windows XP.
Re: (Score:3)
Re: (Score:2)
I'm at work actually, and use XP, you insensitive crow !
Re: (Score:2)
Considering all the people I know that still want to stay with XP no matter what, it doesn't surprise me at all.
I was like that until I realized that Windows 7 is a very good OS. And, as a gamer, I also prefer DirectX 10 over 9.
Re: (Score:2)
Indeed. Most people also prefer a pie in the face over a punch in the jaw.
Re: (Score:2)
What will those people do when Microsoft ends support in less than 2 years. [microsoft.com]
Re: (Score:2)
What will those people [Windows XP lovers] do when Microsoft ends support in less than 2 years.
Be smugly satisfied that they eeked every ounce of use from their software while simultaneously feeling dirty for having to buy Windows 9.
Re: (Score:2)
Be happy that they dont have endure Patch Tuesday any longer.
Re: (Score:2)
Well, there are guys like me: I have a tower running kubuntu, a notebook running W7, and an old Dell someone gave me that I repaired, including XP install disks. I want to use that box to sample LPs and cassettes and burn them to CD. EAC won't run on Linux or on any machine without an optical drive, and Audacity simply lacks the features I need. My only choices are XP on the old junker or buy a brand new computer, or build one from new parts and buy W7.
Nope, XP has to stay until they port EAC to Linux or th
Re: (Score:2)
Re: (Score:2)
Why in gods name do you attribute this only to Microsoft? It's standard practice because the source of these aren't trustworthy and they're moderately easy to detect. I doubt Microsoft gives two shits if you download a keygen for a video game, yet they will pretty much all be detected by such AV software, generally even free software not theoretically bound by corporate purse strings.
Re: (Score:2)
"For keygens, I run them in an isolated VM instance and roll back the disk files after I'm done using them. You can never be too sure."
Or you could, I dunno, not use keygens?
(I'm sure I'll hear a rejoinder about old software that you've lost the key for, but we all know what people are really using them for).
Re:That's why I don't install AV software on my PC (Score:5, Funny)
That's like saying you don't use condoms because you know how to pull out.
Re: (Score:3, Insightful)
No, it's like saying you don't use condoms because you only go to bed with people you know well enough to trust them when they say they're on the pill.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
No, it's more like saying he know how to evaluate (and trust) his sexual partners before engaging in sex, and those that he doesn't trust or can't be sure of, he brings to the clinic to get tested first...
Re:That's why I don't install AV software on my PC (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
If you've got malware calling home, you've already lost, you've already been pwned. You should also know that nowadays many things call home- Chrome, Firefox, etc.
Re: (Score:2)
I don't put AV software on production servers either unless PHBs etc require it. In my experience if you do things right, AV software is more likely to cause you problems than a virus.
And you are the reason why my company gets discounted rates on payment card processing. We actually *pass* the PCI audit every year.
Re: (Score:2)
No infections that you KNOW of.
Re: (Score:3)
You might assume the AV vendor is really good at spotting malware, but their job is like solving the halting problem, only without knowledge of the full inputs and program.
I on the other hand prefer to "solve" the halting problem by ensuring the program actually halts no matter what happens- aka Sandboxing.
Re: (Score:2)
Point is, with or without AV protection, you can never know fer shure.
Re: (Score:2)
You may be surprised what might be crawling around your machine right now.
Re: (Score:2)
They are unlikely to bother with my sort of config since they can already make money from the masses of people who need AV software, or from Gover
Re: (Score:2)
There are a lot of rootkits that embed in atapi.sys. They must have just based their definition on hashes of known-good versions of atapi.sys and missed several revisions of the file.
Re: (Score:2)
Seems that on a Windows Active Directory network, isn't this something an admin can script to run on all the computers at once? Or am I vastly overestimating their management capabilities.
Re: (Score:2)
Yep. I got hit by it, and it took down Google Updater, Java Updater, and its own auto-update. Worst part is that it kept trying to relaunch the Sophos updater over and over again, prompting even more pop-up notices.
Weird thing here is that only 2 people I know were affected including me. Nobody else in this office was affected. My wife works on another campus (state university), and nobody in her office was hit.
Re: (Score:3)
Measure twice, cut once!
That's the old, craftsmanship way. These days, especially with software, it's measure with a micrometer, mark with chalk, cut with an axe.