Slashdot Log In
Microsoft Exploit Predictions Right 40% of Time
Posted by
timothy
on Fri Nov 14, 2008 06:54 AM
from the statistics-94pct-nonsense dept.
from the statistics-94pct-nonsense dept.
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Congratulations? (Score:3, Insightful)
That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.
Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
=Smidge=
Re: (Score:3, Interesting)
Indeed. I swear, I called it: it's easier to predict the holes when you release them yourself [today.com].
After what was expected to be an unusually quiet Patch Tuesday, Microsoft has released eight patches for applications with an insufficient number of security holes. "Our market is the enterprise," said Microsoft security marketer Jonathan Ness. "Information technology professionals know that Windows is the greatest IT job creation scheme in history. Without Patch Tuesday, there's no reason for the experienced IT
Re: (Score:2, Interesting)
That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards?
Well, they're not proud of making exploitable code (if they were, there would have been a giant endless party at Microsoft for the last 20 years), they're proud of predicting when/how fast their code will be exploited.
That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.
No, it's like correctly predicting that you'll get st
Re: (Score:2, Insightful)
No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.
Not at all. It's much more like guessing that you will be stabbed 6.8 minutes after entering a ghetto by 8-9 gang members dressed in red, then actually being stabbed after 17 minutes by 6 gang members wearing pink.
Re:Congratulations? (Score:5, Funny)
It's like running your own car into a pole, providing the mechanic with your estimate of the damages and claiming you were right when he only overcharges you by 60%.
Parent
Re: (Score:3, Insightful)
Re: (Score:3, Funny)
Re: (Score:2)
Re: (Score:2)
Re:Congratulations? (Score:5, Insightful)
Slashdot crowd *loves* MSFT bashing doesnt it.
Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).
So, now, do you think this is bad or wrong or something?
Parent
Re:Congratulations? (Score:5, Insightful)
So, now, do you think that that is not a reason for criticism on their internal software testing?
Parent
Re: (Score:3, Insightful)
No, the criticism of either their coding practices or QA has nothing to do with a new and fairly efficient way to prioritize bug fixes. They already have the software with all the holes built in. Now they should deal with what they have in the best way possible, don't you agree?
Re: (Score:3, Insightful)
Um, no, I know we're all desperate for this to be some terrible mistake on MSFT's part, it just isn't.
This is more like the car company saying: We have found 10 ways that we think our cars can be sabotaged, and we have released free snap-on repair kits that are intended to counter those possibilities, and will distribute them to all customers who request them. As it turns out, only 4 of them have actually been used by saboteurs, but we nonetheless recommend installing all 10 kits just to be safe.
Yes, how ir
Re:Congratulations? (Score:4, Funny)
Based on their success rate, they should flip a coin instead, then they'll be at 50%. That's what everyone's laughing at.
Parent
Re:Congratulations? (Score:4, Informative)
I don't understand how the article got the math completely wrong or how people aren't seeing the extremely obvious flaw in the math.
Parent
Re: (Score:3, Informative)
Actually, they'd have to flip a coin for every bug – and their current statistic, "40% of the bugs we identified as exploitable were exploited", would probably look great compared to the percentage they'd get by flipping a coin.
Basically, you're looking at this wrong. Microsoft correctly predicted 40% of the exploitable bugs, but they also correctly predicted the non-exploitable ones which wouldn't be exploited.
Suppose (and I don't have actual numbers, so I'll make up hypothetical ones) Microsoft find
Re:Congratulations? (Score:4, Insightful)
Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?
There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.
Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.
So again, in what way is this bad?
Parent
That's not too bad (Score:5, Insightful)
Re: (Score:3, Informative)
No. What happened was this - MS spotted 18 potential security holes. 9 of them were regarded as more serious. A company that focussed on protecting against those 9 would not have been affected at all and would have had less disruption than a company that protected against all 18.
They are offering
Re:It is TERRIBLE (Score:4, Informative)
What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.
I know we don't RTFA but please at least RTFS.
'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'
So no, at least according to the summary not every security hole was exploited. If you're going to claim otherwise at least provide some links to an article; hopefully one supporting your claims although that's not always necessary for the +5 informative.
In fact I just actually bothered to RTFA, just to make sure, and it said that no exploit code appeared for the low ranked vulnerabilities.
Parent
Still not getting it. (Score:2)
Well, that's like saying, after you block all your email from getting through, "We rated all the spam accordingly, and let none of them through".
How about, we just guess, a rough fucking guess, that any "remote code execution" or "run with elevated privileges" exploit or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?
Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.
The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsof
Re: (Score:3, Interesting)
You mean, like Apple's Leopard release? Or Apple's iPhone 3G release? Or Apple's mobileme release?
I fail to see how Microsoft has a reputation of releasing 'bananaware' whereas Apple doesn't. I don't recall hearing about major, crippling bugs when Office 2007 came out (one of their biggest apps), and rega
This is why Microsoft software sucks (Score:2, Insightful)
Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.
Hint: 40% is worse than guessing.
Re: (Score:2)
Dear MS. I have a foolproof way of enhancing and improving upon your algorithms to determine the exploitability index.
if it comes up heads, its exploitable. Tails its gonna be ok.
I estimate you will increase your predictive capabilities by a whole 10% using this method.
Re:This is why Microsoft software sucks (Score:5, Insightful)
>if it comes up heads, its exploitable. Tails its gonna be ok.
In this case, wouldn't there be as many false negatives as false positives?
Parent
Re:This is why Microsoft software sucks (Score:5, Insightful)
No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.
In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.
Parent
Re:This is why Microsoft software sucks (Score:5, Informative)
Hint: 40% is worse than guessing.
No - from TFA:
The index, launched last month, rates each vulnerability using a three-step system.
Random guesses would be expected to yield 33% success.
Parent
Re:This is why Microsoft software sucks (Score:4, Insightful)
If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).
Parent
Re: (Score:3, Informative)
40% is worse than guessing only if you have only two choices (e.g. heads or tails). If you have more choices it is a bit better than guessing.
MS was predicting not just whether exploits would appear but the kinds of exploits that will appear. Depending on how specific (e.g. there will be a buffer overrun in module XYZ) or general (e.g. there will be an exploit in Windows *somewhere*) they were about the kinds of exploits, 40% could be either pretty good (i.e. they were insightful) or pretty bad (i.e. t
It depends on which exploits they call correctly (Score:2)
For instance, if they patched the priv. escalation to SYSTEM that has a broad surface area (think, say, remote IIS exploit) over 9 exploits that require physical access and can only get guest access. If someone else has physical access to your box, it's n
Re:This is why Microsoft software sucks (Score:5, Interesting)
40% accuracy in predicting with no false negatives? There are plenty of distaster agencies around the world who would be incredibly pleased with that kind of accuracy
Parent
Re: (Score:2)
Actually 40% is quite good considering, as others have mentioned, that 33% would be the random chance.
it is also worth noting that they have 40% prediction of KNOWN threats.
I would bet there are about as many undiscovered exploits re: these updates, which could drive up or down the percentage.
If I can predict the stock market by +7% over random guessing, that is pretty damn good predicting.
Shows the confidence they have in themselves (Score:2)
Interestingly what they are saying here is that they think that
a) Hackers are smarter than they actually are
b) Microsoft code is easier to exploit than it actually is
So the perception is that Microsoft is better than their prediction, but the implication of that is that Microsoft think they are rubbish.
Maybe all these years of "Microsoft sucks" posts on Slashdot have actually come from the MS security team.
Re: (Score:2)
No. What they say is:
You should fix this bug first, since we believe it is the most likely to be exploited.
You can save these for later, since we don't believe it will be immediately exploited.
There is, however, something to be said for hackers referring to this list to find "unlikely" bugs to exploit.
Re: (Score:2)
So Microsoft thought their code was exploitable and said so, and it was, and instead of doing something about it they just congratulated themselves on predicting it!
Now here's an odd idea rather than predicting if something is exploitable and then publishing it, why not just not write code that is easily exploitable....!
and note the 40% is only the exploits they know about ....so even that is suspect....
Exploitability Threat Level Announcement. (Score:4, Funny)
Microsoft Exploitability Threat Level Indicator is a series of color codes starting from Dazzling Arctic White to Heart of Dick Cheney. Though exact number of these colors is considered a secret, from the past announcements we deduce there are at least 22 million of them.
For PRNewswire, copy edited by Anurag Chakraborty in Bangalore and supervised by Robert Zimmermann in Pittsburgh.
but but but... (Score:2)
there is so many to chose from...
Being right 40% of the time... (Score:2)
...is the same as being wrong 60% of the time.
Doesn't look so impressive when you look at it this way.
Re: (Score:2)
Re: (Score:2)
Without knowing the baseline they're working on, this could range from extremely impressive to completely useless.
Ok. So 4 out of the 9 bugs they expected to see exploits codes for actually had exploits meterialise. How many bugs had exploits coded that were not in thier 9 candidates? What is the total number of bugs taken into consideration?
If you were playing "battleship" on a 3x3 board with 4 "ships", taking 9 guesses to hit all 4 would be pretty dismal. Change that into a 30x30 board and suddenly 9 gues
Re: (Score:3, Insightful)
Doesn't look so impressive when you look at it this way.
Depends on the payoff.
It's not good if you're betting even money on coin tosses. But if you're a venture capitalist, it's great. The general rule for tech VCs is that 7 bets out of 10 will fail, 2 will do ok, and 1 will be a big success. If that 1 success is buying 10% of Google in the very early days, your 70% failure rate is still pretty awesome, because you're still up billions of dollars.
This is what it's come to? (Score:2)
I realize that Windows is a complex hunk of crap...errr...operating system, but wouldn't they be better served trying to find and correct these issues rather then just releasing them into the wild and keeping their fingers crossed?
Their attitude is sort of like pointing the gun at your foot and firing five times, and bragging that you only hit two of your toes.
This is why, every day when I arri
Thanks, Microsoft! (Score:3, Interesting)
No one seems to be looking at this from the opposite angle.
If I'm writing malware that's going to need to exploit Windows, this gives me an easy chart of which exploit I should pick -- the ones with the lowest patch priority, of course.
Re: (Score:3, Funny)
With the exception of points 7. and 9. it all seems quite reasonable.
Maybe one day you'll learn to drive on the right side.
And vinegar is acceptable on salad only, not potatoes.
By the way, I live in Continental Europe and my ancestors, at the time you were wearing animal furs and piling rocks in bizarre patterns, were building aqueducts.
So, in the end, Her Majesty, please
1. learn to drive
2. learn to cook
3. understand that fox hunting isn't a sport
4. stop using that absurd currency that is the pound sterling
Re: (Score:3, Funny)
Alas, "yeild" doesn't seem to be in the American psyche anymore.
It's also not in any dictionary that I'm aware of either. Yield is though. Sorry, couldn't resist :)
Re: (Score:3, Informative)
Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.
Re: (Score:3, Informative)
Ahh, here we go.
http://www.snopes.com/politics/satire/revocation.asp [snopes.com]
More exciting than reading about how badly microsoft can classify security bugs eh? :)
ps. NO FIREFOX, I WILL NOT CAPITALISE THE "M" IN mICROSOFT!
Re: (Score:2)
No... this was a Triumph! [wikimedia.org]
* sorry, as much as I love portal... it's getting old!