Microsoft Exploit Predictions Right 40% of Time 182
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
Re:Attention U.S.citizens (Score:3, Informative)
Actually it originated with One Alan Baxter of Rochester and expanded by other people on Usenet. So if you do give credit where it's due give it where it's actually due.
Re:This is why Microsoft software sucks (Score:5, Informative)
Hint: 40% is worse than guessing.
No - from TFA:
The index, launched last month, rates each vulnerability using a three-step system.
Random guesses would be expected to yield 33% success.
Re:This is why Microsoft software sucks (Score:3, Informative)
40% is worse than guessing only if you have only two choices (e.g. heads or tails). If you have more choices it is a bit better than guessing.
MS was predicting not just whether exploits would appear but the kinds of exploits that will appear. Depending on how specific (e.g. there will be a buffer overrun in module XYZ) or general (e.g. there will be an exploit in Windows *somewhere*) they were about the kinds of exploits, 40% could be either pretty good (i.e. they were insightful) or pretty bad (i.e. they chose the obvious things). In either case they would still be better off than pure random chance.
Re:Attention U.S.citizens (Score:3, Informative)
Ahh, here we go.
http://www.snopes.com/politics/satire/revocation.asp [snopes.com]
More exciting than reading about how badly microsoft can classify security bugs eh? :)
ps. NO FIREFOX, I WILL NOT CAPITALISE THE "M" IN mICROSOFT!
Re:Congratulations? (Score:2, Informative)
Wouldn't it make MORE sense to perhaps spend the human/technical resources FIXING the most exploitable bugs rather than standing around with a beer in hand saying 'yep, that's going to explode for sure'.
Yes it indeed would, and thats exactly what they have done and the story is about the review of the practice that happened at the end of the month (read during a review of what became an exploit and what got fixed at the right time)
Re:It is TERRIBLE (Score:3, Informative)
No. What happened was this - MS spotted 18 potential security holes. 9 of them were regarded as more serious. A company that focussed on protecting against those 9 would not have been affected at all and would have had less disruption than a company that protected against all 18.
They are offering this as a means to tell their bug fixing department and other companies which areas to prioritize.
Re:It is TERRIBLE (Score:4, Informative)
What REALLY happened is this: Every security hole that MS discovered on its own, was exploited BUT we are supposed to be happy because in 40% of the cases MS correctly predicted that it would be exploited.
I know we don't RTFA but please at least RTFS.
'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.'
So no, at least according to the summary not every security hole was exploited. If you're going to claim otherwise at least provide some links to an article; hopefully one supporting your claims although that's not always necessary for the +5 informative.
In fact I just actually bothered to RTFA, just to make sure, and it said that no exploit code appeared for the low ranked vulnerabilities.
Re:Congratulations? (Score:2, Informative)
Re:Congratulations? (Score:4, Informative)
I don't understand how the article got the math completely wrong or how people aren't seeing the extremely obvious flaw in the math.
Re:Congratulations? (Score:3, Informative)
Actually, they'd have to flip a coin for every bug – and their current statistic, "40% of the bugs we identified as exploitable were exploited", would probably look great compared to the percentage they'd get by flipping a coin.
Basically, you're looking at this wrong. Microsoft correctly predicted 40% of the exploitable bugs, but they also correctly predicted the non-exploitable ones which wouldn't be exploited.
Suppose (and I don't have actual numbers, so I'll make up hypothetical ones) Microsoft finds 100 bugs, and 5 of them appear exploitable. 2 of those are actually exploited (40%). However, you should take into account all the non-exploitable bugs that weren't exploited: Microsoft correctly predicted 95 non-exploitable bugs and 2 exploitable ones, which is 97%. They were incorrect only on the 3 bugs that they thought would be exploited and weren't (using these hypothetical numbers).