Microsoft Exploit Predictions Right 40% of Time 182
CWmike writes "Microsoft today called its first month of predicting whether hackers will create exploit code for its bugs a success — even though the company got its forecast right just 40% of the time for October. 'I think we did really well,' said Mike Reavey, group manager at the Microsoft Security Research Center (MSRC), when asked for a postmortem evaluation of the first cycle of the team's Exploitability Index. 'Four of the [nine] issues that we said where consistent exploit code was likely did have exploit code appear over the first two weeks. And another key was that in no case did we rate something too low.' Microsoft's Exploitability Index was introduced last month."
Congratulations? (Score:3, Insightful)
That's great, guys, but don't you think being proud that you were right about your code being exploited is... backwards? That's like being proud you correctly predicted you would get stabbed while walking through a ghetto wearing gang colors.
Then again, this is Microsoft. They probably throw an office party every time something compiles without errors.
=Smidge=
That's not too bad (Score:5, Insightful)
This is why Microsoft software sucks (Score:2, Insightful)
Any engineer who says that "40% is pretty good predicting" is incapable of writing good software, or managing a project, or, even, applying the scientific method.
Hint: 40% is worse than guessing.
Re:This is why Microsoft software sucks (Score:5, Insightful)
>if it comes up heads, its exploitable. Tails its gonna be ok.
In this case, wouldn't there be as many false negatives as false positives?
Re:This is why Microsoft software sucks (Score:5, Insightful)
No, it means that they were able to cut the field of their immediate focus nearly in half while not missing any issues. For such a complex system without any precise mathematical model, that's pretty good.
In this case, flipping a coin is statistically likely to let an unaddressed issue through, and that's a big no-no for applications like this.
Re:Still not getting it. (Score:3, Insightful)
4/9 = 40%? (Score:1, Insightful)
Research also shows Slashdot editors verify submission figures 112% of the time.
Re:Still not getting it. (Score:3, Insightful)
or hell ANY GOD DAMN FUCKING BUG YOU FIND, needs fixing, right Microsoft?
Any goddamn bug doesn't need fixing asap the same way. Software always has bugs, even really good software, so it's a matter of prioritizing which bugs are show-stoppers, which are less problematic and which are minor.
The problem with Microsoft is their habit of releasing bananaware: they ship green software that matures at the customers, at the expense of the customer of course who essentially pays to become a beta-tester for Microsoft. In other terms, when other reputable software shops iron out most bugs in-house before releasing their products, Microsoft just removes show-stoppers and let its customers report all the other bugs.
Re:Congratulations? (Score:5, Insightful)
Slashdot crowd *loves* MSFT bashing doesnt it.
Ok lets see... Some company (say Canonical or MSFT) builds a huge software and releases it. And a third party finds a bug and reports it to them. Now would be good to predict the severity of the bug, so that the more exploitable ones can be fixed first? Thats exactly what they are doing, and they are able to get the severity 40% of the time right, with no false negatives (that not a single severe one has been classified as a low priority one).
So, now, do you think this is bad or wrong or something?
Re:This is why Microsoft software sucks (Score:4, Insightful)
If the steps are sequential, it's less than 33%. The correct figure is 12.5% (50 percent of 50 percent of 50 percent).
Re:Being right 40% of the time... (Score:3, Insightful)
Doesn't look so impressive when you look at it this way.
Depends on the payoff.
It's not good if you're betting even money on coin tosses. But if you're a venture capitalist, it's great. The general rule for tech VCs is that 7 bets out of 10 will fail, 2 will do ok, and 1 will be a big success. If that 1 success is buying 10% of Google in the very early days, your 70% failure rate is still pretty awesome, because you're still up billions of dollars.
Re:Congratulations? (Score:5, Insightful)
So, now, do you think that that is not a reason for criticism on their internal software testing?
Re:Congratulations? (Score:2, Insightful)
No, it's like correctly predicting that you'll get stabbed 17 minutes after entering the ghetto, by 6 gang members dressed in red.
Not at all. It's much more like guessing that you will be stabbed 6.8 minutes after entering a ghetto by 8-9 gang members dressed in red, then actually being stabbed after 17 minutes by 6 gang members wearing pink.
Re:Congratulations? (Score:4, Insightful)
Sure, if you have unlimited resources and can devote an infinite number of people to fixing everything, that would be great. However, if you have finite resources available and have to devote them to fixing up certain areas, how do you know where to devote your attention? If you can come up with a methodology for predicting such a thing, put it to the test, and get decent accuracy in your predictions, then wouldn't that be useful for confirming for you how you should devote your limited resources?
There is nothing unique in what they are doing. I mean, look at the auto industry, for example. They don't just randomly assign engineers to try and make random things safer. They do studies, try to figure out what are the most dangerous aspects of a vehicle, and then assign engineers to work on those specific things.
Fortunately for the auto industry, it's a little easier to do your predictions pre-release, since the "attack vectors" are more limited and well known (there are typically only so many ways you can get into an accident, so it's easier to model a majority of those cases). This allows them to be proactive in fixing flaws. Unfortunately, the attacks vectors in software are a bit more numerous, and you often have to take a more reactive approach. What Microsoft is doing here is trying to model things to see how reasonable it would be to devote resources in certain ways to be proactive.
So again, in what way is this bad?
What do ya mean 40 percent? (Score:0, Insightful)
Has there ever been a Microsoft bug that hackers have left alone?
We've been through this 'a million times' since DOS; there are literally more than a million active viruses out there, with another 100,000 per month. 40 percent chance of an exploit being used seems kinda low, doesn't it?
More fail from MS (Score:2, Insightful)
How can a PR team for one of the largest corporations in the US seriously release a statement like this? What kind of company fails so badly that they can only predict 40% of exploits in their own [proprietary] software?
If a major car (or car part) manufacturer "accurately" predicted that 40% of their automobiles would explode and burn their owners alive due to a fuel system defect....would people still buy their cars? Oh right...firestone.
Re:Congratulations? (Score:3, Insightful)
No, the criticism of either their coding practices or QA has nothing to do with a new and fairly efficient way to prioritize bug fixes. They already have the software with all the holes built in. Now they should deal with what they have in the best way possible, don't you agree?
Re:Congratulations? (Score:3, Insightful)
Re:Congratulations? (Score:3, Insightful)
Um, no, I know we're all desperate for this to be some terrible mistake on MSFT's part, it just isn't.
This is more like the car company saying: We have found 10 ways that we think our cars can be sabotaged, and we have released free snap-on repair kits that are intended to counter those possibilities, and will distribute them to all customers who request them. As it turns out, only 4 of them have actually been used by saboteurs, but we nonetheless recommend installing all 10 kits just to be safe.
Yes, how irresponsible of them, finding and eliminating ways for dedicated deliberate attackers to gain access faster than those attackers can actually accomplish it.