Internet Explorer 9 Caught Cheating In SunSpider 360
dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."
Re:Benchmarks (Score:4, Informative)
And there is no actual evidence that they are actually cheating. The article writer puts forth 2 other highly likely situations but then dismisses them for no good reason for the "cheating".
No proof? (Score:5, Informative)
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.
Re:Cheating allegation too strong (Score:5, Informative)
Re:Embarassing? (Score:5, Informative)
Another misleading tabloid headline from Taco et al.
Short story: Someone notices a perhaps too-fast result for a particular benchmark test with IE 9 and modifies the benchmark code which then throws IE 9 performance the other way. One *possible* conclusion is that MS have done some sort of hardcoding/optimisation for this test, which has been thrown out by the modifications.
Re:Benchmarks (Score:5, Informative)
There is a difference between optimising for a benchmark and cheating at a benchmark. Optimising for a benchmark means looking at the patterns that are in a benchmark and ensuring that these generate good code. This is generally beneficial, because a well-constructed benchmark is representative of the kind of code that people will run, so optimising for the benchmark means that common cases in real code will be optimised too. I do this, and I assume that most other compiler writers do the same. Cheating at a benchmark means spotting code in a benchmark and returning a special case.
For example, if someone is running a recursive Fibonacci implementation as a benchmark, a valid optimisation would be noting that the function has no side effects and automatically memoising it. This would turn it into a linear time, rather than polynomial time, function, at the cost of increased memory usage. A cheating optimisation would be to recognise that it's the Fibonacci sequence benchmark and replaces it with one that's precalculated the return values. The cheat would be a lot faster, but it would be a special case for that specific benchmark and would have no impact on any other code - it's cheating because you're not really using the compiler at all, you're hand-cmpiling that specific case, which is an approach that doesn't scale.
The Mozilla engineer is claiming that this is an example of cheating because trivial changes to the code (adding an explicit return; at the end, and adding a line saying true;) both make the benchmark much slower. I'm inclined to agree. The true; line is a bit difficult - an optimiser should be stripping that out, but it's possible that it's generating an on-stack reference to the true singleton, which might mess up some data alignment. The explicit return is more obvious - that ought to be generating exactly the same AST as the version with an implicit return.
That said, fair benchmarks are incredibly hard to write for modern computers. I've got a couple of benchmarks that show my Smalltalk compiler is significantly faster than GCC-compiled C. If you look at the instruction streams generated by the two, this shouldn't be the case, but due to some interaction with the cache the more complex code runs faster than the simpler code. Modify either the Smalltalk or C versions very slightly and this advantage vanishes and the results return to something more plausible. There are lots of optimisations that you can do with JavaScript that have a massive performance impact, but need some quite complex heuristics to decide where to apply them. A fairly simple change to a program can quite easily make it fall through the optimiser's pattern matching engine and run in the slow path.
Re:Embarassing? (Score:3, Informative)
And their 10-Q definitely indicates that they're not losing money. http://biz.yahoo.com/e/101028/msft10-q.html [yahoo.com]
The stock price is a meaningless indicator unless you are using it indirectly with their P/E ratio.
Re:Embarassing? (Score:4, Informative)
Socially liberally means allowing people to do whatever they want. In their bedrooms, in their homes, in their personal lives, so long as their actions don't physically harm another.
It does Not mean politically-or-fiscally liberal (using gov't to steal money from workers and redistribute it).
Re:Embarassing? (Score:3, Informative)
uh, hate to break it to ya, but ballmer selling 87 million in shares is not necessarily a good sign for the company given current losses. [nwsource.com]
Re:Benchmarks (Score:5, Informative)
Fear not, for I have RTFA and the original article that the digitizor article is based on.
Fortunately for the ethics of Mozilla, the named Mozilla engineer (Rob Sayre) never claimed that IE9 cheated. Instead, he diplomatically refers to it as a "oddity" and "fragile analysis" and filed a bug w/ MSFT.
http://blog.mozilla.com/rob-sayre/2010/09/09/js-benchmarks-closing-in/M [mozilla.com]
http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/ [mozilla.com]
So, blame Digitizor and ycombinator for putting words in Rob Sayre's mouth.
Re:Benchmarks (Score:5, Informative)
1) If you actually read the article, you may have noticed that the engineer is named. It's
right there there at the beginning of paragraph 2: "While Mozilla engineer Rob Sayre"
2) The "cheating" stuff is all from the Hacker News thread and the fucking articl. I
suggest you further read item 1 under "Further Readings" on the fucking article, which
is what Rob actually wrote. The link is: http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/ [mozilla.com]
Just to save you the trouble of reading it, if don't want to, it's pretty clear that IE9 is eliminating the heart of the math-cordic loop as dead code. It _is_ dead code, so the optimization is correct. What's weird is that very similar code (in fact, code that compiles to identical bytecode in some other JS engines) that's just as dead is not dead-code eliminated. This suggests that the dead-code-elimination algorithm is somewhat fragile. In particular, testing has yet to turn up a single other piece of dead code it eliminates other than this one function in Sunspider. So Rob filed a bug about this apparent fragility with Microsoft and blogged about it. The rest is all speculation by third parties.
Re:Benchmarks (Score:5, Informative)
Possibility one: Microsoft cheated. Presented as highly likely.
(I tend to agree that it's quite conceivable - other corporations have been caught doing similar things (like the NVIDIA/FutureMark debacle) and JavaScript execution speed is currently the most-hyped performance metric in the browser market.)
Possibility two: Microsoft have relied entirely on SunSpider when testing their JavaScript engine and over-optimized it to a point where it's now a SunSpider VM that happens to run JavaScript and doesn't work well with anything that isn't SunSpider. This is declared unlikely.
(Although I wouldn't put such a blunder past Microsoft, I do think that their tests extend beyond "how fast is SunSpider".)
Possibility three: The engine is legitimately ten times as fast as everyone else in this test but badly-written and so fragile that it experiences major slowdowns on code that meets currently-unknown criteria. Presented as unlikely.
(Note that in the Hacker News analysis [ycombinator.com] the general consensus now seems to be that IE indeed does something with the code that it shouldn't; an earlier theory of broken dead code analysis couldn't stand up to the fact that any change that causes the bytecode to look differently, even if functionally equivalent, causes slowdowns).
Re:Embarassing? (Score:5, Informative)
Did you look at the diffs [mozilla.com]? The addition of the "true;" operation should make absolutely no difference to the output code. It's a NOP. The fact that it makes a difference indicates that either something fishy is going on, or there is a bug in the compiler that fails to recognise "true;" or "return (at end of function)" as being deadcode to optimise away, and yet the compiler can apparently otherwise recognise the entire function as deadcode. Just to be clear, we are talking about a compiler that can apparently completely optimise away this whole function:
function cordicsincos() {
var X;
var Y;
var TargetAngle;
var CurrAngle;
var Step;
X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ /* AG_CONST * sin(0) */
Y = 0;
TargetAngle = FIXED(28.027);
CurrAngle = 0;
for (Step = 0; Step CurrAngle) {
NewX = X - (Y >> Step);
Y = (X >> Step) + Y;
X = NewX;
CurrAngle += Angles[Step];
} else {
NewX = X + (Y >> Step);
Y = -(X >> Step) + Y;
X = NewX;
CurrAngle -= Angles[Step];
}
}
}
but fails to optimise away the code when a single "true;" instruction is added, or when "return" is added to the end of the function. Maybe it is just a bug, but it certainly is an odd one.
This shows the dangers of synthetic non-realistic benchmarks. I was amused to read Microsoft's comments on SunSpider: "The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others." Indeed.
btw the Hacker News [ycombinator.com] discussion is more informative.
Re:I'm sure there's no hyperbole in this article (Score:5, Informative)
Everyone uses because it is a fairly objective benchmark.
For the record, I caught wind of this a month or two ago and posted it here in a firefox performance article. I was trolled and troll moderated despite pointing to the Mozilla team's own experiments.
The ONLY reasonable explanation, assuming you actually understand the implications of what is it you're (generalized readers, not your specifically) reading, which based on previous happenings is questionable, is that Microsoft is cheating their asses off by identifying the exact benchmark and returning a pre-computed value. Either that, or this is indicative of a horrible optimization bug which would negatively effect all javascript in their browser and it would be impossible for them to be competitive in the least. Given there is no evidence to support the later, the only reasonable conclusion is they are cheating their asses off in these benchmarks.
Re:Embarassing? (Score:3, Informative)
That's actually why there's blackout periods for insiders buying/selling shares, as part of the Sarbanes-Oxley rules. When I was at Dell, I wasn't allowed to buy/sell within 30 days (either way) of any public statement regarding earnings or future plans. These rules exist specifically to prevent insiders from gaming the market by buying up a large amount of stock right before an expected rise (higher than expected earnings announced, for example), or selling right before an expected fall (lower than expected earnings, for example). It's also why there was an obligation to announce any large sale by upper management as a matter of public record.
I'd be *very* surprised if Microsoft didn't play by the same rules, and I sincerely doubt that Ballmer's decision to sell had anything to do with Microsoft's performance as a company. MS could get in a *lot* of trouble with the SEC if it did, as they do take Sarbox seriously.
Re:Old news... (Score:4, Informative)
If you knew anything about JIT compilers, you would know that they have simple heuristics on purpose (compile speed is a strict constraint.) Making something 1 statement longer could remove it as a candidate for quite a few optimizations (inlining, static loop evaluation, loop unrolling, dead code elimination, etc..)
These simple heuristics use quickly evaluated metrics once the source is translated into an abstract syntax tree. The number of nodes in the tree.. the depth of the tree.. the number of conditional nodes..
JIT's are not simply compilers that try to produce the best code possible. JIT's make tradeoff decisions between compile time and the resulting code quality.
Dead Code Elimination (Score:3, Informative)
Hi, we've posted an official response and explanation at the bottom of this post:
http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx [msdn.com]
Bottom line - we built an optimizing compiler into the Chakra JavaScript engine that eliminates dead code where it finds it.
It's a fun conspiracy theory that we'd somehow 'game' SunSpider in this way, but it doesn't make sense that we'd go to all that risk to cheat on just one test out of 26.
Best wishes, Tim Sneath | Microsoft