Internet Explorer 9 Caught Cheating In SunSpider 360
dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."
I'm sure there's no hyperbole in this article (Score:2, Insightful)
Welcome to your daily two minutes hate, Slashdot.
Re:Embarassing? (Score:2, Insightful)
They're kinda like the rich fat cat who constantly puts his foot in his mouth. He knows he should shut up, but then again why should he care...he's rich, bitch!
Benchmarks (Score:5, Insightful)
This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
Re:Benchmarks (Score:3, Insightful)
This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.
Except that the article writer tries to claim that that couldn't possibly be the case and thus claims that Microsoft is "cheating" instead. Basically this is an invented controversy.
Re:Embarassing? (Score:1, Insightful)
I would think Microsoft would be used to embarassing by now..
Embarassment requires a sense of shame.
Do not attribute to malice ... (Score:5, Insightful)
what can be attributed to stupidity.
1) Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2)Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
I see no reason why explanation number one is more likely than explanation number two.
Really? You're Going with That? (Score:5, Insightful)
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
Some things to consider: 1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients. 2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.
Just something to consider when you say we shouldn't be shocked by this.
Three explanations FTFA (Score:5, Insightful)
Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.
Re:I'm sure there's no hyperbole in this article (Score:1, Insightful)
Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.
Cheating allegation too strong (Score:4, Insightful)
Meh I think claiming they are cheating with no evidence seems a little too out there. I've never seen MS brag about how fast their browser is on this particular benchmark, and frankly seems more like a bug than a cheat.
Re:Do not attribute to malice ... (Score:3, Insightful)
Re:Do not attribute to malice ... (Score:4, Insightful)
Never attribute to malice that which is adequately explained by stupidity. [wikipedia.org]
Re:Benchmarks (Score:2, Insightful)
It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.
For that to be true, you'll need to demonstrate that they put more effort into scoring well on the benchmark than they did in improving performance in general. I don't think you can.
Improving performance in general is worth doing and I'm sure it's being done, but it's hard. Improving performance on a benchmark dramatically is often not that hard, and it's worth doing if it gets your product noticed.
I'm sure all browser makers are doing the exact same thing on both counts -- anonymous Mozilla guy is just bitter because he did a shittier job of the less-important task of benchmark performance.
Re:Internet Explorer? (Score:1, Insightful)
> And why bother cheating when you have 51% of the market share?
Because that 51% used to be 91%?
Re:Embarassing? (Score:4, Insightful)
Thanks for someone pointing this out. I mean really, if they were going to throw this test why would they throw it quite this much? And is this the ONLY portion of this test that seems to act this way? If so then why in the world would they throw only this portion and why this much? The original result was uber fast, the result on the modified test pretty slow - if they were going to try and hide something why make it uber fast and not just slightly better?
Something is weird, possibly hinky, but to outright declare cheating based just on this? Really? O_o
Re:Do not attribute to malice ... (Score:3, Insightful)
I see no reason why explanation number one is more likely than explanation number two.
I do. Given the nature of the changes that were used to uncover this, to me (as a programmer) it seems very unlikely that such over-optimization could happen in such a way that it would degrade so severely with those changes. Here is what was changed (look at the 2 diff files linked near the bottom of the article):
1) A "true;" statement was added into the code. It was not an assignment or a function call, or anything complex. Just a simple true statement. Depending on the level of optimization by the interpreter/compiler, this should turn into a noop at best, or simply loading a constant into a register or memory location (and then doing nothing more with it) at worst..
2) A "return" added at the end of a function, instead of using the implicit return. This also should have minimal impact. In the best case, the code generated should be identical to the return code already generated by the implicit return. In the worst case, the compiler might be sloppy and generate 2 return instructions (one for the explicit return, and another unreachable instruction for the implicit return).
So from my experience, it seems EXTREMELY unlikely that this could have happened by accident just through optimization. I'm not going to go so far as to say it CAN'T happen, because I know from experience that things can get really complex and things not so obvious can happen (like cache issues as mentioned in the article). However, in all likelihood, I suspect this is going to be just like when video card drivers detect the filename of the executable and optimize specifically for that (I think it was quake3.exe that we saw that happen with before).
Re:Old news... (Score:4, Insightful)
It is actually a couple of months old [mozilla.com]. The thing that makes me doubt the claims of cheating is that nobody has been able to find other examples of performance variations in this benchmark in all the time since this came to light. If they were going to cheat, why limit it to the cordic test? Nobody would base their browser choice on this obscure test.
I don't have the beta installed yet, but what I would like to see is the actual calculation changed and then run the tests again. Don't just put in weird code like "true;" but make the javascript plausible. It could be that the addition of these unusual statements are enough to confuse the optimiser so that it resorts back to a completely unoptimised version.
Re:Benchmarks (Score:2, Insightful)
In none of the cases is MS doing something legitimately. Optimizing to one test is invariably a bad idea, no matter how well designed, and quite honest at this point they should be able to code an engine that's a lot more resilient than that.
Re:Embarassing? (Score:5, Insightful)
It is like when ATI/Nvidia made their drivers do some funky shit on the benchmarks to make their products seem way better; This was also called cheating at the time.
If Microsoft is cheating... (Score:3, Insightful)
If Microsoft is cheating, why wouldn't they cheat a bit better? Of the five browsers, including betas, IE is second from last [mozilla.com]. Last place is, of course, Firefox, even with the new JS engine. Oh, and that stats image? Taken from the same blog post [mozilla.com] that originally discovered the Sunspider IE9 issue over a month ago.
Rob Sayre, the Mozilla Engineer who discovered this, filed a bug [mozilla.com] with Microsoft to get them to look at this issue. However, he didn't file said bug until today, which is likely why this is in the news now rather than a month ago.
Mods, please RTFA! (Score:4, Insightful)
Really? Are you sure about that that?
Second paragraph of the article:
The parent is not insightful, it is merely a troll.
Re:Embarassing? (Score:3, Insightful)
The problem is that that is the most logical conclussion.
The other possible conclussions are both more unrealistic and worse for MicroSoft.
The benchmark modifications were trivial and non-functional; they shouldn't have made that big of a difference.
Re:Old news... (Score:3, Insightful)
Putting a
return;
at the tail of a function is unusual? Are you high?
Re:Embarassing? (Score:1, Insightful)
The driver stuff was damning because if you renamed the test executable performance would suffer. They were special-casing doing things fast-but-wrong only if you were a known benchmark. Haven't seen anything saying that IE is treating the test differently - maybe they just optimized specific tested functionality in general, which isn't nearly as bad. That's not cheating, that's just teaching the test.
Re:Embarassing? (Score:3, Insightful)
The purpose of a benchmark is to try to show how performance will be in the real world. If a given application has been programmed to do very well in a given benchmark yet does not do as well with a real-world situation, then the benchmark results are flawed. The idea of coding an application just to have good benchmark numbers that would not be seen in the real world is considered cheating. In this case, we are talking about JavaScript speeds, so you would be very surprised if you believed that IE 9 was really fast due to the benchmark results, yet was really slow going to other sites that use JavaScript.
Re:Embarassing? (Score:4, Insightful)
Well, yes, taking some of your money. But since the only way that you can make money is because the wider society sees that as a benefit, suck it up and pay your goddamned taxes. This illusion that somehow the money in your pocket came to you by yourself alone is the greatest lie of Libertarianism.
Re:Embarassing? (Score:5, Insightful)
Socially liberal does not mean "spend money".
Any other definition necessarily requires taking my money and giving it to someone else.
Ah, the "anti-tax" argument. I'm happy with taxes. Honestly. Do I wish they were lower - of course. Do I think that we spend money on stupid things? Yep.
Put taxes are still cheaper than having my own private doctor and hospital, my own roads, my own water towers and power generation, my own private library, swimming pool, and so on. Governments should do these things, because it's cheaper for everyone to pitch in.
Comment removed (Score:2, Insightful)
Re:I'm sure there's no hyperbole in this article (Score:3, Insightful)
Let's review that argument, shall we?
Statement 1: The ONLY reasonable explanation is that they are cheating. Or there is an optimisation bug which screws performance.
Comment: I think you need to look at meaning of "ONLY" as you have used it, and the way the rest of the world uses is.
Statement 2: there is no evidence of an optimisation bug, therefore it must be cheating
Comment: One could plausibly argue that there is no actual evidence of cheating, therefore it's an optimisation bug. Since there is no internal evidence of any kind.
Yes, it COULD be cheating, but you can't argue that it must be so based on the information in the article. There are enough weasel phrases in your analysis to populate a weasel world theme park.
Re:I'm sure there's no hyperbole in this article (Score:5, Insightful)
1) Microsoft beats everyone else by a factor of 10.
2) Making any of a number of effectively cosmetic changes to the function results in Microsoft taking twice as long as everyone else.
3) Making the inner loop 10x longer makes everyone else take 10x longer, except MS, who takes 180x longer.
Sorry, but if that counts as an optimization "bug", I have a bridge to sell you.
Re:Embarassing? (Score:2, Insightful)
libertarians are reluctant to spend other people's money.
Republicans and Democrats do not.
libertarians warn of the dangers of an authoritarian state.
Republicans and Democrats embrace the authoritarian state. It helps with their warfare/welfare agenda.
libertarians don't want to save the world, just their neighbors.
Republicans and Democrats each are 100% sure they know best how to save the world either from terror or poverty or inquality, at gun point if necessary. They don't give a rat's ass about their neighbors though.
real libertarians don't want to even be called Libertarian or libertarian, in fact they find the idea of collectivist labeling the domain of left brain sheeple.
when I was a kid, I was a democrat, then I grew up and became a republican, then I realized that both parties were filled with lying sons of bitches, and became Libertarian, then found that card carrying Libertarians are full of hot air, so I became libertarian (small L), then realized that I didn't fit that mold either.
Now I find most politicians, partisans, fox news, cnn, msnbc, etc, and a surprisingly large number of slashdotters to be lying, self deluded sacks of shit.
Don't be surprised if you fall into that category.
I didn't make the world. I just live in it.
Re:Embarassing? (Score:2, Insightful)
Re:Embarassing? (Score:4, Insightful)
I think by "socially liberal" he means "nothing should be illegal unless it harms someone else." But as to fiscal conservatism, that's the Libertarian Party, not libertarians in general. Personally, I agree that anything that doesn't trample someone else's rights should be legal, but at the same time I'd like to see universal health care, and have the poor taken better care of. "There but for the grace of God go I".
And I'd like to see corporate power reigned in, and I'd like the corporates to stop getting government welfare and getting away without paying taxes. So I seldom vote for a Libertarian candidate.
Re:Embarassing? (Score:4, Insightful)
The benchmark in question can be considerably optimized by dead code elimination, since a computationally expensive function in there (one that loops computing stuff) does not have any observable side effects, and does not return any values - so it can be replaced with a no-op. It is a perfectly legitimate optimization technique, but the one which tends to trip up naively written benchmark suites because they assume that "for(int i=0; i < 1000000; ++i) {}" is going to be executed exactly as written.
Thee was actually a similar case with artificial tests in the past - Haskell (GHC) scores on the Programming Language Shootout. Most tests there were also written as loops with some computations inside and no side-effects, on the presumption that compilers will leave the computations intact even though their results are never used. Well, one thing that GHC has always had is a particularly advanced dead code eliminator, and it compiled most of those tests down to what is essentially equivalent to "int main() { return 0; }" - with corresponding benchmark figures. Once they've changed the tests to print out the final values, this all went back to normal.
In this case it's not quite that simple, because seemingly trivial changes to benchmark code - changes which do not change the semantics of the code in any way - trip off the dead code elimination analyzer in IE9 JS engine. However, it is still an open question on whether it is deliberate, or due to bugs in the analyzer. One plausible explanation was that analyzer is written to deal with code which at least looks plausible, and neither of the suggested optimizer-breaking changes (inserting an extra statement consisting solely of "false;" in the middle of the function, or "return;" at the end of it) make any sense in that context. Any dead code elimination is necessarily pessimistic - i.e. it tries to guess if the code is unused, but if there are any doubts (e.g. it sees some construct that it doesn't recognize as safe) it has to assume otherwise.
The only true way to test this is to do two things:
1. Try to change the test in other ways and see if there are any significant diffs (such that they are not reasonably detectable as being the same as the original code) which will still keep the optimizer working.
2. Write some new tests specifically to test dead code elimination. Basically just check if it happens on completely different test cases.
By the way, the guy who found the discrepancy has opened a bug [microsoft.com] in MS bug tracker regarding it, in case you want to repro or track further replies.
Re:Embarassing? (Score:3, Insightful)
All JS functions return values. If no value is specified in the "return" statement, or if the return happens due to reaching the end of the function, "undefined" is returned. So adding a "return;" at the end of the function which does not otherwise return anything does not change its meaning in any way.
That said, it is quite possible that the optimizer does not know this, and treats any "return" as a signal that the function returns a meaningful value. Which then indicates a bug in the optimizer.
That doesn't adequately explain why "false;" is not optimized away consistently, though.
Re:Embarassing? (Score:4, Insightful)
The return statement was "return;" i.e. a return statement that did not return anything. Looking at the other JavaScript engines, that line added at most 1ms, while with the IE engine it added 19ms. If the IE9 JS engine is handling this function in a super-efficient way that is not due to cheating, the optimisation must be highly sensitive to variance.
One way to check if the IE9 engine is doing some sort of special casing (e.g. hashing the text for the function) would be to change the name of a variable. This should not change the behaviour of the engine as it is the same code (there are no extra elements in the tree, like additional returns). If the IE9 engine is cheating, this should jump from 1ms to 20ms like the other variances. If it is an optimisation bug, the performance should be 1ms for both cases.
Re:Do not attribute to malice ... (Score:3, Insightful)
Viola, you have the exact behavior seen here - no cheating necessary.
When you make the for-loop count backward, it suddenly decides to execute it. Explain that. (http://news.ycombinator.com/item?id=1913315 [ycombinator.com], “Edit (like...#14)”).