A Look At Microsoft's 'Mini Internet' For Testing IE 241
MrSeb writes "With the grandiose bluster that only an aging juggernaut can pull off, Microsoft has detailed the Internet Explorer Performance Lab and its extraordinary efforts to ensure IE9 is competitive and IE10 is the fastest browser in the world. Here are a few bullet points: 128 test computers, 20,000 tests per day, over 850 metrics analyzed, 480GB of runtime data per day, and a granularity of just 100 nanoseconds. The data is reported to 11 server-class (16-core, 16GB of RAM) computers, and the data is stored on a 24-core, 64GB SQL server. The 'mini internet' has content servers, DNS servers, and network emulators (to model various different latencies, throughputs, packet loss)."
And still... (Score:4, Insightful)
Was /. been bought or what? (Score:1, Insightful)
What is this now? MSDN blogs? Seriously, gimme old slashdot back please!
Re:And still... (Score:5, Insightful)
And when all we care about is the fastest browser - in nanoseconds! - will we begin to forget the truly important criteria for choosing a browser?
Or better still, by the time IE is on par with Chrome the actual browser will be irrelevant because mobile platforms - in which IE has little share - will do to traditional computers what Cromagnons did to Neanderthals. The next generation will use integrated devices, unaware they were using a browser, and with little or no need for even a choice.
Re:Could use the real internet eh! (Score:5, Insightful)
Re:IE Crap (Score:5, Insightful)
Correctly according to whom? Neither CSS3 nor HTML5 are completed standards and various portions remain in flux. Of the more mature bits IE9 and IE10 implement quite a bit of it and do so quite comprehensively.
MS is also one of very few organizations that is very actively involved with the W3C Test Suite by submitting test cases for each portion of the standard under various circumstances to demonstrate correct behaviors. What Google and Mozilla do instead is slap together a partial implementation and call it a day. More than once has their implementations been found to be not only incomplete but also incorrect.
Stop relying on scores given by non-authoritative tests demonstrating exceedingly limited and selective interpretations of non-standardized functionality. Oh, and HTML5 Video does NOT specify a codec, in fact it was designed to handle many simultaneous codecs, including h.264, which is explicitly referenced in the draft.
Re:But will it run Linux? (Score:4, Insightful)
I couldn't resist. But with all the work and effort and resources going into this, how is it that operations a tiny fraction of this can generate fast, reliable and standards complaint browsers better than MSIE?
Microsoft, the problem isn't that you're not spending enough money. It's that you're not doing it right.
I'm not familiar with IE 8 and 9, but in the past the issue was many years and revisions of code reuse and accumulation of cruft, an insane amount of backwards compatibility and some poor initial design choices combined to make each new version bigger, slower and buggier. I imagine this combines to make developing and especially testing any new release a massive undertaking.
To be fair, I would argue that Firefox is starting that downward spiral now. Each new version is slower and has a bigger footprint. Personally, I've switched to Chrome, but when Chrome inevitably starts lagging, I'll be on the lookout for the next completely new browser. Not merely because it's new, but because it's less likely to have years of bad decisions weighing it down.
The problem in Microsoft's case is that they seem incapable of dumping what they have and doing a complete rewrite. There may be marketing reasons for this, granted, but if they made a clean break it'd be better for them in the long run.
Re:Could use the real internet eh! (Score:2, Insightful)
they don't want all those fancy test clients to be picking up the latest in drive-by syphilis; even microsoft knows better than to go on the real internet with explorer.
Re:Could use the real internet eh! (Score:5, Insightful)
You have to figure out what the variables that you have problems with in real world usage, before you can start optimising your product to account for them.
There has to be iterative cycles of real world, then fake internet testing to really make it work well.
It would also help if you were able to test your competition alogn the same lines.
I additionally wonder if they are accouting for all of the different behavious of all of the various webservers out there. If they are only testing agianst iis, well, that's not very good.
Re:Could use the real internet eh! (Score:4, Insightful)
Re:And still... (Score:4, Insightful)
Firefox hasn't become any slower. What's happened is that everything else become so much faster. There used to be a "pregnant pause" when entering a domain, I remember when 30 seconds to load a page was the norm, over 56k.
Now, I expect the results from a google search to appear as I type, interactively. This isn't just an improvement, it's a whole 'nother animal at that level of performance.
Re:Granularity of 100 nanoseconds (Score:4, Insightful)
It means the runtime data they have collected is stored in a 100ns interval.
Re:Could use the real internet eh! (Score:5, Insightful)
So, what's going to make anyone use a Microsoft browser? They've been losing market share for ages because their browsers suck. How do they get people back? Make a good browser. Your argument might mean something if Microsoft sold IE as a standalone product, but they don't. It costs nothing (in terms of cash coming straight out of your pocket) to switch browsers, and users are notoriously not fond of switching. Since you can't count on your competitors' products to be lousy, you can only compete by making yours better. The browser market is about as Darwinistic as a software market could be.
Firefox isn't slow at all. (Score:5, Insightful)
While slashdot mocks the computer industry marketing for describing computers using a single metric, you seem to be quite happy with that when it comes to browser performance.
An example: Chrome (v8 engine) has this reputation for amazing speed, but IE9 absolutely grinds Chrome into the dust when it comes to simply repositioning elements on screen; something which today's web apps spend a lot of their time doing. You can feel it too if you know what you're looking for. I don't follow IEs development as closely as Chrome or Firefox, but IE must be hardware accelerating these translations.
I fully expect Google to focus on performance cases which help their specific apps. Again, a conflict of interest, akin to Microsoft pre-caching masses of junk, so that Office can appear to start up much faster than the competition.
Why not? (Score:5, Insightful)
MSDN blogs are often very technically detailed, written by people who know this stuff from the inside, and if it's about a topic that's of general computing interest then it seems that's a good thing. And the blog in question is chock full of some really good detailed stuff about how they're doing performance testing, reasons why the lab is architected the way it is, detailed graphics on how they measure performance, how they analyze it...on and on.
Frankly, this seems more akin to old Slashdot than a lot of the nonsense we see here today. (That story the other day about a girl sent home from school because her lunch wasn't healthy, and then quickly called into question over what happened? Really? What was that topic even doing on Slashdot in the first place?) Whatever you think about Microsoft, having this extremely detailed look into how one of the world's biggest software vendors (or are they the biggest now?) goes about performance testing, and how they ensure consistent results, should be really, really interesting to anyone involved in IT.
Re:Could use the real internet eh! (Score:5, Insightful)
Rule #2 of IT that should never be broken [earthlink.net]: Never let a web designer design your web page.
Giving free reign to a web designer to design a web site is like giving a two year-old a Faberge egg.
Re:Could use the real internet eh! (Score:4, Insightful)
Probably an evolving tick-tock setup between real and test.
Step1. Performance Test App against real world
Step2. Document Real world issues
Step3. Create test environment to run your issue cases against
Step4. Optimize app against test environment
Step5. Goto Step1, adding any new cases
Steps 1 and 2 can happen independent of 3 and 4. Step5 is just to make the logic seem serial.
Re:IE Crap (Score:2, Insightful)
Um, no. Standards word when the standardization committee defines the behaviors, argues through the issues and then rubber stamps the behavior. They don't just take the accidental similarities in draft implementation between Google and Mozilla and standardize "do whatever they did." CSS2 is final and Chrome doesn't implement all of it; just see what happens if you try to combine the first-line selector with a text-transform of lowercase. Conversely there are behaviors which all of the browsers have more or less agreed upon but which are entirely non-standard, like window.screenX.