Google Kills Off Octane JavaScript Benchmark Due To 'Diminishing Returns and Over-Optimization' (betanews.com) 88
Google has announced that its widely used Octane JavaScript benchmark is being retired, with Google saying that it's no longer a useful way for browser developers to determine how best to optimize their JavaScript engines. From a report: Google goes as far as saying that developers were essentially cheating the system. It says that compiler optimizations needed to achieve high benchmark scores have become common and, in the real world, these optimizations translate into only very small improvements in webpage performance. In fact, in some instances it was found that tactics used to boost benchmark performance actually had a detrimental effect on real-world performance. Developers exploited known bugs in Octane to achieve higher scores than were warranted, and Google believes the time has now come to retire the system completely.
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
OP is obviously older than the dot com bubble since OP can conceive of a world without Google.
I didn't read it that way.
How fucking stupid are you, fat ugly stupid troll.
I made the president's list for maintaining a 4.0 GPA when I got my A.S. degree in computer programming while working 60+ hours as a video game tester and teaching Sunday school after the dot com bust.
Re: (Score:2)
While I don't wish to lessen your earlier achievements, you place a great deal of emphasis on looking for some sort of approval that you believe you're due for early life achievement, but you don't follow up with "and since then I have accomplished...".
That's because I'm not here looking for approval.
Don't look for approval from people on Slashdot, you have to post something truly exceptional to be greater with anything but flames.
The purpose of Slashdot is to keep me amuse while I'm waiting for a script to finish at work.
Don't cheapen it to the point that people actually begin to remember you as "The guy who posts his high school resume on every thread"
This is Slashdot, not LinkedIn. No one gives a shit about what I do on Slashdot. On LinkedIn, I have connections to 800+ recruiters. So you won't see me post anything on LinkedIn.
Re: (Score:2)
1) video game tester: ridiculous joke of a job for stupid people
As a video game tester for six years, I wrote 30,000+ bug reports. As a lead video game tester, I was responsible for leading 10 titles through QA, trained three assistants to become lead video game testers, and worked extensively with developers all over the world. Stupid people don't last in this job if they don't figure out the difference between playing and testing video games.
2) associates degree in computer programming: you went to Devry because you couldn't get into college.
I transferred to university and got kicked out in my junior year for playing too many games of Magic: The Gathering. When I we
Re: (Score:2)
1) yeah man. that's what smart people with marketable skills want to do. write 30k bug reports. you're a fucking video game tester. you have no skills. testing video games is one step up for data entry. you are literally below helpdesk monkeys.
I was a help desk technician for nearly six years. I closed 300 to 700 tickets per month with a 98.8% SLA rate. I was always number three the department because the phone guys got all the five-minute tickets.
2) so... you got kicked out of college and got an a non-degree from the shithole where secretaries learn msoffice. gotcha.
Spreadsheet manipulation wasn't something I learned at college but on the job. One of my coworkers recently found a neat DOS trick for merging CSV files into one CSV file:
copy *.cvs single.cvs
Instead of working off of multiple spreadsheets, we can merge them together into one spreadsheet with a minim
Meaning (Score:1, Insightful)
Chrome isn't the best at this benchmark so we've retired it
Now if more benchmarks would take a bow... (Score:2)
Across the industry, benchmarks become a double edged sword when the industry embraces it too much and matures.
There are certain benchmarks that drive technology to make choices that can get them 2-3% wins compared to other things on the market, but that translates into real world performance that can be 50% slower in pretty much anything but the particular benchmark.
Re: (Score:2)
No, benchmarks do not become a double-edged sword. They are simply misused and misunderstood.
An automated software benchmark has value only to a developer or development team evaluating their own code. Anyone using such a benchmark for any other purpose (like comparing the performance of their code to the performance of others' code) is misusing the tool, and in doing so potentially drawing faulty conclusions.
Automated benchmarking tools are not a perfect measurement and they never will be. They are not
Goodhart's Law (Score:2)
Octane Recruiter... (Score:4, Informative)
A recruiter called yesterday about wanting to "octane my particular skill set" for a job.
https://twitter.com/cdreimer/status/852671049942446081 [twitter.com]
Re: (Score:1)
Please stop spamming your Twitter crap here.
You're confused. I was sharing something, not selling something.
Whatever you're going on about has nothing to do with the Octane benchmark that we are talking about here.
This is Slashdot. You must be new around here.
"Oh crap, other browsers beat us at our own test!" (Score:1, Insightful)
Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P
Re: (Score:2)
Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P
That would be insightful if it weren't for the fact that Chrome is one of the best benchmarked browsers in Octane.
But the difference between browsers is just so minimal that quite frankly no one should give a crap about these benchmarks anymore. It's nothing more than e-penis measurement at this point.
Re: (Score:2)
The fact that you said "one of the best" instead of "the best" is the point here. Any company that makes a benchmark (or other test) for a product they make never intends for a competitor to come out as #1.
Dumb. Chrome goes through Canary, Developer, and Beta channels before the general public ever sees a new build. You can use Octane on any of those. By your reckoning, of course no version of Chrome ever comes out behind. But what actually happens is the benchmark score bounces up and down with ongoing development of the engine.
Re: (Score:3)
Because that would be very un-google of them. Google rarely fixes known bugs, but frequently retires platforms.
If you are coding around a performance benchmark.. (Score:2)
As a professional developer: if you are coding around a performance benchmark, you are doing it wrong.
These kinds of tools exist for developers to evaluate the performance of their own code. Anyone who uses them in any other fashion (like to evaluate the performance of someone else's code, for example) needs to take the results with a grain of salt, and that will always be the case for any automated software benchmark.
That Google feels the need to retire Octane over this is almost unbelievable... there mu
Re: (Score:3)
That Google feels the need to retire Octane over this is almost unbelievable... there must be some ulterior motivation.
Why that assumption? Google explained its reasons quite clearly:
Investigations into the execution profile of running Octane versus loading common websites (such as Facebook, Twitter, or Wikipedia) revealed that the benchmark doesn’t exercise V8’s parser or the browser loading stack the way real-world code does. Moreover, the style of Octane’s JavaScript doesn’t match the idioms and patterns employed by most modern frameworks and libraries (not to mention transpiled code or newer ES2015+ language features). This means that using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents.
In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.
If you think about the above, consider also that every JavaScript engine in use today that I can think of is open source. That means the projects accept contributions from independent developers all over the world. Many of those developers may be submitting patches designed to improve the performance of the engine. It may even be that most of the patches are designed to improve performance. But if the "proof" that the patches increase performan
Lesson re-learned: (Score:2)
People are smarter than computers.
hmmmmm (Score:2)