Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Google The Internet

Google Kills Off Octane JavaScript Benchmark Due To 'Diminishing Returns and Over-Optimization' (betanews.com) 88

Google has announced that its widely used Octane JavaScript benchmark is being retired, with Google saying that it's no longer a useful way for browser developers to determine how best to optimize their JavaScript engines. From a report: Google goes as far as saying that developers were essentially cheating the system. It says that compiler optimizations needed to achieve high benchmark scores have become common and, in the real world, these optimizations translate into only very small improvements in webpage performance. In fact, in some instances it was found that tactics used to boost benchmark performance actually had a detrimental effect on real-world performance. Developers exploited known bugs in Octane to achieve higher scores than were warranted, and Google believes the time has now come to retire the system completely.
This discussion has been archived. No new comments can be posted.

Google Kills Off Octane JavaScript Benchmark Due To 'Diminishing Returns and Over-Optimization'

Comments Filter:
  • Meaning (Score:1, Insightful)

    by Anonymous Coward

    Chrome isn't the best at this benchmark so we've retired it

  • Across the industry, benchmarks become a double edged sword when the industry embraces it too much and matures.

    There are certain benchmarks that drive technology to make choices that can get them 2-3% wins compared to other things on the market, but that translates into real world performance that can be 50% slower in pretty much anything but the particular benchmark.

    • No, benchmarks do not become a double-edged sword. They are simply misused and misunderstood.

      An automated software benchmark has value only to a developer or development team evaluating their own code. Anyone using such a benchmark for any other purpose (like comparing the performance of their code to the performance of others' code) is misusing the tool, and in doing so potentially drawing faulty conclusions.

      Automated benchmarking tools are not a perfect measurement and they never will be. They are not

    • "When a measure becomes a target, it ceases to be a good measure".
  • Octane Recruiter... (Score:4, Informative)

    by __aaclcg7560 ( 824291 ) on Friday April 14, 2017 @02:03PM (#54236003)

    A recruiter called yesterday about wanting to "octane my particular skill set" for a job.

    https://twitter.com/cdreimer/status/852671049942446081 [twitter.com]

  • by Anonymous Coward

    Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P

    • Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P

      That would be insightful if it weren't for the fact that Chrome is one of the best benchmarked browsers in Octane.

      But the difference between browsers is just so minimal that quite frankly no one should give a crap about these benchmarks anymore. It's nothing more than e-penis measurement at this point.

  • As a professional developer: if you are coding around a performance benchmark, you are doing it wrong.

    These kinds of tools exist for developers to evaluate the performance of their own code. Anyone who uses them in any other fashion (like to evaluate the performance of someone else's code, for example) needs to take the results with a grain of salt, and that will always be the case for any automated software benchmark.

    That Google feels the need to retire Octane over this is almost unbelievable... there mu

    • by PCM2 ( 4486 )

      That Google feels the need to retire Octane over this is almost unbelievable... there must be some ulterior motivation.

      Why that assumption? Google explained its reasons quite clearly:

      Investigations into the execution profile of running Octane versus loading common websites (such as Facebook, Twitter, or Wikipedia) revealed that the benchmark doesn’t exercise V8’s parser or the browser loading stack the way real-world code does. Moreover, the style of Octane’s JavaScript doesn’t match the idioms and patterns employed by most modern frameworks and libraries (not to mention transpiled code or newer ES2015+ language features). This means that using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents.

      In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.

      If you think about the above, consider also that every JavaScript engine in use today that I can think of is open source. That means the projects accept contributions from independent developers all over the world. Many of those developers may be submitting patches designed to improve the performance of the engine. It may even be that most of the patches are designed to improve performance. But if the "proof" that the patches increase performan

  • People are smarter than computers.

  • Chrome no longer gets top score.... This benchmark is irrelevant and time to be decommissioned. I am sure this is just pure coincidence!

If it's worth hacking on well, it's worth hacking on for money.