Intel Performance Strategy Team Publishing Intentionally Misleading Benchmarks (servethehome.com) 42
An anonymous reader shares a post: This week something happened that many may not have seen. Intel published a set of benchmarks showing advantage of a dual Intel Xeon Platinum 9282 system versus the AMD EPYC 7742. Vendors present benchmarks to show that their products are good from time-to-time. There is one difference in this case: we checked Intel's work and found that they presented a number to intentionally mislead would-be buyers as to the company's relative performance versus AMD.
Classic Intel (Score:5, Insightful)
I'm not even shocked, this is really par for course. They have a long rich history of anti-competitive behavior and frankly, the fact that they haven't been smacked down by the FTC is disappointing.
Re:Classic Intel (Score:4, Informative)
They have been smacked down by the FTC repeatedly, they just keep doing it. What it really means is that Intel doesn't see the fines as an issue, not even an issue that impacts their bottom line.
Re:Classic Intel (Score:4, Insightful)
They have a long rich history of anti-competitive behavior
This isn't anti-competitive behaviour, this is marketing, and EVERY company is guilty of it basically on a continuous basis.
I found it very hard to get upset at Intel's misleading benchmarks when AMD published boost figures for Ryzen 3000 which only about 10% of their processors actually hit. Then there's the likes of benchmark specific optimisations that are done.
This isn't even desktop CPUs. It's done in GPUs, it's done by ARM, it's done by Apple and Samsung. It's done for all pieces of hardware, and shit it was even done for software (Browser ACID test anyone)?
The only thing you can do is never trust a benchmark or comparison published or paid for by one of these companies.
Sidenote: This is small fry. What really annoyed me was Intel's dishonest benchmark where they paid a 3rd party which carefully tuned and de-tuned the BIOS in the Intel and AMD systems respectively to misrepresent performance under the guise of being "independent". That was incredibly dodgy.
This is just marketing.
Re: (Score:1)
A crime, U.S. Code title 15 [cornell.edu]
Liable as a RICO as well for all fraudulent gains x 3
Re: (Score:2)
Lying in advertising
Show where anyone has lied. Misrepresenting the truth by careful selection of data is ... advertising.
Liable as a RICO as well
The great thing about people who invoke RICO is that they straight away give themselves away as now having a clue how the legal system ever works. Hint: It's never RICO.
Re: (Score:2)
They're not going to overly honest about the downsides of their products, after all it's marketing and they're trying to sell it. And it doesn't matter whether it's Intel, AMD, nVidia, or manufacturers that use their GPUs on their own custom PCBs.
For example I remember the AMD performance charts for the first Ryzen generation. Where they showed that their CPUs were on par with Intels in gaming. But omitt
Intel has been misleading their customers (Score:4, Insightful)
Intel has been misleading their customers for years, why anyone would consider Intel as their best option on price or performance is beyond me.
One of the main problems with Intel is their design is not lead by engineers, key architectural design decisions are not made by engineers, this is most obvious with a garbage that is the CPU cache and all the problems that surround if the cache implementation was designed by an engineer then the cache would be directly accessible and controllable by the programmer or compiler writer, as it stands now the most important performance feature of the CPU is randomly populated and de-populated by a proprietary algorithm which has so many flaws in it its is ridiculous, and I am not just referring to the serious security flaws there are also some serious performance flaws that compiler optimiser's attempt to work around without being given the ability to directly control the resources.
While Intel's focus is on manufacturing technical products, if they do not move back to engineer lead design they will continue to lose ground in the market place and will never again be a market leader, because their decisions about what makes a good product and the decisions about how to manufacture a a good product will not be based on the principals of good engineering.
Instead of investing in making better products they are investing in lying about their sub-par products- a very clear sign they have no idea what they are doing.
Re: (Score:3)
I'm confused by what I'm reading. Are you saying what, the janitorial staff dictated the design? Maybe.. human resources? Or was it a VP of marketing who said "You do the cache this way, I really care about the CPU cache!"?
Must be some pretty fucking smart cafeteria workers that designed it and dictated the design to the engineers! Christ, some of the shit I read around here.
Re: (Score:3, Interesting)
Re: (Score:2)
Graphene Or Borophene CPUs Might Help (Score:2, Interesting)
Re: (Score:3)
If you can't beat your rivals with Silicon based CPUs, why not try something different Intel?
They can't even get their silicon process working, now they're supposed to do something radical? GLWT
I agree it would be nice to see something extraordinary, instead of Intel's business-as-usual: attempting to mislead and misdirect customers away from the facts about their malfeasance.
Time to change an analogy (Score:5, Interesting)
Benchmarks are like Bikinis. What they reveal is interesting, what they hide is crucial.
Re: (Score:2)
Are they hiding a penis?
Grammar matters (Score:2)
To quote the article:
"Vendors present benchmarks to show that their products are good from time-to-time."
I do believe he meant to say, "Vendors present benchmarks from time-to-time to show that their products are good."
As stated, vendors try to show that their products work well only sometimes, and sometimes not.
As stated, it is a very prescient grammar slip up, accidental accuracy. (Or, is that what they really meant?)
Comment removed (Score:5, Funny)
Inflammatory Title. (Score:2)
The word "intentionally" was used in the title but gave no evidence that it was intentional.
The problem they complain about was using the 2019.3 version of GROMACS instead of the 2019.4 version. They updated the result with the 2019.4 version and the result was a little different but not a lot.
So the use of 'intentionally' is just clickbait.
Re:Inflammatory Title. (Score:4, Informative)
The problem they complain about was using the 2019.3 version of GROMACS instead of the 2019.4 version. They updated the result with the 2019.4 version and the result was a little different but not a lot.
According to TFA the 2019.4 version was out for over a month before Intel published their results. And their results only showed them beating the AMD chip by a little bit. So as it turns out, a little difference can actually be a big difference — if it's the difference between being faster (as Intel claims) or slower (as the current version of the benchmark for a month before they published shows.) It's the difference between 0 and 1.
Re: (Score:2)
I'm not arguing that there's a difference. Read my post. I'm saying there is no reason to think it's an intentional misleading. It's not even misleading - the results are what was measured with the same code on both platforms. That new code appeared while the tests were being developed does not indicate that they intentionally held back the later version. It's entirely feasible that they just ran with the code they had. I don't know these people but I wouldn't assume they intentionally ignored the update.
Re: (Score:2)
I don't know these people but I wouldn't assume they intentionally ignored the update.
It was either intentional or incompetent. So... par for the course at Intel.
Re: (Score:2)
I don't know these people but I wouldn't assume they intentionally ignored the update.
It was either intentional or incompetent. So... par for the course at Intel.
or the results were simply obtained before the newer version was available to them!
Re: (Score:2)
Seriously, just stop. They've been caught before. Multiple times. Now you are here saying how innocent they might be. That ship sailed a long time ago.
Re: (Score:2)
or the results were simply obtained before the newer version was available to them!
From TFA:
On October 2, 2019 the GROMACS team released GROMACS 2019.4
From the Intel's notes (https://www.intel.com/content/www/us/en/benchmarks/2019-xeon-scalable-benchmark.html):
Intel measured as of October 8, 2019
So there goes that...
That's one of several problems (Score:5, Informative)
That's one of the several issues that the article points out.
They also ran two threads per core on Intel, limited it to one thread per core for AMD. They a different and suboptimal type of memory for AMD, etc. Lots of little things.
Re: (Score:2)
Did you read it or skim it? The 1 tpc was a typo.
Re: (Score:2)
Do you care to point out this typo?
I read enough to see that there were several different issues with the methodology. Frankly I don't care about the details. What's clear is that, shockingly, Intel likes to show their products in a positive light.
Re: (Score:2)
I read the whole thing and found two substantive complaints. The version and the typo. Not several.
Re: (Score:2)
The word "intentionally" was used in the title but gave no evidence that it was intentional.
So they "accidentally" ran 2 different benchmarks in systems setup in different ways to get a favourable result?
Re: (Score:2)
The word "intentionally" was used in the title but gave no evidence that it was intentional.
So they "accidentally" ran 2 different benchmarks in systems setup in different ways to get a favourable result?
They ran the same benchmark, or that's what TFA said..
Re: (Score:2)
There are lies, damn lies, and benchmarks (Score:2)
"There are three kinds of lies: lies, damned lies, and benchmarks." ~Mark Twain V2.0
Seriously, if you plan to run simulations and don't bench on trail hardware, you deserve what you get.
I stopped trusting vendor benchmarks 2 decades ago when Intel tried to convince me Hyperthreading was as good as a second CPU core.
Re: (Score:1)
"I stopped trusting vendor benchmarks 2 decades ago when Intel tried to convince me Hyperthreading was as good as a second CPU core."
It was as good as a second core, originally, until programmers forgot how to write efficient fucking code.
Re: (Score:3)
Stop sucking Intels insecure cock.
Re:There are lies, damn lies, and benchmarks (Score:4, Interesting)
It was as good as a second core, originally, until programmers forgot how to write efficient fucking code.
No, it was not, and even Intel never claimed that it was. They always said it increased performance by wide margins only on some workloads. If both threads on a core need to retire instructions that use the same functional units, which is sometimes unavoidable, there is little to no benefit — since the cost of context switches was already reduced dramatically by register renaming.
yawn! (Score:3)
Intel Compiler used (Score:2)
The article does not raise this topic specifically: the compiler that was used was Intel Compiler. Wikipedia has this to say about it (https://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler):
The compilers generate optimized code for IA-32 and Intel 64 architectures, and non-optimized code for non-Intel but compatible processors, such as certain AMD processors.
Other libraries seem to be also Intel-optimized. And they are using RedHat Enterprise Linux for Intel vs. CentOS for AMD.
For select benchmarks, they are actually using the AMD optimizing compiler, but it is known to actually provide worse performance in select cases. Makes me wonder if they cherry-picked such cases.
The biggest thing I care about: (Score:2)
Have they fixed any of the silicon flaws that require software mitigations which slow down performance?
If not, did they properly patch the systems against all the vulnerabilities?
Who cares if they're 20% faster, if adding the mitigations slows down performance 30%?
I think it's safe to say.. (Score:1)