Intel Caught Cheating In 3DMark Benchmark 216
Posted
by
kdawson
from the try-something-smarter-than-detecting-the-app-name dept.
from the try-something-smarter-than-detecting-the-app-name dept.
EconolineCrush writes "3DMark Vantage developer Futuremark has clear guidelines for what sort of driver optimizations are permitted with its graphics benchmark. Intel's current Windows 7 drivers appear to be in direct violation, offloading the graphics workload onto the CPU to artificially inflate scores for the company's integrated graphics chipsets. The Tech Report lays out the evidence, along with Intel's response, and illustrates that 3DMark scores don't necessarily track with game performance, anyway."
Hmm... (Score:5, Informative)
On the other hand, though, the old "run 3Dmark, then run it again with the executable's name changed" test looks pretty incriminating. Historically, that has been a sign of dodgy benchmark hacks.
In this case, however, TFA indicates that the driver has a list of programs for which it enables these optimizations, which includes 3Dmark, but also includes a bunch of games and things. Is that just an extension of dodgy benchmark hacking, taking into account the fact that games are often used for benchmarking? Or is this optimization feature risky in some way(either unstable, or degrades performance) and so only enabled for whitelisted applications?
If the former, intel is being scummy. If the latter, I'm not so sure. From a theoretical purist standpoint, the idea that graphics drivers would need per-application manual tweaking kind of grosses me out; but, if in fact that is the way the world works, and intel can make the top N most common applications work better through manual tweaking, I'm can't really say that that is a bad thing(assuming all the others aren't suffering for it).
If you're too lazy to RTFA... (Score:5, Informative)
Just look at the pics. Changing the name of the executable changed the results dramatically. The driver is apparently detecting when it's running a 3DMark (or some other specific apps) and switches to some other mode to boost its scores/FPS markings.
Re:Eh? (Score:5, Informative)
Re:Wonder if AMD plays fair? (Score:5, Informative)
xbitlabs was onto them.. (Score:1, Informative)
http://www.xbitlabs.com/articles/mainboards/display/amd785g-intelg45_6.html#sect1 [xbitlabs.com]
quote
The obtained numbers are pretty interesting. The thing is that although AMD 785G solution is ahead of Intel in 3DMark06, it falls behind the competitor in 3DMark Vantage suite. It is especially strange keeping in mind that Radeon HD 4200 is considerably more powerful than GMA X4500HD according to formally calculated theoretical performance. However, the fact is undeniable: Intel G45 chipset does produce higher 3DMark Vantage score in Windows 7. By the way, this is only true for the upcoming operating system, because Intel graphics accelerator can't repeat its success in Windows Vista. And it means that we can conclude that this sudden success demonstrated by Intel G45 can only be explained by certain driver optimizations and not the GPU architecture.
it's only cheating if... (Score:1, Informative)
It's nothing new that integrated/cheap gpus use the cpu for various things. By itself, this is not cheating. it's just a subpar solution. It's only cheating if the drivers are fudging the settings per-application without telling the user. If they fudge for 3dmark and not for other applications, this might mislead the user's intuition about the gpu's performance elsewhere. The per-game profiling offered in the control panels for ati/nvidia are different because they can be switched off and the user is made aware of them.
Re:Good reporting there Ric (Score:5, Informative)
Re:Why not? (Score:3, Informative)
It's not special drivers for specific games. It's regular drivers with exceptions coded in to make them appear faster on "standardised" tests, which are meant to be an all-purpose benchmark to help consumers identify the sort of card they need (and to compare competing cards). This is cheating to increase sales among the early adopter/benchmarker crowd, impress marketing types and get more units on shelves, and is generally at the cost of the consumer.
Re:Wonder if AMD plays fair? (Score:3, Informative)
I don't know all the details of when (in relation to AMD buying out ATi) but...
ATi was notorious for cheating on the IQ benchmarks - essentially using a different anisotropic filtering method for the IQ test (the good one), and then the cheating one during the other tests.
The ridiculous part was that Nvidia was caught doing a similar thing, and the outcry (in part driven by ATi calling out Nvidia) forced Nvidia to include admit it and later driver option to select the optimization level used. When ATi was later caught doing the exact same thing, there was no outcry, there was no admission (despite proof), and there was no option in the drivers to turn off the "optimization".
I don't recall the details of why the particular optimization was considered a "cheat" and others weren't (I believe it killed off IQ to the point of ass, and it was something that would never be used in-game).
This was back around the 6800 (Nvidia) vs x800 (ATi) days.
Re:Wonder if AMD plays fair? (Score:5, Informative)
The question is probably more easily answered if asked, "Who *doesn't* cheat?". http://www.xbitlabs.com/news/video/display/20030526040035.html [xbitlabs.com]
Re:Why a bad hack when you are close to much more? (Score:3, Informative)
I'm just guessing here, but maybe because offloading this work to the CPUs decreases CPU performance substantially, they don't want to make these changes generic because it'd make it look like systems with Intel video are slow, especially in any CPU-oriented benchmarks. After all, they pointed out in the article how Intel does this same thing for the "Crysis" game, but even with this offloading working, the game only got a measly 15fps with all extra effects off, which is downright unusable.
The whole thing just looks really shady.
Re:Wonder if AMD plays fair? (Score:5, Informative)
Re:Doesn't 3DMark cheat too? (Score:3, Informative)
You may be thinking of changing the CPUID on Via chips to GenuineIntel vs AuthenticAMD vs CentaurHauls.
There's one of the 'big' benchmark suites where the chip's score is roughly the same on AuthenticAMD and CentaurHauls, but gets a boost on GenuineIntel. Via's chips are the only ones with (user) changeable cpuid, so we don't know how differently IDed AMD or Intel do, but still interesting.
(First google'd link talking about it.)
http://www.maximumpc.com/article/news/pcmark_memory_benchmark_favors_genuineintel_over_authenticamd [maximumpc.com]
Re:Wonder if AMD plays fair? (Score:5, Informative)
It's really hard to get 15% better performance without doing something underhanded unless your previous drivers were beta quality.
I used to work for a video card manufacturer and game and video developers often did totally retarded things which just happened to work on the cards they developed on but made the software run like crap on ours. We routinely had to implement workarounds for individual games to make them run properly on our cards.
One particular example which springs to mind -- I won't mention the developer or the game -- was an engine which used a feature which we supported in hardware but a certain other card manufacturer whose cards they used performed in software. Rather than configuring said feature once as they should have done, retarded developer repeatedly reconfigured it numerous times in the course of a single video frame, which required us to reconfigure the hardware every time -- slow as heck over an AGP bus -- whereas other card manufacturer just had to execute a few CPU instructions. We had to detect the game and disable our hardware support, so that we would fall back to software and run the retarded code much faster; in that instance there were places in the game where, far from a measly 15%, we'd literally be going from seconds per frame to numerous frames per second.
So it's quite possible to need to detect individual games or applications in order to work around retarded coding which cripples performance on your hardware. The line you shouldn't cross -- and which I don't believe we ever did -- was to render something other than what the developer intended, for example by detecting a shader used by a benchmark and replacing it with one that looked similar but didn't do as much work.
Similarly, the issue here is not Intel punting processing to the CPU when the GPU is overloaded, but the fact that they do so by detecting the name of the benchmark rather than by monitoring the GPU loading and dynamically switching between hardware and software so that it would work on any application. General optimisation is fine, workarounds for retarded developers are fine, but special optimisations for benchmarks which don't affect real applications is getting pretty close to the line.
Re:Wonder if AMD plays fair? (Score:4, Informative)
Just to get things straight bloodhawk said:
Don't remember AMD cheating, But then I have only more recently become a big ATI fan, However Nvidia has a long history of benchmark cheating in drivers in order to make there stuff look better than it is and many times it was far more blatant than what intel is doing here
At the time of quack.exe ATI wasn't owned by AMD, cheating or no cheating we've got to be clear on that one.
Duh (Score:4, Informative)
3DMark Vantage was never a legit benchmark. Heavily tuned for Intel CPU and nVidia GPU architectures it never actually meant a damm thing.
Just compare performance of gf285/295 v. radeon 4870/5870 (any review) in 3DMark and in games. In 3DMark Vantage nVidia cards have close to 50% advantage while in real games radeons sometimes score higher.
The statistical anomaly alone is sufficient to dismiss 3DMark Vantage results as outlier.
Re:Wonder if AMD plays fair? (Score:4, Informative)
You cite a 2001 issue as one of the earliest examples? Poor form. ;) :)
Go back another 4 years. nVidia released the Riva 128 which started winning most benchmarks against ATI's Rage Pro (and Rendition Verite etc). Well, a few publications started noticing that the speed advantage was due to the image quality being much worse with no tri-linear filtering, no fog (at least for a few iterations of drivers) and some sort of compressing the textures that made rendered text on some games illegible (a couple of games had the misfortune of having that problem even with their menu system). I remember the comparison images for the nicest benchmark/demo of the day called "Final Reality" were quite telling of the IQ difference. However, most publications of the time just went with fps numbers, so that left ATI with no choice but to "optimize" their new driver set (called "Turbo") especially for 3D benchmarks
Re:They are all guilty of cheating at some point (Score:3, Informative)
If and only if the Intel driver would make the necessary adjustments for a real world app (as opposed to one that just happens to have the same name as a popular benchmark), THEN it would be acceptable IF they also specified the CPU required to get that performance. Otherwise, it's doing something in the benchmarks that it will not do when you run your application and if your CPU is busy, it may not even run the benchmark as well on your machine.