FutureMark Confirms nVidia's Benchmark Cheating 406
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
This is why artificial benchmarks don't matter (Score:2, Insightful)
Re:This is why artificial benchmarks don't matter (Score:5, Insightful)
Re:This is why artificial benchmarks don't matter (Score:2)
As the report said, the drivers can even detect when you're really playing the game as opposed to running a benchmark, and adjust visual quality appropriately.
Nothing is safe from these... though your original point of diversifying could help.
a mirror (Score:3, Informative)
Worse than that! (Score:5, Informative)
According to the article, that's only half the story. I could almost accept it if they were "optimizing" in the sense that, in certain situations, they slightly reduced image quality for a significant gain. That's kind of sketchy, as the card isn't then doing what it's claiming, but you could argue, perhaps, that the tradeoff is worth it. And if this activity were optional, it might be a benefit.
What they're doing here is different, and much worse. They're actually detecting what program is running - whether it is 3D Mark or not. Effectively, what it does is disobey 3DMark, and only 3DMark, when it issues certain commands that would reduce throughput. That has no purpose but to deceive.
So, not only are these not optimizations in that they don't really improve performance, they're not optimizations in that they don't even take effect when you run a program not called 3DMark.
Quite frankly, I think this could be considered false advertising and nVidia should get in deep shit for this. This is the worst kind of cheating, and quite frankly, this could be what puts nVidia down the Voodoo path. I don't know whether I'll ever buy another of their cards.
Re:Worse than that! (Score:4, Informative)
Re:Worse than that! (Score:4, Informative)
From what it sounds like, nvidia purportedly altered something for the specific purposes of deceiving a benchmark. A benchmark has the sole purpose of benchmarking, so there is absolutely no justification for "optimizations" for a benchmark.
The point is that ATI had a pretty tenuous justification (that they were optimizing for Quake 3 as it's the engine behind a large number of games), but if this is the case then nvidia has none.
Re:Worse than that! (Score:3, Insightful)
Only if they didn't understand what they were doing, which I doubt. Since there aren't many quake-based games that are named quake.exe, and at the time, Quake 3 was an aging game used mainly for benchmarks, and the stunning similarity between the two, you're just searching for a way to justify it.
This case certainl
Re:Of course (Score:3, Insightful)
Re:Worse than that! (Score:3, Funny)
They both suck. Build your own video card. Just make sure none of the parts you use are manuafactured by companies who do anything you disagree with.
Re:This is why artificial benchmarks don't matter (Score:5, Insightful)
But the nVidia driver also substituted a shader for one of the water effects, which degraded/modified the image quality.
And past history has shown that companies are willing to sacrifice quality for performance (see ATI's Radeon 8500 drivers and Quake 3 [tech-report.com] for an example)...
It's almost like this is a cold war, of sorts, between the testers/benchmarkers and the card manufacturers.
Re:This is why artificial benchmarks don't matter (Score:2)
Re:This is why artificial benchmarks don't matter (Score:3, Interesting)
Re:This is why artificial benchmarks don't matter (Score:5, Interesting)
It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.
Re:This is why artificial benchmarks don't matter (Score:3, Insightful)
In fact, they say in the article that with "applications/games people really use", it is even harder to detect driver cheats.
Re:This is why artificial benchmarks don't matter (Score:4, Insightful)
This is why.. (Score:5, Insightful)
Re:This is why.. (Score:5, Funny)
[Next Page]
reviews that
[Next Page]
don't read like Cat in
[Next Page]
the Hat with ads, you
[Next Page]
should try
[Next Page]
AnandTech [anandtech.com] or ExtremeTech [extremetech.com] or even HardOCP [hardocp.com].
Re:This is why.. (Score:5, Funny)
The linked
[Next Page]
article was at
[Next Page]
ExtremeTech, and it
[Next Page]
still ran to ten or so pages, most having two or fewer paragraphs. Maybe that's why the site wasn't Slashdotted--nobody had the patience to click through the whole article.
Re:This is why.. (Score:5, Informative)
Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
A reviewer's job is what? (Score:4, Insightful)
Isn't that the definition of a good reviewer? Fans of the current top of the line stuff - damn their history?
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
Case in point...
Cheaters! (Score:5, Funny)
WHAT?? My FX 5800 Leaf Blower only has a range of five feet and not six? I want a refund!
Re:Cheaters! (Score:5, Funny)
Re:Cheaters! (Score:3, Funny)
Ah well, if I'd gone with an SiS onboard I'd look like Justin from American Idol. *cringe*
Re:Cheaters! (Score:3, Funny)
Re:Cheaters! (Score:3, Informative)
lies and statistics. (Score:4, Interesting)
I remember SPEC benchmarking ment something, and companies putting special routines to make chips seems faster than they were.
Thats why "Real world testing" is important. While not always the greatest comparison, its much better in most cases.
Re:lies and statistics. (Score:4, Informative)
Re:lies and statistics. (Score:5, Funny)
Re: (Score:3, Insightful)
Re:lies and statistics. (Score:5, Insightful)
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
The other problem (Score:3, Insightful)
Instead, I want testing that approximates the sorts of games that I'll want to buy years from now. Unfortunately those games don't ex
Re:As a non-indy game developer, (Score:4, Interesting)
Would I prefer it that way? (Who doesn't like free goodies??) Heck yes! I'd like to get the latest card and evaluate its robustness (Very important to television...) right away so that I can qualify it for use in our systems only a few weeks after it comes out instead of months.
On that note, I'm also constrained by lack of support for Linux on the latest cards (at times). For example, the 9800 doesn't yet have an accelerated linux driver. Dangit! Now, I love the 9700 pro, but I'd love to have that 256 meg on-card.. It is amazing how quickly you can eat up texture memory when you're doing things the card manufacturers didn't think of (like chroma-keying, video mapping, interlaced frame rendering, blah blah)
Isn't this standard practice? (Score:5, Insightful)
While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!
Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.
-n
Re:Isn't this standard practice? (Score:4, Insightful)
Yes, since ATI was caught as well. (Score:4, Informative)
Of course if the article title was, "Everybody cheats on our benchmark!" then that would do more to undermine their benchmark than anything else. Instead they made the focus of the article the fact that NVidia is cheating.
Re:Isn't this standard practice? (Score:5, Informative)
Nvidia (and ATI before) are guilty of using deceit to attempt to sell more video cards. Thus, they are guilty of fraud.
Re:Isn't this standard practice? (Score:3, Insightful)
No, they are not guilty of fraud. They did not misrepresent their benchmark score; merely to optimize for the benchmark score. Whether or not benchmark scores are representative of general real world performance is not their responsibility.
This is similar to Intel realizing that MHz meant everything to silly consumers, and optimizing their CPUs to achieve the highest MHz rating possib
Re:Isn't this standard practice? (Score:3, Interesting)
Sure. So it may be in violation of, say, OpenGL specifications. I don't know the licensing details, but OpenGL might prohibit NVidia from using its logo or claiming compatibility until that's fixed. That's about as close as you can get to a "legal" remedy.
The market remedy is far simpler. Just don't buy NVidia products if you don't agree with the way th
Re:Isn't this standard practice? (Score:3, Insightful)
Has nVidia (or ATI for that matter) ever claimed that any benchmark was indicitive of real world performance? Sure, they may boast individual benchmark numbers and say that their card is fast, despite having optimized routines for individual benchmarks, but they're really only claiming that their cards are fast, which is a subjective measure, and achieve those numbers on those tests, which they do. The benchmark writers may try
This makes me sick! (Score:5, Funny)
Oh wait, my medication just kicked in. It's just business as usual. I will just go on checking my MSN e-mail, while watching MSNBC, drinking my Coke and eating my McDonalds burger.
Never mind.
Re:This makes me sick! (Score:4, Funny)
Re:This makes me sick! (Score:2)
Re:This makes me sick! (Score:2, Informative)
Here is a short list [amazon.com].
Coke never lies, huh? (Score:3, Informative)
Well, friend. It's time you learn that nothing is sacred. Yes, Virginia, even Coca-Cola lies and squashes people to keep its bottom line intact. Read the sad and infuriating tale of judicial corruption and corporate fraud of Bob Kolody vs. Coca-Cola [guerrillanews.com]. I was outraged for days.
correction (Score:3, Funny)
- The 3D graphics performance war just keeps getting more and more interesting!
WrongDon't don the tinfoil hats prematurely... (Score:4, Informative)
So it's quite likely that NVidia was just anticipating optimizations and not outright "cheating."
Re:Don't don the tinfoil hats prematurely... (Score:5, Informative)
Something else that may shock you: it appears that ATI is doing the same thing, although to a much lesser extent.
Re:Don't don the tinfoil hats prematurely... (Score:5, Insightful)
Try reading the article.
Re:Don't don the tinfoil hats prematurely... (Score:3, Interesting)
With NVIDIA, it was about 25% lower overall, which I don't believe they
Re:Don't don the tinfoil hats prematurely... (Score:5, Informative)
With bugs like this... (Score:2)
Re: (Score:2, Interesting)
Re:Shit... (Score:2)
Re:Shit... (Score:5, Funny)
cheats, not "optimizations" (Score:2, Insightful)
It would be as if a CPU manufacturer substituted its own algorithms stealthily in a CPU performance benchmark and only when running that benchmark.
Sure, you get a higher number, but you aren't measuring what the benchmark designer intended to measure.
nVidia thanks jlouderb (Score:5, Funny)
History repeats itself a thousand times over... (Score:5, Insightful)
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Re:History repeats itself a thousand times over... (Score:2)
An online Starcraft RPG! Free only at [netnexus.com]
In soviet russia, all your us are belong to base!
Karma: Redundant
Re:History repeats itself a thousand times over... (Score:5, Insightful)
Re:History repeats itself a thousand times over... (Score:4, Insightful)
That's completely different from what happened here. They looked at a particular test where the camera travels on a set path and hard-coded it so that things were beautiful on that path. As soon as you hop the camera off the rails, the driver goes to crap.
Gah. Read the article. Or, if you're not up to that task, read the dozen posts before mine which say the same thing.
Doom3 (Score:4, Interesting)
They also have another benchmark here [hardocp.com] where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article
screw you Mrs.Goldstein! (Score:5, Funny)
9th grade, you told me cheaters never make money
well 'pbhtbhtbthbth'
ATI Did The Same... (Score:3, Informative)
Re:ATI Did The Same... (Score:3, Informative)
ATI was trying to make my Quake3 faster. That's good. They screwed up and hampered my image quality. Innocent mistake while trying to make my life better.
nVidia was blatantly cheating by hardcoding viewpoints. That's bad. You can't do that in a real-world driver, so it's blatant and evil.
You can't compare these two incidents. Maybe ATI has done similar things, but they have not been caught at anything as bad as this.
Bryan
Re:ATI Did The Same... (Score:4, Interesting)
Re:ATI Did The Same... (Score:3, Interesting)
ATI cheaping too (Score:3, Insightful)
It not about cheating... but about how much you cheat.
The settlement (Score:2)
NVidia's response was brief and to the point...
I wouldnt have cheated on them (the benchmarks), But they have been sleeping around with ATI for years.. and I cant stand being cheated on with someone who cant even write good drivers...
Wow, amazing. (Score:2)
PDF Mirror (Score:5, Informative)
Re:PDF Mirror (Score:2, Informative)
Sorry.
Performance Difference Due to These Cheats (Score:5, Interesting)
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
Company based benchmark constant (Score:3, Interesting)
Maybe nvidia is 0.80 and ATI is 0.90
so then 100fps on a geFrorce card, is really 80 fps, and it would be 90 on an ATI...
For those of you too lazy to read the article... (Score:5, Informative)
HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.
For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.
So what? (Score:2, Insightful)
eg. Fillrate, Vertex manipulation, Texture rasterizer, Shader technology, Texture sampling techniques, Shadow buffering etc.. etc...
Some cards will be better than other at these tasks, and some games will take advantage of differing ratios of these technologies.
The unreal engine has a reliance on poly-count and texture resolution, and it looks like the doom engine will tend to tax shader, and multitextu
As an ex-NVidia employee (Score:5, Informative)
Anyway, I'm posting this as an AC for obvious reasons.
Re:As an ex-NVidia employee (Score:5, Funny)
That's not a cheat... (Score:3, Insightful)
You have just described an optimization, not a cheat. The point of cheats is that they take advantage of knowledge that's not available to normal processes. If your "cheat" takes no such advantage (e.g. calculating its s
trusty bit torrent (Score:3, Informative)
Who found it? (Score:5, Interesting)
Certainly any negative publicity for NVidia is good for ATI and vice versa.
Re:Who found it? (Score:4, Interesting)
Simply put, if ATI brings it to light, many people would claim it was planted, biased, etc... if Extremetech (or another source not directly attached to ATI) brings it to light, then ATI still gets the benefit of burning Nvidia, but without the negative PR they might generate. I wouldn't be surprised if ATI tipped off the people over at Extremetech...
Re:Who found it? (Score:5, Interesting)
Right, like maybe getting a fix posted? Oh, wait, looks like Hard|OCP is taking credit for that:
Futuremark has released a patch for 3DMark 2003 that eliminates "artificially high scores" for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week. According to the PDF on Futuremark's site, the patch causes a 24.1% drop in score for NVIDIA..."
I'm amazed at the OCP's coverage of this whole deal. They didn't break the story, so they cast doubt on ExtremeTech's findings, and allude to suspicious "motives" that were never proven.
Then, when the fix is released, they claim the fix is released "in response to a news item we posted last week", as if they're directly responsible. A week ago they're bashing ExtremeTech for even insinuating driver cheating, and this week they're taking credit for getting the fix released (as if they broke the story themselves).
Wasted Code (Score:5, Insightful)
ATI possibly cheating as well (Score:3, Interesting)
"Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
Gasp, what a shock. Everyone seems to be guilty of having cheated on synthetic benchmarks at some time. This has happened before, it will happen again.
ATI is tweaking stuff too.. (Score:3, Informative)
From the article:
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.
Good for futuremark (Score:4, Insightful)
NVidia is losing. Their chips and cards are worse than ATI's. What's worse than that, though, is that they are still trying to pretend that it's not the case. They need to seriously sit down and work on their designs but instead they are pissing money away working on cheating on benchmarks. That is a really bad sign for a company. It means managament is diverting money away from becoming successful twords appearing to be successful. A mentality like that is disasterous to the real value of a company.
SELL! SELL NOW! Buy again when they have fixed their mangement and design issues.
Contravertial != Overrated. Reply if you disagree, I'll read it.
Re:Good for futuremark (Score:4, Interesting)
Since when? Jen-Hsun Huang admits defeat [bayarea.com] (But promises a comeback):
"Tiger Woods doesn't win every day. We don't deny that ATI has a wonderful product and it took the performance lead from us. But if they think they're going to hold onto it, they're smoking something hallucinogenic."
Quack (Score:4, Informative)
game or demo? (Score:3, Interesting)
if i remember correctly some of the people who funded futuremark had something to do with a demo named "second reality". a good old school demo on 2 discs.
if 3dmark was TRULLY a bench it would then resort on code that we find in games!! opts are expected for thoses...even more for stuff...
what if you told carmack that the opt he made for quake and tweaked openGL implementation are just cheats? Sure you remember 3dfx ogl implementation and riva128 drivers...
what if you told ppl from the 'scene that their demo sucks because they don't properly handle Z buffering.
They all rely on tricks.(beter than opts or cheating from a coder point of view), even processors rely on thoses. they're based on user experience, not bogomips or whatever. page-flipping was a inproper behavior at a time when VESA was not VESA but scene called mode-X, eventually it became best practice. Sprites asm hard-coding was the same and most 2d shooters are based on that.
I'm pretty sure ppl at futuremark include some kind of sleazzy code in their bench as coders always do.
the only difference b/w cheating and proper optimization is only PR. if nvidia told us "wow! we made an optimization that runs 3dmark faster" as it would with a game none would complain.
it's just that for a lot of us 3dmark is supposedly an untouchable thing. It's not. it should reflect real world 3d. and in real life you expect those kind of code workaround.
then i ask myself a question... why doesn't futuremark distribute freely a playable bench.
why put us in front of a demo claiming it's a synthetic bench and then why aren't we believing it?
because it'a a lie. either they're real world gaming and tricks are OK, either they're pure demos and tricks are not options.
thanks, but... (Score:4, Insightful)
Please optimize your drivers and hardware for the actual applications and games I run, not the synthetic benchmarks designed to simulate workloads. Benchmarks don't use your products, end-users do.
NVidia's counterclaims (Score:4, Informative)
From Bluesnews [bluesnews.com] (from an unlinked CNet article):
"Recently, there have been questions and some confusion regarding 3DMark 03 results obtained with certain Nvidia" products, Futuremark said in the statement. "We have now established that Nvidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark 03."
A representative at Nvidia questioned the validity of Futuremark's conclusions. "Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."
Playing the devil's advocate... (Score:3, Interesting)
Some of the cheats potentially reduce image quality, but we are talking about OpenGL and DirectX here - nobody really aims for 100% visual quality, and indeed there is no target to shoot for since neither standard specifies "correct" rendering down to the pixel level.
You might complain that 3DMark is being treated specially, that other software wouldn't receive the same speedups. That is true. But application-specific optimization has a long history. Just look at Windows - the more recent versions detect and flag certain programs that are known to break or run slowly due to compatibility issues. Nobody says Windows is "cheating" because it refuses to install a driver that its internal knowledge base knows will trash your system. In the CAD world, video card makers almost always tweak drivers to support specific CAD and 3D applications. (3DLabs' control panel used to have a box where you could select "optimize for AutoCAD/3D Studio/Maya/etc...")
ATI should be happy that NVIDIA engineers are wasting time fixing specific benchmarks when they could instead be improving performance in general. But I wouldn't read much more than that into this.
Making your buying decision based on a synthetic benchmark, rather than in-context with your intended application, is always going to distort the picture. (Looking at SPEC benchmarks, Itanium blows the competition away - just tell that to the millions of people who are *not* buying IA64 chips!)
If you, the OpenGL developer, end up writing the next wildly-successful game, I'm sure NVIDIA will be happy to tweak their drivers for it.
Re:/.'ed already??? (Score:5, Informative)
Be nice and download the zip [mskf.org] or the bzip2'd [mskf.org] version instead, if you're able.
Would it really kill Taco if... (Score:3, Insightful)
I've said it before, and I'll say it again: doing this would be a win-win situation. It's a pity that the editorial team are too busy playing with MAME/whatever to actually do something of real benefit to the wider community.
Re:Would it really kill Taco if... (Score:3, Informative)
Re:What is wrong with us? (Score:3, Informative)
Re:well.. (Score:3, Insightful)
Re:They didn't cheat! (Score:5, Informative)
Further, the problems change depending on which part of the demo you're in (for instance, the "background not being cleared" bug conveniently only shows up in the part of the space demo where a largely black sky is being displayed, and so no background clear is necessary). This is cheating, plain and simple.
Re:Stop your FUCKING whining, Slashdot! (Score:4, Insightful)
It was Microsoft."
Right. All manufacturers... whose hardware works with windows. I'll take cross platform compatability thank you very much.
Before you might argue that nobody uses OpenGL, what about all those licensees of the Quake 3 engine? And what about all those who will license the Doom 3 engine?