Slashdot Log In
FutureMark Confirms nVidia's Benchmark Cheating
Posted by
michael
on Fri May 23, 2003 12:30 PM
from the red-handed dept.
from the red-handed dept.
jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
This is why.. (Score:5, Insightful)
Re:This is why.. (Score:5, Funny)
[Next Page]
reviews that
[Next Page]
don't read like Cat in
[Next Page]
the Hat with ads, you
[Next Page]
should try
[Next Page]
AnandTech [anandtech.com] or ExtremeTech [extremetech.com] or even HardOCP [hardocp.com].
Parent
Re:This is why.. (Score:5, Funny)
The linked
[Next Page]
article was at
[Next Page]
ExtremeTech, and it
[Next Page]
still ran to ten or so pages, most having two or fewer paragraphs. Maybe that's why the site wasn't Slashdotted--nobody had the patience to click through the whole article.
Parent
Re:This is why.. (Score:5, Informative)
Funny, I seem to remember Toms Hardware being rabidly AMD fanboyish about 1.5 years ago when AMD still had the fastest processor. I'm not saying they aren't biased fanboys, what I'm saying is they're fairweather fans.
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
Parent
A reviewer's job is what? (Score:4, Insightful)
Isn't that the definition of a good reviewer? Fans of the current top of the line stuff - damn their history?
To keep it on-topic, I also seem to remember ATI doing the exact same thing nVidia is now doing with quake "optimization" for the 8500 cards... Do a google search for "quake quack"
Case in point...
Parent
Cheaters! (Score:5, Funny)
WHAT?? My FX 5800 Leaf Blower only has a range of five feet and not six? I want a refund!
Re:Cheaters! (Score:5, Funny)
Parent
lies and statistics. (Score:4, Interesting)
I remember SPEC benchmarking ment something, and companies putting special routines to make chips seems faster than they were.
Thats why "Real world testing" is important. While not always the greatest comparison, its much better in most cases.
Re:lies and statistics. (Score:4, Informative)
Parent
Re:lies and statistics. (Score:5, Funny)
Parent
Re:lies and statistics. (Score:5, Insightful)
I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.
Get over it people.
Parent
Isn't this standard practice? (Score:5, Insightful)
While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!
Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.
-n
Re:Isn't this standard practice? (Score:4, Insightful)
Parent
Yes, since ATI was caught as well. (Score:4, Informative)
Of course if the article title was, "Everybody cheats on our benchmark!" then that would do more to undermine their benchmark than anything else. Instead they made the focus of the article the fact that NVidia is cheating.
Parent
Re:Isn't this standard practice? (Score:5, Informative)
Nvidia (and ATI before) are guilty of using deceit to attempt to sell more video cards. Thus, they are guilty of fraud.
Parent
This makes me sick! (Score:5, Funny)
Oh wait, my medication just kicked in. It's just business as usual. I will just go on checking my MSN e-mail, while watching MSNBC, drinking my Coke and eating my McDonalds burger.
Never mind.
Re:This makes me sick! (Score:4, Funny)
Parent
Don't don the tinfoil hats prematurely... (Score:4, Informative)
So it's quite likely that NVidia was just anticipating optimizations and not outright "cheating."
Re:Don't don the tinfoil hats prematurely... (Score:5, Informative)
Something else that may shock you: it appears that ATI is doing the same thing, although to a much lesser extent.
Parent
Re:Don't don the tinfoil hats prematurely... (Score:5, Insightful)
Try reading the article.
Parent
Re:Don't don the tinfoil hats prematurely... (Score:5, Informative)
Parent
nVidia thanks jlouderb (Score:5, Funny)
History repeats itself a thousand times over... (Score:5, Insightful)
I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.
I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".
Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.
- Jeff
Re:History repeats itself a thousand times over... (Score:5, Insightful)
Parent
Re:History repeats itself a thousand times over... (Score:4, Insightful)
That's completely different from what happened here. They looked at a particular test where the camera travels on a set path and hard-coded it so that things were beautiful on that path. As soon as you hop the camera off the rails, the driver goes to crap.
Gah. Read the article. Or, if you're not up to that task, read the dozen posts before mine which say the same thing.
Parent
Doom3 (Score:4, Interesting)
They also have another benchmark here [hardocp.com] where they compare the 5900 ultra and the radeon 9800 pro. In that article it says that NVIDIA told them not to use 3DMark03 I recommend reading that article
screw you Mrs.Goldstein! (Score:5, Funny)
9th grade, you told me cheaters never make money
well 'pbhtbhtbthbth'
PDF Mirror (Score:5, Informative)
Performance Difference Due to These Cheats (Score:5, Interesting)
3DMark03 build 320.
The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in
that build gets 4679 3DMarks - a 24.1% drop.
Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.
For those of you too lazy to read the article... (Score:5, Informative)
HOWEVER, in the development version of 3dmark 2k3, you can take the camera "offroading". When you do that, it becomes apparent that things are being drawn incorrectly -- that there are hard-coded limits that result in the video card doing less work than the program requests.
For those of you whining about how they should use "real life" games for benchmarks, this technique could be applied to anything where the camera path is predetermined. It has nothing to do with 3dmark 2k3 specifically.
As an ex-NVidia employee (Score:5, Informative)
Anyway, I'm posting this as an AC for obvious reasons.
Re:As an ex-NVidia employee (Score:5, Funny)
Parent
Who found it? (Score:5, Interesting)
Certainly any negative publicity for NVidia is good for ATI and vice versa.
Re:Who found it? (Score:4, Interesting)
Simply put, if ATI brings it to light, many people would claim it was planted, biased, etc... if Extremetech (or another source not directly attached to ATI) brings it to light, then ATI still gets the benefit of burning Nvidia, but without the negative PR they might generate. I wouldn't be surprised if ATI tipped off the people over at Extremetech...
Parent
Re:Who found it? (Score:5, Interesting)
Right, like maybe getting a fix posted? Oh, wait, looks like Hard|OCP is taking credit for that:
Futuremark has released a patch for 3DMark 2003 that eliminates "artificially high scores" for people using NVIDIA Detonator FX drivers. This is in response to the news item we posted last week. According to the PDF on Futuremark's site, the patch causes a 24.1% drop in score for NVIDIA..."
I'm amazed at the OCP's coverage of this whole deal. They didn't break the story, so they cast doubt on ExtremeTech's findings, and allude to suspicious "motives" that were never proven.
Then, when the fix is released, they claim the fix is released "in response to a news item we posted last week", as if they're directly responsible. A week ago they're bashing ExtremeTech for even insinuating driver cheating, and this week they're taking credit for getting the fix released (as if they broke the story themselves).
Parent
Wasted Code (Score:5, Insightful)
Good for futuremark (Score:4, Insightful)
NVidia is losing. Their chips and cards are worse than ATI's. What's worse than that, though, is that they are still trying to pretend that it's not the case. They need to seriously sit down and work on their designs but instead they are pissing money away working on cheating on benchmarks. That is a really bad sign for a company. It means managament is diverting money away from becoming successful twords appearing to be successful. A mentality like that is disasterous to the real value of a company.
SELL! SELL NOW! Buy again when they have fixed their mangement and design issues.
Contravertial != Overrated. Reply if you disagree, I'll read it.
Re:Good for futuremark (Score:4, Interesting)
Since when? Jen-Hsun Huang admits defeat [bayarea.com] (But promises a comeback):
"Tiger Woods doesn't win every day. We don't deny that ATI has a wonderful product and it took the performance lead from us. But if they think they're going to hold onto it, they're smoking something hallucinogenic."
Parent
Quack (Score:4, Informative)
Re:This is why artificial benchmarks don't matter (Score:5, Insightful)
Parent
Worse than that! (Score:5, Informative)
According to the article, that's only half the story. I could almost accept it if they were "optimizing" in the sense that, in certain situations, they slightly reduced image quality for a significant gain. That's kind of sketchy, as the card isn't then doing what it's claiming, but you could argue, perhaps, that the tradeoff is worth it. And if this activity were optional, it might be a benefit.
What they're doing here is different, and much worse. They're actually detecting what program is running - whether it is 3D Mark or not. Effectively, what it does is disobey 3DMark, and only 3DMark, when it issues certain commands that would reduce throughput. That has no purpose but to deceive.
So, not only are these not optimizations in that they don't really improve performance, they're not optimizations in that they don't even take effect when you run a program not called 3DMark.
Quite frankly, I think this could be considered false advertising and nVidia should get in deep shit for this. This is the worst kind of cheating, and quite frankly, this could be what puts nVidia down the Voodoo path. I don't know whether I'll ever buy another of their cards.
Parent
Re:Worse than that! (Score:4, Informative)
Parent
Re:This is why artificial benchmarks don't matter (Score:5, Insightful)
But the nVidia driver also substituted a shader for one of the water effects, which degraded/modified the image quality.
And past history has shown that companies are willing to sacrifice quality for performance (see ATI's Radeon 8500 drivers and Quake 3 [tech-report.com] for an example)...
It's almost like this is a cold war, of sorts, between the testers/benchmarkers and the card manufacturers.
Parent
Re:This is why artificial benchmarks don't matter (Score:5, Interesting)
It is by Nvidia's negligence that the optimisations were found. That's why (among other things) the beta program exists with those features. I think we can probably expect this and other cheat hampering features in future versions.
Parent
Re:This is why artificial benchmarks don't matter (Score:4, Insightful)
Parent
Re:Shit... (Score:5, Funny)
Parent
Re:/.'ed already??? (Score:5, Informative)
Be nice and download the zip [mskf.org] or the bzip2'd [mskf.org] version instead, if you're able.
Parent
Re:They didn't cheat! (Score:5, Informative)
Further, the problems change depending on which part of the demo you're in (for instance, the "background not being cleared" bug conveniently only shows up in the part of the space demo where a largely black sky is being displayed, and so no background clear is necessary). This is cheating, plain and simple.
Parent
Re:ATI Did The Same... (Score:4, Interesting)
Parent
Re:Stop your FUCKING whining, Slashdot! (Score:4, Insightful)
It was Microsoft."
Right. All manufacturers... whose hardware works with windows. I'll take cross platform compatability thank you very much.
Before you might argue that nobody uses OpenGL, what about all those licensees of the Quake 3 engine? And what about all those who will license the Doom 3 engine?
Parent