NVIDIA's Drivers Caused 28.8% Of Vista Crashes In 2007

PaisteUser tips us to an Ars Technica report discussing how 28.8% of Vista's crashes over a period in 2007 were due to faulty NVIDIA drivers. The information comes out of the 158 pages of Microsoft emails that were handed over at the request of a judge in the Vista-capable lawsuit. NVIDIA has already faced a class-action lawsuit over the drivers. From Ars Technica: "NVIDIA had significant problems when it came time to transition its shiny, new G80 architecture from Windows XP to Windows Vista. The company's first G80-compatible Vista driver ended up being delayed from December to the end of January, and even then was available only as a beta download. In this case, full compatibility and stability did not come quickly, and the Internet is scattered with reports detailing graphics driver issues when using G80 processors for the entirely of 2007. There was always a question, however, of whether or not the problems were really that bad, or if reporting bias was painting a more negative picture of the current situation than what was actually occurring."

Time to open up those drivers NVIDIA

Just sayin

Re:Time to open up those drivers NVIDIA

Let's see... 1,000,000 knowledgeable geeks vs a couple dozen at nVidia... Yeah, I'd say we could.

So, you are saying 1.000.000 knowledgeable geeks would be working to fix the driver ? Talk about wishful thinking.

I say 10, at most.

Awesome!

I really hope there's some way I can use those same drivers under linux!

Oh....wait.

Re:Awesome!

Actually, what TFA doesn't say is that the Vista driver instability was done intentionally because they were sick and tired of listening to us Linux users complain. I guess they figured it would be easier to level the playing field rather than to fix the bugs.

Hey, at least we got through to them.

Re:WARNING - Shock site

Parent links to shock site - do not click. This [youtube.com] is much more amusing, if you want to click on something ;)

It's a good thing I read xkcd [xkcd.com] Or I'd have never known what "being rickrolled" meant........

The ow starts now

This is descending below lawsuit territory, I'm starting to think that the whole PC hardware industry should be taken out back and shot. They supported MS in the release of an OS with crap under-powered hardware with smiles and big adverts, in full knowledge that these systems would never work or just were not ready for Vista.

"The Wow Starts, oh around 2009 if you'll just let us fix this, upgrade that and force you to buy some new stuff" Should have been the tagline for Vista.

I'm relieved

28.8% of Vista's crashes over a period in 2007 were due to faulty NVIDIA drivers

Well then it's a good thing their driver support is so crappy with Linux!

Oh wait...

More seriously, I rag on Nvidea for poor Linux support, and this is more of a chance to bash them, but their drivers work fine under XP. If Microsoft provided better documentation of their APIs, as the EU has been demanding, perhaps writing drivers wouldn't be such a pain in the ass?

I also wonder why closed source vendors don't open their code. They don't have to release it under the GPL, they can reatain all their copyrights, just publish the source. How could it hurt them? They retain copyrights and presumably patents so it's not like anyone could copy them.

Is closed source closed so that nobody will realise just how abysmally shitty their kludges are?

If your OS crashes, your OS is crap. Microsoft, fix your OS and publish the code. Nvidea, fix your shitty drivers and open the code. Don't give up any rights, just open it.

I'd like to see copyright law changed so that executables can't be copyrighted unless the source is also provided. How can IBM tell what parts of their code they stole from SCO? Of course the answer was "none". Time to reboot copyright law!

-mcgrew

Re:I'm relieved

I also wonder why closed source vendors don't open their code. They don't have to release it under the GPL, they can reatain all their copyrights, just publish the source. How could it hurt them? They retain copyrights and presumably patents so it's not like anyone could copy them.

Only the companies know for sure why they keep it closed-source, but explanations that have been suggested at various times include:
1. The drivers contain code licensed from third-parties, such that opening the source would require extensive contracts, negotiations, and more licensing. Probably most of these third-party software vendors won't agree to have their code opened for the same reasons that all closed-source companies keep their source closed.
2. Modern video cards (and other hardware too, probably) contain a surprising amount of their logic and "acceleration magic" in the driver. The card itself, though dedicated to a particular hardware task, is quite general and thus the code controlling the card contains many of the important 'tricks' to get good performance. (In fact I've been told that the difference between some cards and higher models is only in the driver.) In such cases, releasing the software code would be like releasing the hardware circuit diagram: it would reveal many of their trade secrets (some of which may be patent-protected, others not).
3. Even if it would be illegal, some people would modify and redistribute the code. Hobbyist hackers would alter the code and recompile. This might allow end-users to bypass restrictions on the card, enable other features (effectively upgrade the card by bypassing lockouts), and so on. This makes lock-in harder, and might reduce the frequency that people upgrade their hardware.
4. Their code, in all likelihood, violates a large number of competitor patents. As long as the violations are buried inside a binary, no one will notice. Opening the code would make it easy for a competitor to spot violations and sue. Probably all the companies violate each other's hardware and software patents, but they maintain an uneasy balance by all being secretive. If one company released too much information, the others would use it against them.
5. The company may worry about other liabilities that they become exposed to when users and competitors can peruse the codebase.

As I said, only the companies know for sure. But there are plenty of plausible reasons for why a hardware company wouldn't want to release driver source code. They are not great reasons (many of us would be more willing to buy the hardware if it had more documentation and/or open code), but they make business sense.

Certified

Did MS certify they drivers? If so, it's still _their_ fault

Re:Certified

Indeed. And its why you had to see internal mails to know that MS were saying it was Nvidia's fault. Considering anytime Windows crash, MS gets the blamed (even if a significant amount of times its not Windows' fault directly...Creative, I'm looking at you), if they felt it REALLY wasn't their fault, they would have said it reeeeally quick.

If they didn't, its partly because they took the blame, as they should.

I'll vouch for this

We're a dell hardware shop. We buy on a 4 year cycle, every machine gets replaced every 4 years with the latest latitude line shipping model of laptop. In this past few cycles they've been NVidia based. They all have 2 gigs of ram, sata hard drives, dual core higher end processors and of course, NVidia Mobile chipsets. So, all 800 people at my company with nvidia chipsets cannot deploy vista until a) the drivers are fixed. b) the hardware cycle comes up in 4 years. All the people getting new machines right now are perfectly happy because the hardware is supported, but just those purchased 6 months ago and before (D820's) are not capable of running vista with dual monitors without gambling on whether or not they will be alive after a weekend on screensaver.

No such problems for me

This is why i use my computer without graphics card. Nor screen. I am guided purely by instinct.

Linux users unite!

Since apparently you can have a class action lawsuit for drivers not working, lets open up the floodgates and punish the manufacturers for not having compatible software! And why stop at video drivers? Lets sue all the makers of legacy hardware. And wifi hardware. Have an OLD 5 1/4 floppy? Sue! Have one of those old HP video-now PCMCIA cards? Sue!

Sue sue sue!!!

Well...

NVIDIA vs ATI drivers - I don't really care.
"It worked for me" - I don't really care.
Statistics on the cause of crashes - I don't really care.
Anybody running unsigned drivers and experiencing crashes - I don't really care

Hang on. Let me explain.

The fact that you can STILL crash a Windows machine with a dodgy driver - that I care about. I thought everything was supposed to be userspace. I thought the error-handling was supposed to be better. I thought that Windows was supposed to be more stable and secure. I thought people who were using signed drivers were supposed to be "approved" and relatively crash-free.

Unsigned drivers? You can't support that no matter who you are, unless you're confident they are PURE userspace - they could be doing anything (like the 3DFX drivers that used to open access to all sorts of things it shouldn't in order for a primitive user-space part to actual drive the hardware). That's why you have to click that "CONTINUE Anyway" button with the dire warning. That's the Windows equivalent of kernel tainting. Once you've done that, nobody cares. The fact that most XP drivers are still using uncertified drivers is a bit of a problem but I can understand the reasons why. But you can't blame MS for crashes in uncertified drivers under XP. I thought Vista was supposed to be different, though.

If a certified driver is crashing that often, then you have an entirely different matter. The certification effectively becomes worthless. Nobody trusts it. Therefore every driver manufacturer ignores certification and just tells users to click "Continue". Then you will have nothing BUT uncertified drivers. Catch-22.

Blue screens should not happen. They certainly shouldn't happen often enough that people have coined the term "blue-screen" or BSOD to mean a crash. When they DO happen, when the driver goes absolutely nuts and starts stomping memory, aren't things like DEP and the user-space driver model supposed to STOP that happening and recover in some half-decent fashion? Or shouldn't the machine at least what the cause was and provide the user with some hint of what went wrong (i.e. "You installed an uncertified driver. Tough.").

Let's compare for a second - Linux kernels crash too. They crash much more often if third-party drivers are installed and nobody really cares about that except the third-party and their users. When they do crash, there's not much you can do but most of the time you'll get all sorts of debugging information and usually you can carry on. You might lose X, which may or may not load up again - I have a laptop that likes to crash X if I run more than one copy of Xine at a time but the worst that happens is X dies and restarts and then carries on working for hours/days/weeks as if nothing had happened (and yes, I need to update the kernel/X on that machine!) but things keep on working as best they can. You can do pretty much what you like in terms of software but the worst that'll happen if you're not actually loading a kernel module or patching a kernel or playing with kernel-level features is a software crash and be chucked back to the command-line. Sometimes you might even end up taking out X, like my example above.

You can rip out the harddrive and *make* the kernel crash but most of the time things will carry on, just without the component you ripped out (i.e. the IDE layer may die, but it'll still keep running as best it can without it). Even when Linux comes to a complete halt and freezes, you have debugging information and logs with which to narrow down the cause yourself, without needing to consult Linus himself.

When Windows crashes (even with certified drivers and clean installs), there's bugger all to go on. Half the time the event log doesn't show anything at all. The second you see a blue screen, the computer is down and there's little arguing. There's zero information to go on. You have no idea what caused the crash at all because usually all you get is a generic STOP error and a list of hexadecimal. Ordinary software can cause a crash, or make the computer unusable. Nothing EVER recovers. I've never seen a Windows machine even try to recover from an error like that. It's either running or it's dead. And when it's dead, safe mode, last known good, etc. are a waste of time usually but work very occasionally. If you're mouse driver crashes, that's it. Game over. Everything stops. When, of course, only the mouse should stop.

Two months ago I had a Server 2003 machine that was running a small network stop for absolutely no reason. The STOP error said there was a problem with ARCSTOR.SYS. That was it. No logs. No errors. The file didn't even exist on the system or its daily backups, or it's clients, or in any of the files mentioned in the weblogs for the network. No changes had been made to the machine. I checked everything I could find - not a single clue as to the cause. But the server just stopped serving clients and sat executing NOP's until I came in and physically rebooted it. I still have no idea why it did it (I suspect storage devices but they were all clean and the RAID never even tried to rebuild, the tape drive wasn't scheduled to do another backup for hours and it had finished its last one, with verify and eject, hours before the crash) and the server is still running now. I don't even know WHEN it crashed, except to guess at when the last log entry was made but I have no idea if that was the last time it COULD have written to the logs. So I have a eight-nine hour window when it could have crashed, at best guess (between the last entry and it being reported not working). How is that any help to me?

And if it does it again, I have nothing to go on. Nothing at all. MS could probably analyse the dumps and do all sorts of fancy stuff but I need to know WHY it did it and nothing on the machine will help me.

Lots of crashes are the fault of the actual hardware (Bad RAM, overheating etc.). Lots of crashes are the cause of an error in the kernel. Lots of crashes are the cause of third-party kernel-level software. But in-kernel crashes caused by shipped or "approved" drivers shouldn't really be there at all and should be fixed incredibly quickly. And in-kernel crashes caused by non-kernel software should be eliminated as a matter of priority, without excuses.

And to the people commenting - Never have "reboot on error" switched on in Windows. It's a waste of time. If you need the PC to reboot on a crash, use a real hardware watchdog timer. All the Windows one does is stop you reading error messages and getting into your computer. And "Last Known Good Configuration" is a waste of space that I only ever witness actually do ANYTHING about 1 time in 20. In all those cases, the system is then so fragile and finnicky that you're better off restoring from backup.

The other 83.7% ....

... were caused by Vista internals.

These statistics were calculated using Excel.

Huh?

ATI was 9%

Re:Huh?

I know you say that in jest, but the article states that ATI have 9.3% of the problems. It stands to reason that it is representative of their market share.

The part that seems to have been missed is the fact that Microsoft had 17.9% of the crashes related to their own drivers. IMO this is much more significant and interesting than Nvidia beta drivers crashing and should be the real news here.

Re:Huh?

I know you say that in jest, but the article states that ATI have 9.3% of the problems. It stands to reason that it is representative of their market share.

This was a little surprising to me as well, but ATI had about 20% of the market during 2007.

GPU Market Share
=================
Intel 37.6%
Nvidia 32.6%
AMD 19.5%

Source: http://www.news.com/8301-13579_3-9752280-37.html [news.com]

It would seem that AMD has managed to turn around their driver's stability and it is better than nVidia's, who apparently has a pretty poor record at the moment.

Re:Huh?

In my (somewhat limited) experience, the best drivers are those written by a third party. The more complex the hardware, the bigger the hardware and driver teams get. When you have a really complex bit of hardware, like a GPU, you have a huge team of hardware designers (who don't really understand software) and a huge team of driver developers (who don't really understand software). If they are both in house then you generally have pretty poor documentation because both teams have access to the other's work, but not the expertise to understand it fully. The hardware guys all think that the software team can get most of what they need from the HDL, and just fill in the gaps with their documentation.

When a third party is writing the drivers, you don't want them to have access to anything proprietary and so the interfaces need to be very thoroughly documented because the external team isn't allowed to have access to the implementation details at all. A lot of the early XFree86 accelerated drivers were developed in this way and, at the time, were a lot more stable than their Windows counterparts, as were the early Radeon drivers written by the open source community.

Re:Not surprised

The linux drivers for nvidia suck too, nvidia clearly take a long time to get up to speed on new operating systems, it's one reason I no longer use them. Having said that, they're pretty damn solid, so its most likely becuase vistas so mucked up when it comes to drivers.

Well, from my experience (not trolling), but they historically have sucked somewhat less than the ATI drivers, which have been known to cause freezes when switching to a console, etc., due to bugs in the driver, firmware, AMD processors (ironically enough), various chipsets and all sorts of things.

The problem is that in the race to produce the biggest, baddest, fastest, video cards for gamers, ATI/AMD and NVIDIA have often overlooked stability for performance. I don't know about you, but I'd gladly trade off a couple of FPS for a card that was rock solid stable.

Re:Not surprised

Someone has already pointed out that if you want a rock-solid stable video card under Linux, buy a board with an Intel G965 or G33/35 chipset, so I won't make that argument (although I will say the drivers aren't completely rock solid and lack many of the options I'm used to with the nVidia driver, like OGL vsync to stop "tearing" when I play full screen video).

However, I will say that ATI's Linux drivers have come on leaps and bounds since AMD took the helm. They're still sucky, but they now only about twice as sucky as nVidia, as opposed to the binary equivalent of disemboweling yourself with a grapefruit spoon. The fact that, thanks to AMD publishing the specs for the silicon, a fully OSS, clean room, accelerated driver is now possible is also a colossal boon, and I suspect that within a few months the RadeonHD driver will be featureful and stable enough to be more than adequate for most people, once the distros start picking up on it.

Then, of course, it'd be nice if someone could write a way of accelerating video so that all us Linux users without eleventy billion jiggahurtz processors could play back 1080p H.264...

Re:Minimal problems with 169 series

So let me get this straight: When X Crashes you lose your current session, right? Which means that OOo document you were working on just went "poof" - your media player shuts down, along with all your other apps that launch within the context of the X session.

Now, your uber OS may have stayed "on" in that it could reload all that crap without having to spend 20 seconds rebooting, but for all intents and purposes from a user perspective, your whole OS just freaking crashed.

Re:O RLY?

Well, this wouldn't be the first time Nvidia drivers are responsible for instability.

At 28.8%, nVidia still has a long way to go to reach the epitome of device driver excellence that is ATI's collection of video drivers. Those extrusions of fecal material have accounted for more cases of alopecia on users than most other kinds of software. I'm actually surprised that the submitter didn't take a swipe at ATI while writing about driver crashes; the urge to do that must've been immense. In fact, ATI driver problems where the single biggest contributor to Jerry Pournelle's best writing ever in Byte Magazine's Chaos Manor column.

Re:What is the standard procedure?

Well if NVidia is the only one with MAJOR driver problems....lets look at the math. 80% of the drivers work and they were built with the DDK while 20% (including NVidia's drivers) do not work and they were built with the DDK. I would think the 20% did not write their drivers correctly.

Re:Sounds a lot like finger pointing to me

Here is a link:

http://technet2.microsoft.com/windowsserver/en/library/eb1936c0-e19c-4a17-a1a8-39292e4929a41033.mspx?mfr=true [microsoft.com]

Depending on what version of "blame Microsoft" you are responding to the complaint may or may not be legitimate.

Windows NT 3.51 may have been the most stable version of Windows in history. I think it was the one on which Microsoft spent the most time and money on testing and on a fairly massive scale went out and helped hardware and driver people with their testing (providing labs with a large variety of configurations etc.). They were trying to solidify the Windows base within businesses, and convince businesses that Windows was no longer a toy (i.e. gaming) operating system only. The goal, among other things was to get people off of OS/2, older versions of Windows (93 and WFW).

The program was a great success. Not only did large parts of the federal government switch, I even made the switch on my home machines. Unless you were a gamer (in which case you would have still been running 95 or then 98) you could have experienced a relatively unbloated and crash-free Windows experience. It was the lat time I tried running Windows for days on end without regular restorative reboots.

As the link states:

"In Windows NT 4.0, drivers were moved into kernel mode to improve performance. However, when a kernel-mode driver fails, it can crash an entire system, whereas the failure of a user-mode driver causes only the current process to crash."

In point of fact, video drivers could "fail" prior to 4.0 and only cause minor screen corruption or glitches, or in fact be asymptomatic. After 4.0 though, the same failure might cause a system crash, or might cause other programs to appear to crash, or might cause disk I/O buffers to contain garbage that would subsequently be written out to disk and cause crashes hours later, not to mention you wondering why your spreadsheets were deteriorating over time.

I don't remember Microsoft going out and asking video vendors if they thought this was all a good idea. In fact the element of surprise was very important to MS for some reason on the 4.0 announcement... no pre-announcement of features being added or removed as there were for years leading up to Vista. They certainly didn't ask me. I left the meeting telling my colleagues taht this was nuts. And I don't think they gave either vendors or users much time to adjust to the changes as I went from thinking that Windows had finally arrived to wishing I had stayed with OS/2.

From what I read, MS no longer does the extensive testing they did for 3.51, and in fact they make driver and hardware makers pay them for any help they get in order to be "certified". Having won the game of becoming THE business operating system, MS said "screw you" to the partners that helped them get there. Typical.

MS engineers bragged about being geniuses during the 4.0 product roll-out for moving drivers to kernel space, but the move was necessary due to GUI bloat that was added for that release. Subsequent bloat of that nature has made each subsequent version of Windows seem less snappy and take up more memory, and no doubt the next product roll-out after 4.0 (at which point I had stopped attending) I'm sure the MS engineers bragged about being geniuses for moving drivers back into user-mode for reliability reasons. Both moves might have cause significant adjustments to be made by driver makers on short notice depending, for example, on whether they were relying on memory protection and changing the nature of their context switches.

If you don't blame Microsoft for some of these driver problems you either work there, or haven't been paying attention for long enough.