Mark Russinovich On Vista Network Slowdown

koro666 writes "In his latest blog post, Mark Russinovich analyzes the network slowdown experienced by some users when playing multimedia content. 'Tests of MMCSS during Vista development showed that... heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver... [to] pass along, at most 10 packets per millisecond (10,000 packets per second)... [T]he networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.'"

Okay...

[WhatAmIDoingHere]

So why can my Windows 98/95/2000/ME/XP computers play mp3s without this happening?

Re:Okay...

[KiloByte]

So why can my Windows 98/95/2000/ME/XP computers play mp3s without this happening?

Slower Network Cards.

Then why exactly XP can handle the music just fine on the very same network card on the very same computer on the very same network?

Vista is a turd

[Anita Coney]

Why should Vista have any problems playing audio and videos?! I have an ancient 550Mhz PIII with only 256 megs of ram running W2K and it plays MP3s and video (divx and xvid) much more smoothly than my wife's Vista system (2600+ AMD, GeForce 6800XT 512MB 256-bit GDDR2, 1.5 gigs Ram). My wife's system used to run XP Pro and it rocked for everything, including games. Now even old games such as Sonic Heroes will barely run on Vista.

I gave it an honest chance, but Vista is a turd. If it can't play decades old MP3 technology MS should really give it up.

Re:Failed engineering

[blahlemon]

On the other hand (and I'm not disagreeing with you) maybe when they were testing the media functions of the operating system they didn't look at the network traffic performance cause they've got nothing to do with each other. Kinda like hearing a noise in your engine; you're not going to check the drivers side door hinges. On the other hand, you're right about the least common denominator. Fortunately we've come to expect Microsoft to play to the least common denominator.

Re:Failed engineering

[morgan_greywolf]

On the other hand (and I'm not disagreeing with you) maybe when they were testing the media functions of the operating system they didn't look at the network traffic performance cause they've got nothing to do with each other.

They have nothing to do with each other -- until you're listening an Internet radio station or a webcast of the keynote from [insert your favorite conference here].

Isn't MS going about this all wrong?

[tkrotchko]

This reminds me of the classic way program managers attack problems; they don't think about what the problem really is, nor do they care. They were given a problem to solve, and they take the shortest (easiest/cheapest) path to fix it.

The *symptom* is that Vista will glitch multimedia playback under certain circumstances.

The *issue* is that multimedia is too costly on system resources.

The *answer* is to streamline the way that Vista processes multimedia. A dual core Intel processor with 2 gig of memory should handle gig ethernet file copying, playing blu-ray disks, serve as a network router, and render video all at the same time. The idea that handling interrupts from a network card causes MP3's to glitch should be a cause of concern to both Microsoft and it's customers.

Perversely though, Microsoft took exactly the opposite approach... since multimedia is so slow, they handled it by starving everything else and giving even more CPU time to multimedia. Which solves the symptom, but doesn't actually touch on the issue. I think the real fear is the DRM layers that are built into Vista are at the root of this problem. If so, it's going to take a helluva service pack to make Vista as good as promised.

Re:Russinovich

[Christopher_G_Lewis]

No.

Have you ever seen a talk by Mark?

While he might be on the Microsoft payroll, he is definitely NOT one to sell-out.

Re:Failed engineering

[dave420]

So by your logic various flavours of Linux are even more humiliatingly bad than Vista, as I've had a slew of problems on Linux that make this bug seem like desired functionality. If you're going to troll, at least try and do it logically!

Re:Okay...

[Opportunist]

The question stands, though, why is an older system capable of playing multimedia content without throttling the network throughput on the same hardware? We're facing the same silicon, so whatever Vista uses to schedule or priorize, it has to mean Vista is less performant than its predecessors. If it was not, there would be no throttling, since said previous versions are capable of playing MMC without throttling the NIC.

There are two possible scenarios now:

1. Vista is actually less performant and the inferior system.
2. We're just plain lucky that we get to play MMC on XP and 2k without interruption, and the system throttles network performance on a "just in case" basis. In this case it's a bug that should be fixed.

Engineering for profit vs. for improvement

[IgD]

I think this is a great example of why the open source development model will lead to better outcomes. Microsoft apparently tweaked Windows for profit instead of to improve efficiency or user experience. This design flaw would have been identified immediately in the open source world and would have been rightly discarded.

Re:Well that's just not true

[gazbo]

Hello, I enjoyed your post and therefore felt you deserved a reply. Feel free to add that to your CV.

http://it.slashdot.org/comments.pl?sid=280101&cid= 20366549 [slashdot.org]
http://it.slashdot.org/article.pl?sid=07/08/26/162 8200 [slashdot.org]
http://it.slashdot.org/comments.pl?sid=280101&cid= 20377327 [slashdot.org]

If they are meant to be humourous then my sense of humour must be completely broken. I'm sure there are more comments of equal paranoia to be found in previous installments of this saga.

Re:Failed engineering

[gladish]

As a long-time Linux user (over 10 years) I was always in the "Windows Sucks" camp. Recently I decided to really understand windows at the level of my understanding of Linux. I just finished a 5 year stint doing Linux systems level programming with the latter 2 years doing more on BSD. After reading "Windows Systems Progamming" by Johnson Hart, I was astonished at the complexity of the windows api (win32). Things that are really straight forward with posix programming are a genuine mess with win32. The nubmer of synchronization mechanisms is overwhelming. But after a while you begin to appreciate the flexiblity that the system provides. I decided to move on and buy "Microsoft Windows Internals" by Russinovich and Solomon and am currently reading that. Again, they expose some nasty details of windows and again you'll be saying to yourself, "Oh my god, they over engineered the shit out of this thing." But they continually bring up what the design goals were and again you begin to appreciate what Microsoft has accomplished with windows. Of course you can't expect the system to be flawless. Linux certainly isn't. If you're a windows user, just be glad there are people like Russinovich who can actually understand the windows kernel enough so that Microsoft can continue to make improvements. If you couldn't care less about windows, then I'd still reccomend either book. If you're into Linux (or any posix-like) systems level programming, check out Johnson Hart's book. It's audience is unix converts. If you're just interested in the windows kernel or are a sys admin, check out Russinovich's book. It's really interesting.

Re:Failed engineering

[Rude Turnip]

The decoding of the sound takes place on the receiving end (ie xbox, media receiver)...

Re:Failed engineering

[dave420]

So you don't have any idea what you're talking about. This issue has nothing to do with rights management or money farming. It was a mistake that is being rectified. As for the WGA breakdown - that didn't affect anyone negatively for more than a day or two. Microsoft issued help to get it fixed for those hit, and all is well.

Linux isn't in strong shape on the desktop. It doesn't have the application support it needs, its drivers aren't able to perform as well as their Windows counterparts, which means it's constantly making excuses for not being able to use 100% of the computer its on. But then everyone knows this.

I'd rather have an OS that runs every bit of software I want (including games, video editing, office suites, open source apps, etc.) and may have occasional problems, than one that doesn't run everything I want, and still has occasional problems.

I admire your spirit, though :)

Re:Failed engineering

[jvkjvk]

But it really, really, REALLY doesn't matter if it was derived from experiments, or that they didn't test other configurations.

The fact is that there should be absolutely no need to throttle network processing when playing back content.

It is simply ridiculous that the latest, ahem, greatest OS from Microsoft has this limitation.

What's even more ridiculous is that the engineers at MS thought that it was OK.

If the OS requires throttling network traffic to play an MP3 "glitch free" and this is the best solution that Microsoft could come up with, there are a major issues with the guts of the system.

Vista media playback worse than 486

[farbles]

I used to play mp3s with no stuttering on a 486-100 using DosAmp. I cannot play mp3s or video without stuttering on Vista with a dual core 2.4 GHz CPU, 4 GB RAM, 500 GB SATA drive. I can put my XP SP2 drive on the same computer and play media flawlessly while (gasp) multitasking. Like the man says, Vista is a turd.

When my old XP HD crashed I was forced to use Vista exclusively for several weeks. It was like my computer was sick and in the hospital. No TV from my ATI x800 All-in-Wonder (though I did get the FM radio working after a week or two), sucky video game frame rates, unstable network card and sound card drivers and crap multimedia playback. P2P kept crashing the network stack.

Some people say that this isn't Microsoft's fault, it's those third party driver writers to blame. I say fuck that, these folks can write good drivers for the exact same computer in several other operating systems. It's Vista's fault.

MS fanboys will all come out and say their systems all work perfectly. Horseshit. I've now had hands on with more than two dozen Vista machines ranging from laptops to upgrades and in every single case, that's 100% MS fanboys, not 99%, not 80%, all of them had stuttering media playback.

There is no excuse for this sort of crap. My goodness it was such a relief to get an XP install back. My computer was perkier and all of a sudden everything worked again.

If Microsoft does not fix this with the mother of all service pack releases rewriting Vista from the core out then my next post-XP os will not be Windows. My best guess is Vista SP1 will be lipstick on a pig rather than the thorough cleaning out that poor excuse for a beta release really needs though.

Re:And then again...

[AaronW]

I had to implement something like this to dynamically throttle packets back based on the load in a router type box. It's not rocket science. I think Linux (if it doesn't already do this) could do it fairly easily with their NAPI networking interface, since the OS can slow down polling and assign a higher priority to audio and video.

Re:Failed engineering

[Anonymous Coward]

Gigabit TCP communication using software processing alone is enough to fully load a 2.4 GHz Pentium 4 processor, resulting in little or no processing resources left for the applications to run on the system. As of 2006, very few consumer network interface cards support TOE.

From Wikipedia [wikipedia.org]

Re:Failed engineering

[Anonymous Coward]

Testing? What's that?

I'm finally leaving MS after some number of years largely because of this. The overriding philosophy seems to be: Test against the spec. Test shouldn't be responsible for independent thought. Gone are the days when Test was charged with understanding how their feature would integrate with the whole and not only test the feature but "test" the spec. Now it's all about creating automation and making sure the (very often, dev written) spec matches the feature. If the feature makes the rest of the product suck, there's little the test org can do.

Add on top that an amazing lack of apathy about the product over all. My (rather large) group is adding features that will piss the user off instead of working on stability. Why? Because marketing is more concerned about getting new users than making current users happy.

In the end this products will either collapse under the weight of their features or there will be a large shift at MS. Either way, I don't care. My group's product has gone from decent to great to riddled with useless crap that actively pisses users off.

Re:Failed engineering

[JasterBobaMereel]

This would be the same Mark Russinovich who set up his own company to sell the tools he wrote to find out why windows was so bad and to fix the holes and mistakes in Windows, and so he could write books telling people how to work around the flaws in windows (sysinternals)

And when Microsoft found out people were listening to him... they bought it and hired him to be a Microsoft Advocate, but don't seem to have listened to him ?

The Windows kernel (NT Kernel) was designed by Dave Cutler (ex of DEC) who designed a quite nice kernel then was promoted out of the way while Microsoft ruined his design...

R.I.P Mark Russinovich

[Coward Anonymous]

Back when sysinternals was still independent, Mark provided real information with real criticism when things were wrong. Apparently, things have changed.
His "analysis" here is not much more than a series of rationalizations and excuses:

"Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet's protocol, updates the connection state, finds the receiving application, and copies the received data into the application's buffers." (emphasis mine)

The issue at hand is related to gigE NICs. Please find me a single gigE NIC that does not support TCP/IP checksum offload (even the lowly Realtek does).

His graph showing 40% CPU utilization during a file copy must be a joke or an admission of a dismally performing network stack. There are only 2 possible explanations for that number:
1. His file copy was saturating a 1gigE link - if you've saturated the link, 40% is not great but is decent. However, the test is not applicable to most people who've seen the issue. It also means there is another 60% of the CPU for processing audio - that should be plenty.
2. His file copy was nowhere near saturating the link and Vista's network stack is horribly inefficient. My experience with pervious incarnations of Windows (2K, 2K3 and XP) has shown that under ideal conditions a single file copy will max out (because of inefficiencies in CIFS but that's another story) at ~35MB/s (roughly 1/3 of a gigE link in one direction). If Vista performs at roughly the same rate, then 40% CPU for 35MB/s is terrible. No wonder there is a degradation problem that required network throttling.

Looking down further to the NDIS packet graph, it appears that it is indeed explanation 2 that is correct. Peak throughput through the system was 24.6MB/s (17215*1500). If this test was similar to the CPU test for the previous screenshot, we are seeing 40% for 24.6MB/s. It appears the system will saturate its CPU at 50MB/s half-duplex?!? That's horrible. Or Mark is showing different numbers from different tests. I'm not sure which I want to believe.

Something appears to be very wrong with the network stack in these experiments. I don't have Vista. Can anyone test this?

Re:Failed engineering

[Bartold]

Actually, this has nothing to do with failed engineering or Microsoft and everything to do with all of you cheap end users that don't want to shell out money for hardware accelerated audio. Software audio solutions require fine grained timing in order to minimize the mixing latency. Hardware solutions only require big buffers of data to achieve virtually zero CPU usage. I just want to know how many of you suckers out there paying more $1000 out there for a 5% faster CPU instead of $50-$75 for a sound card which would have probably given you then same 5%. Sure you get more generalized CPU processing time for those massive Linux compiles, but the instant you do anything multimedia related you can kiss it goodbye.