Mark Russinovich On Vista Network Slowdown

koro666 writes "In his latest blog post, Mark Russinovich analyzes the network slowdown experienced by some users when playing multimedia content. 'Tests of MMCSS during Vista development showed that... heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver... [to] pass along, at most 10 packets per millisecond (10,000 packets per second)... [T]he networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.'"

Failed engineering

[setagllib]

Once again, over-complication and stupid engineering lead to a humiliatingly bad operating system. It's obvious it didn't receive a modicum of real testing.

Re:Well that's just not true

[MBraynard]

While that might reduce bandwidth for your intended puposes, it would not limit the total reported network bandwidth.

But don't let logic or common sense get in your way.

Re:Okay...

[Anonymous Coward]

Because they're not Vista. Think of Vista as the operating system that the movie and music industry produced.

Re:Failed engineering

[morgan_greywolf]

Almost, but not quite. Really, it's Microsoft's drive to appeal to the least common denominator. Dumb end-users aren't likely to notice a speed decrease in their network throughput -- not even a significant one. So maybe they did test it, but ignored any performance feedback about the network because it was ignored as smart power users being 'overly picky', since their target customer requires that the CD cases be printed on drool-proof cardboard.

Dumb dumb dumb

[Anonymous Coward]

"MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver, which is the driver that gives packets received by network adapter drivers to the TCP/IP driver, that causes NDIS to "indicate", or pass along, at most 10 packets per millisecond (10,000 packets per second).

Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won't see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you'll see throughput drop to roughly 15%."

That is one of the dumbest things I have heard in a while. Let's see:

• It's a poor solution to begin with.
• It's incorrect. Did no one even bother to calculate the drop-off? Was there not one single engineer amongst them who ever said "Hey, you know, Gigabit is pretty popular these days."?
• It should be unnecessary. Why does standard media playback and networking require so much power that there is not enough time to schedule both of them correctly?
• It is wrong. Why is media playback is more important than network performance? If the network is heavily loaded, well gee, maybe there's a reason for that?

What an over-engineered non-solution to what should have been a non-problem in the first place. Microsoft is supposed to employ some of the smartest engineers in the world: can none of them optimise their code?

Wow...

[Kr3m3Puff]

I find this totally interesting. It goes to the heart of what is wrong with Microsoft these days... All seperate groups of folks, not talking to each other, to try and do "what is best" for the user, and then totally stomping on each other. Instead of really looking at thread management and optimising the kernel, they cludge together something to make multi media work by simplying saying "in certain situations, I can't guarentee the thread because of a crappy kernel, so I am going to tell everyone else to slow down".

It is these sorts of things and things like the teams and teams debating the "Shutdown Menu" in Vista that are really showing Microsoft needs to really change if they are going to survive. It amazes me how a bunch of open source developers with all their own agendas do a better job then a bunch of folks all paid by the same company. Of course then there is Apple of an example of a group that shows you can pull it off and still all look like the same organisation.

Re:Well that's just not true

[PJ1216]

I haven't been really on the lookout for it, but I haven't seen any posts explaining that as the cause. I'd expect if that really were the cause, there'd be a much bigger outrage from people and it would have blown up and I'd see articles on it whether I wanted to or not. I don't really see any useful DRM techniques for unprotected MP3s anyway. There'd really be nothing that MS could do with that sort of information.

However, this actually does make sense. In all honesty, they probably would have worked on a better answer than cutting back on networking, but with the time crunch on releasing it, they probably cut corners here and there (and by probably, i mean definitely and by here and there, i mean everywhere). They probably viewed this as an acceptable cut for the time being because for a majority of users, they use very little of their networking bandwidth. If its just a PC connected to the internet, they'd most likely never notice. The only time this would be an issue is for heavy network usage, which would normally only occur on work-related machines because let's face it, aside from geeks and techies, not many people have systems set up that max out their network bandwidth, so, if they were work-related machines, well, they probably wouldn't be playing that much music to begin with.

I'm not a MS shill, though I don't assume everything they do has evil intentions. We have to admit that they are great code writers, just not the best. Just because they do shady things here and there (mostly in business practices however) doesn't mean everything they do is evil. This was a problem they ran into and they made a workaround that would only affect a relatively small amount of their users. They were probably hoping no one would notice it at all until they either A) had a fix or B) just let it go because maybe no one would notice it.

Remember, this wouldn't really slow down your internet unless you have an *extremely* high bandwidth and even then, bottlenecks on the information before reaching you would probably still mask the problem. This is only an issue on system that have heavy network usage on some sort of intranet or other type of local area network, because these would account for the majority of networks that could even use a decent amount of your possible networking bandwidth.

Re:Okay...

[rudy_wayne]

"MMCSS is the Multimedia Class Scheduler Service, which a new feature in Vista -- it is not in 98/95/2000/ME/XP. That's why."

Winodws XP -- can play an MP3 file and video file at the same time with no reduction in network speed.

Vista -- same computer, same hardware, -- major reduction in network speed.

In other words, Microsoft tried to "fix" something that wasn't broken.

Re:Failed engineering

[Brian Gordon]

They specifically said they throttled network speed. It's not like something they should have tested for and never found, it's something they did themselves.

Fix for this can be downloaded from here.

[Anonymous Coward]

http://www.ubuntu.com/getubuntu/download [ubuntu.com]

Re:Dumb dumb dumb

[PJ1216]

• It's a poor solution to begin with.

Yes, I agree. They should have fixed the problem instead of masking the symptoms.

It's incorrect. Did no one even bother to calculate the drop-off? Was there not one single engineer amongst them who ever said "Hey, you know, Gigabit is pretty popular these days."?

I don't know what you mean by calling it "incorrect." And honestly, just because you have a Gigabit card doesn't mean thats actually the speed all that information is going at.

It should be unnecessary. Why does standard media playback and networking require so much power that there is not enough time to schedule both of them correctly?

It's only the networking that requires the power. When the network traffic is heavy, it causes the sound to hang due to sound being one of the few things that can't "go slow." A lot of programs will probably just run slower, whereas sound will start skipping and be more annoying than just waiting for a program to finish its various operations.

It is wrong. Why is media playback is more important than network performance? If the network is heavily loaded, well gee, maybe there's a reason for that?

They probably viewed it as this: If you're using that heavy of networking traffic, you probably are doing something very important. Most important stuff on that level would probably be in the workplace, not home use. Therefore they probably viewed media as not being a big factor here, because they figured no one would be playing media. Therefore, they were then thinking about the home networks that maybe had various spikes in network traffic from network drives on gigabit switches or something, in which case they decided in slowing down that transfer was a better solution because in this case A) the network traffic probably isn't *that* important and B) there's a greater chance there's music playing a C) music skipping is *really* annoying.

Tag: defectivebydesign

[Anonymous Coward]

If this isn't "defective by design", I don't know what is.

New definition of a Kludge here, I think

[Ancient_Hacker]

Okay, I probably would have applied this patch to my software, at 2Am, with a mental note to remove it in the morning and do the right thing, smarten up the task scheduler, perhaps with an app callback saying "I'm falling behind, could you boost me up a bit?".

As goes without saying, arbitrarily throttling one particular task, at some arbitrary level, is the wrong thing.

Perhaps this could go in Wikipedia under "Kludge"?

Re:Failed engineering

[setagllib]

Various flavors of Linux can take a flying leap. The mainline Linux kernel is generally in very strong shape, and I say this after spending years loathing many bad choices in Linux. Many mainstream distributions are doing very well too. Most of all, Linux does not compromise basic performance for "rights management", which Vista does.

Vista's worst engineering decision is to make a system optimized for restrictions and money-farming, not for user experience. The WGA breakdown is the best example. The legitimate users who paid a ridiculous sum to use Vista's 'ultimate' features (you know, the ones which are free in Linux and at least standard in MacOSX) had their systems crippled, and the pirates who bypassed WGA were not even affected. The whole feature does exactly the opposite of what it was supposed to do. That's failed engineering, any way you look at it.

Re:Wow...

[TeknoHog]

It's interesting to note how Unix philosophy ties in with this difference between Microsoft and open source. With unix, there's no single defined 'user experience' to be optimized, because the components can and will be combined in various ways. Then it's the individual components and the interfaces between them, that will be tweaked and optimized.

Re:Okay...

[Daengbo]

Shouldn't any computer powerfull enough to run Vista be "powerful enough to not need such a draconian throttling?"

Hang on a minute...

[Spad]

The throttling rate Vista uses was derived from experiments that reliably achieved glitch-resistant playback on systems with one CPU on 100Mb networks with high packet receive rates. The hard-coded limit was short-sighted with respect to today's systems that have faster CPUs, multiple cores and Gigabit networks

"Today's systems"? Vista's only been out for a year, just how fucking short-sighted are they?

Re:Dumb dumb dumb

[Jose]

Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s.

And this seems like a strange conclusion to jump to...especially coming from Mark.

maybe I am just confused, but the NDIS driver handles sending and receiving of pkts, so is the pkt rate limited to 10,000 pps coming and going? (he mentions packets received by network adapter drivers, but I am still curious). if it is limited to 10,000 pps in either direction...then you the theoretical limit comes down by quite a bit.

Even at that, he is assuming full sized packets, which is a bit of stretch, there is a good chance that not all of them will be the full 1500 bytes, factor in broadcast traffic, and other crud which may be running...and you start seeing a noticable drop even on a 100mbit connection.

Re:Dumb dumb dumb

[rbochan]

...skipping music is in any way acceptable compared to slight network performance penalties?...

Actually, this is 2007, with stupidly fast processing, memory levels, and network throughput. There's no reason whatsoever that either effect should be showing up when both activities are happening at the same time.
And it's not "slight network performance penalties". It's ridiculously harsh network performance penalties.

Re:Failed engineering

[T-Bone-T]

You've already been made fun of for saying it, but I would like to add a qualifier that I can't believe nobody ever uses.

Cause no one needs more than 100mb, YET. I don't care that my network is slowing down, it won't slow down enough to hamper my internet connection.

And then again...

[Spy der Mann]

it was an IMPOSED, HARDCODED limit WITHOUT ASKING the user. They could just add a registry entry of "maximum network packets per millisecond when playing multimedia files" or something.

Microsoft has a long history of hardcoding stuff without thinking of power users. Remember the 10-limit for open TCP connections per program? They did this because viruses and malware open many TCP connections. "Hey, what about P2P?" "What's P2P?".

Re:Failed engineering

[WhiteWolf666]

There's a better criticism. These "maximum" speeds are NOT sufficient to stream medium-to-high bit rate video, particularly HD stuff, something which should be a strength for a "media center" OS.

Furthermore, in the workplace, many people listen to music and access large files on network shares. Clearly, Vista is *broken* for these uses. Not a good indication of Vista being business ready.

Frankly, I don't know why Windows is considered the best business OS. You're much better off with a unixy OS in any environment where gaming isn't important.

Re:Failed engineering

[TubeSteak]

It's not like something they should have tested for and never found, it's something they did themselves.

After reading your post, the parent post and the grandparent post (and every other +3 post in the thread) I feel like I'm the only one who made it to the end of TFA:

The throttling rate Vista uses was derived from experiments that reliably achieved glitch-resistant playback on systems with one CPU on 100Mb networks with high packet receive rates.

Things they apparently didn't bother to test for:

• Multiple NICS
• Gigabit NICS
• Multiple CPUs/Cores

Those things just seem like glaring oversights, especially considering how many people have wifi in addition to the mobo's onboard NIC.

One thing I don't get is how he managed 41.61% CPU utuilization [technet.com] while transferring a file. Did he have the ethernet equivalent of a winmodem?

Re:Failed engineering

[grasshoppa]

have you used vista?
it's a far better user experience than windows XP. if they did put some DRM related stuff in there, I haven't noticed, nor will 99.99% of its userbase.

Jesus, have *you* used vista? The user intended user experience could be orgasmic, but I'll be damned if I can get the thing stable given the state of drivers for my vista approved hardware.

In a year it may be better than XP ( and at best, marginally so ), but right now it's hit and miss.

How short-sighted?

[wild_berry]

I'd had two CPU's and Gigabit Ethernet for three years by the time that Vista was on sale to the public. That's not simply "short-sighted with respect to today's systems", that's a total let down to businesses who have high-performance workstations.

Re:Engineering for profit vs. for improvement

[Mathinker]

> tweaked Windows for profit instead of to improve efficiency or user experience

Did you read the article? It was obviously tweaked to improve the "user experience"; the painful difference between OSS and this being that Microsoft arbitrarily decided for all of Vista's users what "user experience" they would like to experience (i.e., skipless media playback as opposed to maximum network performance). There were bugs in Microsoft's solution, but there are also bugs in OSS.

OSS projects, however, are (usually) much less dictatorial in deciding what the user wants; they can't be, actually, because if he doesn't like what they give him, he can just fork-and-run.

Re:Dumb dumb dumb

[PJ1216]

Wouldn't it be better to just let the music skip, as that's most likely the more unimportant and can easily be shut off? Heck, if not listening to music makes the difference between going home on 5 and working for two hours longer I'd happily choose silence.

That's also an assumption as well. Personally, I rarely have any heavy network usage, though it will spike. If I used Vista, I would rather the network be throttled than the music skip. I'm assuming a majority of computer users are not hardcore users and therefore they were the ones catered too. However, you're second point about not knowing its the media player causing the throttling is a valid point. It would have been better to make this throttling mechanism an option as opposed to forcing it on people. Though, if I were to force the option, I'd choose the same as they did. It's much more likely they'll appease more people than piss off. Hell, most of the people that would be pissed off probably aren't using Vista anyway =P

Re:And then again...

[Lonewolf666]

They could just add a registry entry of "maximum network packets per millisecond when playing multimedia files" or something.

Better yet, allow "throttling as needed if multimedia buffers run low". That would allow unimpaired network performance in systems with enough CPU power.

But then again, that would have required early planning to include the necessary feedback in audio and graphics drivers. I speculate that the problem was discovered late in the development of Vista, and since nobody wanted to be responsible for another delayof Vista's release, some quick hack was applied ;-)

Re:Dumb dumb dumb

[Panaflex]

The important point, which the vast majority of posters here are missing, is that the problem should never even exist in the first place.

They know why.. it's the kernel-mode encryption required to send audio to the card. There's two engineering failures here:
1. thread locking is accomplished by raising the interrupt level to DPC (KeAcquireSpinLock)
2. Requiring several steps/levels of encryption to interract with the audio card.

The real issue is a combination of utilizing DPC interrupts for basic thread locking (which thrashes the scheduler during long halts) and encryption (which requires long halts).

The real fallacy, IMHO, is thatMS thinks that because it's in kernel mode that it's immune or safer from attacks - so they created lots of "security features" in the kernel. In many ways this makes attacks much simpler - as you can simply move your code into kernel mode which has fewer limits than user mode!

Corporate Lingo

[DaveDerrick]

Just what is a "glitch-resistant experience" ? Before he was in M$ payroll, Mark would have called this something else. Sorry, but his "technical authority" value, that Microsoft are hoping to use to explain away their bugs, has lost all value now he's making their excuses for them.

P2P?

[Bill, Shooter of Bul]

I don't think that you could convince Microsoft to change their code to allow software who's primary usage is piracy, including that of Microsoft products. In fact, I think this particular behavior was indeed by design.

From the article...

[mariushm]

As written in the article...

Besides activity by other threads, media playback can also be affected by network activity. When a network packet arrives at system, it triggers a CPU interrupt, which causes the device driver for the device at which the packet arrived to execute an Interrupt Service Routine (ISR). Other device interrupts are blocked while ISRs run, so ISRs typically do some device book-keeping and then perform the more lengthy transfer of data to or from their device in a Deferred Procedure Call (DPC) that runs with device interrupts enabled. While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run, and can therefore impede media playback threads. They're saying that every packet received causes an interrupt request, which causes the CPU to get loaded at high transfer speeds.

Apparently they haven't heard of interrupt moderation [google.com] or polling [google.com], technologies that are used by network cards to offload the CPU.

Even my Marvell semi-hardware (I think) Gigabit on-board network card used about 14% CPU (Barton 1833Mhz) when transferring files at about 45Mbps.

I don't know, everything seems really stupid, and I'm not sure it's just a "bug", or their description is just a part of what really happens behind the scene.

Re:Russinovich

[LordSnooty]

I think it's right to at least pose the question, though. Before, Mark was a Windows expert working independently, and was able to voice opinions as he saw fit. Now he's a Windows expert being paid by the company that makes Windows - the very success of Windows Vista will dictate how long his job lasts. He now has an interest in assuring customers and investors that things aren't as bad as they might be. Now, it's all about the bottom line. Of course, he's built up a lot of trust amongst the community, trust which MS themselves are now paying money for. Whilst we can continue to trust him, until proved otherwise, it's not wrong to least ask the question.

Re:Hard coded numbers (10k packets/sec)?

[WhiteWolf666]

It's not a flawed implementation, its defective by design (and I don't mean DRM'd, I mean literally).

On Linux, with the CFS and/or SD schedulers, if your nice levels are set correctly, sound (MP3) will play just fine with your processor(s) pegged at 100. Heck, forget about sound; you can run multiple Quake 4s with high-speed LAN transfers in the background, and everything works just fine (network transfers slowdown slightly, Quake 4's FPS scales down linearly with the number of sessions running, but there are no "hitches" or "glitches", and everything runs smoothly).

A common Microsoft approach to problems with Windows is to create a new daemon (oh, excuse me, Service) that "regulates" the offending behavior. This is not the correct way to fix these problems; rather, there are underlying issues that need to be resolved.

You say:
The MMCSS is for improving multimedia performance on EXTREMELY heavily-loaded processors. I use XP, and my PC is occasionally heavily loaded with a dozen threads, and in those cases I occasionally experience glitches. Thus, I have to manually adjust thread priorities, but it's annoying anyway.

I say it's not about manually adjust thread priorities, or creating a Service that will automatically (dynamically or not) do that for you. Rather, you should have a kernel that better manages multitasking in processor starved scenarios. There's no reason that a particular program running at a particular nice level shouldn't demand a minimum CPU percentage, which for stuff like playing MP3s cannot possibly be much.

Re:And then again...

[Xtravar]

it was an IMPOSED, HARDCODED limit WITHOUT ASKING the user.

Not to start a flame war, but isn't this exactly what people find so great about Macs - that the OS designers made all the decisions for them?

Most coders don't want to add a registry setting. Most users don't want to touch it.

There's obviously just something wrong with their big picture view if they can't get this shit straight. It's probably because the network and multimedia teams are separate and don't know what the others' doing.

Re:Failed engineering

[robbiethefett]

though there is a certain humor in that what it says is basically true.

Ok, so on one hand we have what can be described as the sum of all human knowledge, and on the other hand we have the belief that God exists, created mankind and dinosaurs at the same time about 10,000 years ago, and will send angels riding on flying horses to doom the earth.
Yea, it's totally reasonable to think that it takes more faith to exist in the real world than it does to believe in ghosts and boogie-men.

Re:Failed engineering

[Z0mb1eman]

>But after a while you begin to appreciate the flexiblity that the system provides

>you begin to appreciate what Microsoft has accomplished with windows

I've always assumed there's more to it than just "Windows sucks", but I've never had the time to learn about how Windows and Linux work more in-depth so I can meaningfully compare them (nor will I anytime soon).

Care to give an example or two of things Windows gets right?

Mod parent redundant ;-)

[Ernesto Alvarez]

1% to bring them all and in the darkness bind them. In the land of buffers, where the packets lie.

So, basically the GP poster was right: 1% goes to WGA.

Re:Failed engineering

[Master of Transhuman]

Or you're one of maybe 100 million office workers who play MP3's they have on the workstation with Windows Media Player or Winamp through headphones while they're working on files over the local office network.

I can just understand Microsoft not being aware of that scenario - except you can guarantee that EVERY SINGLE MICROSOFT EMPLOYEE just does that. So nobody thought to test that scenario - that was just dumb of Microsoft.

The real question is why the engineers involved didn't understand the size of the impact on performance. I mean, if you made the mod in order to avoid network performance screwing up media playback, then why didn't they explicitly test the degree of impact on the networking performance AS WELL AS the media playback? They were explicitly degrading network performance in favor of media playback. Why didn't they SEE the performance hit?

So one has to conclude that this is correct: they simply didn't test it. They just tested the media playback - if in fact they tested that at all.

I'm reminded of the post at a Microsoft employee's blog last year where a member of the Vista testing team explicitly said that setting up tests was a nightmare that took most of a week - and then when Vista failed the tests horribly, management would STILL sign off on the components as having passed.

And this is the obvious result of that process.