Slashdot Log In
Mark Russinovich On Vista Network Slowdown
Posted by
kdawson
on Tue Aug 28, 2007 07:09 AM
from the nuts-and-bolts dept.
from the nuts-and-bolts dept.
koro666 writes "In his latest blog post, Mark Russinovich analyzes the network slowdown experienced by some users when playing multimedia content. 'Tests of MMCSS during Vista development showed that... heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver... [to] pass along, at most 10 packets per millisecond (10,000 packets per second)... [T]he networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.'"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Aaah (Score:5, Funny)
Failed engineering (Score:4, Insightful)
Re:Failed engineering (Score:5, Insightful)
Parent
Re:Failed engineering (Score:4, Interesting)
Parent
Re: (Score:3, Interesting)
Re:Failed engineering (Score:5, Funny)
Err...or was that something else?
Parent
Re:Failed engineering (Score:5, Insightful)
Parent
And then again... (Score:5, Insightful)
Microsoft has a long history of hardcoding stuff without thinking of power users. Remember the 10-limit for open TCP connections per program? They did this because viruses and malware open many TCP connections. "Hey, what about P2P?" "What's P2P?".
Parent
Re:And then again... (Score:5, Insightful)
Better yet, allow "throttling as needed if multimedia buffers run low". That would allow unimpaired network performance in systems with enough CPU power.
But then again, that would have required early planning to include the necessary feedback in audio and graphics drivers. I speculate that the problem was discovered late in the development of Vista, and since nobody wanted to be responsible for another delayof Vista's release, some quick hack was applied
Parent
Re:And then again... (Score:4, Interesting)
Parent
Re:And then again... (Score:4, Insightful)
Most coders don't want to add a registry setting. Most users don't want to touch it.
There's obviously just something wrong with their big picture view if they can't get this shit straight. It's probably because the network and multimedia teams are separate and don't know what the others' doing.
Parent
Re:Failed engineering (Score:5, Insightful)
- Multiple NICS
- Gigabit NICS
- Multiple CPUs/Cores
Those things just seem like glaring oversights, especially considering how many people have wifi in addition to the mobo's onboard NIC.One thing I don't get is how he managed 41.61% CPU utuilization [technet.com] while transferring a file. Did he have the ethernet equivalent of a winmodem?
Parent
Re:Failed engineering (Score:5, Funny)
4.4% to draw the moving file animation (it re-reads it every time the anim loops).
3.8% to report to MS about the file you're copying.
2.1% is wasted on old code that constantly scans memory for pictures of rabbits (Balmer is scared of them)
1% is needed for WGA.
2.5% because Vista constantly swaps all application code in and out of the first 640k. Bill still believes its enough.
1.7% to actually copy the file.
the rest is just wasted to make CPU graphs look pretty.
Parent
Re:Failed engineering (Score:5, Funny)
Parent
Re:Failed engineering (Score:5, Funny)
One thing I don't get is how he managed 41.61% CPU utuilization while transferring a file. Did he have the ethernet equivalent of a winmodem?
No, he had the OS equivalent of a Winmodem.
Parent
Re:Failed engineering (Score:4, Funny)
Parent
Re:Failed engineering (Score:5, Informative)
If you're talking about MTU (which is 1500 bytes, not 1.5K), that's the maximum, not the standard.
The average packet size depends on type of network traffic. On most ethernet networks I've managed, average packet size was 700 to 800 bytes.
Parent
Re:Failed engineering (Score:5, Insightful)
Furthermore, in the workplace, many people listen to music and access large files on network shares. Clearly, Vista is *broken* for these uses. Not a good indication of Vista being business ready.
Frankly, I don't know why Windows is considered the best business OS. You're much better off with a unixy OS in any environment where gaming isn't important.
Parent
Re:Failed engineering (Score:5, Informative)
Okay. But, math doesn't match up with the numbers you typed.
1,500 Bytes is not the average packet, it's the maximum on most ethernet segments. But, the subject original subject is a stressful network environment effecting music playback. 10,000 packets per second is REALLY cranking the data.. so this isn't simple WWW browsing, etc. This is bulk file transfer. So, a large average packet size becomes more realistic in that environment.. say 1400 Bytes.
1,400 Bytes * 10,000 Packets per second = 14,000,000 BYTES / sec = 112,000,000 bits/sec = 112 Mbps
Obviously, that's not even possible on most common home networks, which are 100Mbps. But, an increasing number of people are doing Gig-E at home, in which case 112Mbps is well within the norms for bulk file transfer.
On modern fast multi-core systems, enforcing a pre-set cutoff for packet rate seems like a poor choice. As the linked article showed, the system had plenty of CPU left and didn't need to be throttled at that low a rate. There are also NIC and Driver factors in there.. others might be more or less efficient than the author's equipment -- offload of parts of packet processing and interrupt minimizing techniques can make a big difference.
In any case.. It's easy to say "that's what you get for using MS / Vista". But, really.. that's true in this case. Windows gives you the lowest common denominator. It's designed to be usable with any hardware, by users of any experience level, and to avoid problems by assuming a worst case scenario. So, that's the kind of solution you get given the assumptions MS uses. As we've seen in the Linux world, the solution is to take great pains to build a scheduler that holds up to ridiculous stresses.
Parent
Re:Failed engineering (Score:5, Interesting)
Parent
Re:Failed engineering (Score:5, Insightful)
Vista's worst engineering decision is to make a system optimized for restrictions and money-farming, not for user experience. The WGA breakdown is the best example. The legitimate users who paid a ridiculous sum to use Vista's 'ultimate' features (you know, the ones which are free in Linux and at least standard in MacOSX) had their systems crippled, and the pirates who bypassed WGA were not even affected. The whole feature does exactly the opposite of what it was supposed to do. That's failed engineering, any way you look at it.
Parent
Re:Failed engineering (Score:5, Interesting)
Linux isn't in strong shape on the desktop. It doesn't have the application support it needs, its drivers aren't able to perform as well as their Windows counterparts, which means it's constantly making excuses for not being able to use 100% of the computer its on. But then everyone knows this.
I'd rather have an OS that runs every bit of software I want (including games, video editing, office suites, open source apps, etc.) and may have occasional problems, than one that doesn't run everything I want, and still has occasional problems.
I admire your spirit, though
Parent
Re:Failed engineering (Score:4, Insightful)
it's a far better user experience than windows XP. if they did put some DRM related stuff in there, I haven't noticed, nor will 99.99% of its userbase.
Jesus, have *you* used vista? The user intended user experience could be orgasmic, but I'll be damned if I can get the thing stable given the state of drivers for my vista approved hardware.
In a year it may be better than XP ( and at best, marginally so ), but right now it's hit and miss.
Parent
Okay... (Score:4, Interesting)
Re: (Score:3, Funny)
Slower Network Cards.
Re:Okay... (Score:5, Interesting)
Slower Network Cards.
Parent
Re:Okay... (Score:5, Insightful)
Winodws XP -- can play an MP3 file and video file at the same time with no reduction in network speed.
Vista -- same computer, same hardware, -- major reduction in network speed.
In other words, Microsoft tried to "fix" something that wasn't broken.
Parent
Re:Okay... (Score:5, Informative)
No, in other words, Microsoft/**AA tried controlling something they weren't in control of before.
Where do you want to go today, indeed.
Parent
Re:Okay... (Score:4, Interesting)
There are two possible scenarios now:
1. Vista is actually less performant and the inferior system.
2. We're just plain lucky that we get to play MMC on XP and 2k without interruption, and the system throttles network performance on a "just in case" basis. In this case it's a bug that should be fixed.
Parent
Oblig (Score:4, Funny)
No problems here (Score:4, Funny)
Dumb dumb dumb (Score:5, Insightful)
Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won't see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you'll see throughput drop to roughly 15%."
That is one of the dumbest things I have heard in a while. Let's see:
What an over-engineered non-solution to what should have been a non-problem in the first place. Microsoft is supposed to employ some of the smartest engineers in the world: can none of them optimise their code?
Re:Dumb dumb dumb (Score:5, Insightful)
And this seems like a strange conclusion to jump to...especially coming from Mark.
maybe I am just confused, but the NDIS driver handles sending and receiving of pkts, so is the pkt rate limited to 10,000 pps coming and going? (he mentions packets received by network adapter drivers, but I am still curious). if it is limited to 10,000 pps in either direction...then you the theoretical limit comes down by quite a bit.
Even at that, he is assuming full sized packets, which is a bit of stretch, there is a good chance that not all of them will be the full 1500 bytes, factor in broadcast traffic, and other crud which may be running...and you start seeing a noticable drop even on a 100mbit connection.
Parent
Re:Dumb dumb dumb (Score:5, Insightful)
Actually, this is 2007, with stupidly fast processing, memory levels, and network throughput. There's no reason whatsoever that either effect should be showing up when both activities are happening at the same time.
And it's not "slight network performance penalties". It's ridiculously harsh network performance penalties.
Parent
Re:Dumb dumb dumb (Score:4, Insightful)
They know why.. it's the kernel-mode encryption required to send audio to the card. There's two engineering failures here:
1. thread locking is accomplished by raising the interrupt level to DPC (KeAcquireSpinLock)
2. Requiring several steps/levels of encryption to interract with the audio card.
The real issue is a combination of utilizing DPC interrupts for basic thread locking (which thrashes the scheduler during long halts) and encryption (which requires long halts).
The real fallacy, IMHO, is thatMS thinks that because it's in kernel mode that it's immune or safer from attacks - so they created lots of "security features" in the kernel. In many ways this makes attacks much simpler - as you can simply move your code into kernel mode which has fewer limits than user mode!
Parent
Re: (Score:3, Funny)
Apparently not, if you use Microsoft products.
Wow... (Score:5, Insightful)
It is these sorts of things and things like the teams and teams debating the "Shutdown Menu" in Vista that are really showing Microsoft needs to really change if they are going to survive. It amazes me how a bunch of open source developers with all their own agendas do a better job then a bunch of folks all paid by the same company. Of course then there is Apple of an example of a group that shows you can pull it off and still all look like the same organisation.
Re: (Score:3, Insightful)
Vista is a turd (Score:5, Interesting)
I gave it an honest chance, but Vista is a turd. If it can't play decades old MP3 technology MS should really give it up.
Completely Unfair Scheduler (Score:5, Funny)
New definition of a Kludge here, I think (Score:4, Insightful)
As goes without saying, arbitrarily throttling one particular task, at some arbitrary level, is the wrong thing.
Perhaps this could go in Wikipedia under "Kludge"?
Hang on a minute... (Score:5, Insightful)
How short-sighted? (Score:3, Insightful)
Engineering for profit vs. for improvement (Score:3, Interesting)
Wasn't AV playback conquered years ago? (Score:3)
Sure, video (especially HD content) has much higher bandwidth demands, but local video playback has NEVER been a problem on any machine I've owned in the last 10 years. I remember IBM thinkpads with PII 266 processors that could easily play DVDs.
The only explanation for Vista's media playback design decision must be to compensate for the huge processing overhead that Vista creates. Poor fundamental design decisions necessitated hacks like this prioritization scheme.
-ted
Vista media playback worse than 486 (Score:5, Interesting)
When my old XP HD crashed I was forced to use Vista exclusively for several weeks. It was like my computer was sick and in the hospital. No TV from my ATI x800 All-in-Wonder (though I did get the FM radio working after a week or two), sucky video game frame rates, unstable network card and sound card drivers and crap multimedia playback. P2P kept crashing the network stack.
Some people say that this isn't Microsoft's fault, it's those third party driver writers to blame. I say fuck that, these folks can write good drivers for the exact same computer in several other operating systems. It's Vista's fault.
MS fanboys will all come out and say their systems all work perfectly. Horseshit. I've now had hands on with more than two dozen Vista machines ranging from laptops to upgrades and in every single case, that's 100% MS fanboys, not 99%, not 80%, all of them had stuttering media playback.
There is no excuse for this sort of crap. My goodness it was such a relief to get an XP install back. My computer was perkier and all of a sudden everything worked again.
If Microsoft does not fix this with the mother of all service pack releases rewriting Vista from the core out then my next post-XP os will not be Windows. My best guess is Vista SP1 will be lipstick on a pig rather than the thorough cleaning out that poor excuse for a beta release really needs though.
R.I.P Mark Russinovich (Score:4, Interesting)
His "analysis" here is not much more than a series of rationalizations and excuses:
"Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet's protocol, updates the connection state, finds the receiving application, and copies the received data into the application's buffers." (emphasis mine)
The issue at hand is related to gigE NICs. Please find me a single gigE NIC that does not support TCP/IP checksum offload (even the lowly Realtek does).
His graph showing 40% CPU utilization during a file copy must be a joke or an admission of a dismally performing network stack. There are only 2 possible explanations for that number:
1. His file copy was saturating a 1gigE link - if you've saturated the link, 40% is not great but is decent. However, the test is not applicable to most people who've seen the issue. It also means there is another 60% of the CPU for processing audio - that should be plenty.
2. His file copy was nowhere near saturating the link and Vista's network stack is horribly inefficient. My experience with pervious incarnations of Windows (2K, 2K3 and XP) has shown that under ideal conditions a single file copy will max out (because of inefficiencies in CIFS but that's another story) at ~35MB/s (roughly 1/3 of a gigE link in one direction). If Vista performs at roughly the same rate, then 40% CPU for 35MB/s is terrible. No wonder there is a degradation problem that required network throttling.
Looking down further to the NDIS packet graph, it appears that it is indeed explanation 2 that is correct. Peak throughput through the system was 24.6MB/s (17215*1500). If this test was similar to the CPU test for the previous screenshot, we are seeing 40% for 24.6MB/s. It appears the system will saturate its CPU at 50MB/s half-duplex?!? That's horrible. Or Mark is showing different numbers from different tests. I'm not sure which I want to believe.
Something appears to be very wrong with the network stack in these experiments. I don't have Vista. Can anyone test this?
Re:Well that's just not true (Score:5, Insightful)
However, this actually does make sense. In all honesty, they probably would have worked on a better answer than cutting back on networking, but with the time crunch on releasing it, they probably cut corners here and there (and by probably, i mean definitely and by here and there, i mean everywhere). They probably viewed this as an acceptable cut for the time being because for a majority of users, they use very little of their networking bandwidth. If its just a PC connected to the internet, they'd most likely never notice. The only time this would be an issue is for heavy network usage, which would normally only occur on work-related machines because let's face it, aside from geeks and techies, not many people have systems set up that max out their network bandwidth, so, if they were work-related machines, well, they probably wouldn't be playing that much music to begin with.
I'm not a MS shill, though I don't assume everything they do has evil intentions. We have to admit that they are great code writers, just not the best. Just because they do shady things here and there (mostly in business practices however) doesn't mean everything they do is evil. This was a problem they ran into and they made a workaround that would only affect a relatively small amount of their users. They were probably hoping no one would notice it at all until they either A) had a fix or B) just let it go because maybe no one would notice it.
Remember, this wouldn't really slow down your internet unless you have an *extremely* high bandwidth and even then, bottlenecks on the information before reaching you would probably still mask the problem. This is only an issue on system that have heavy network usage on some sort of intranet or other type of local area network, because these would account for the majority of networks that could even use a decent amount of your possible networking bandwidth.
Parent
Re: (Score:3, Interesting)
Have you ever seen a talk by Mark?
While he might be on the Microsoft payroll, he is definitely NOT one to sell-out.
Re: (Score:3)