Mark Russinovich On Vista Network Slowdown 423
koro666 writes "In his latest blog post, Mark Russinovich analyzes the network slowdown experienced by some users when playing multimedia content. 'Tests of MMCSS during Vista development showed that... heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS' glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver... [to] pass along, at most 10 packets per millisecond (10,000 packets per second)... [T]he networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.'"
Aaah (Score:5, Funny)
Failed engineering (Score:4, Insightful)
Re:Failed engineering (Score:5, Insightful)
Re:Failed engineering (Score:4, Interesting)
Re: (Score:3, Interesting)
Re: (Score:2)
Re:Failed engineering (Score:5, Funny)
Err...or was that something else?
Re: (Score:3, Interesting)
Re: (Score:2, Insightful)
Cause no one needs more than 100mb, YET. I don't care that my network is slowing down, it won't slow down enough to hamper my internet connection.
Re: (Score:3, Insightful)
I can just understand Microsoft not being aware of that scenario - except you can guarantee that EVERY SINGLE MICROSOFT EMPLOYEE just does that. So nobody thought to test that scenario - that was just dumb of Microsoft.
The real question is why the engineers involved didn't understand the size of the i
Re:Failed engineering (Score:5, Insightful)
And then again... (Score:5, Insightful)
Microsoft has a long history of hardcoding stuff without thinking of power users. Remember the 10-limit for open TCP connections per program? They did this because viruses and malware open many TCP connections. "Hey, what about P2P?" "What's P2P?".
Re:And then again... (Score:5, Insightful)
Better yet, allow "throttling as needed if multimedia buffers run low". That would allow unimpaired network performance in systems with enough CPU power.
But then again, that would have required early planning to include the necessary feedback in audio and graphics drivers. I speculate that the problem was discovered late in the development of Vista, and since nobody wanted to be responsible for another delayof Vista's release, some quick hack was applied
Re:And then again... (Score:4, Interesting)
Re: (Score:3, Informative)
You're almost right. The limit is for half-open connections: these are connections in the process of being made; however, this can effectively limit your amount of connections for things like bittorrent, because you connect slower than other peers. When you're constantly disconnecting and reconnecting to new peers every second, it becomes a p
Re:And then again... (Score:4, Insightful)
Most coders don't want to add a registry setting. Most users don't want to touch it.
There's obviously just something wrong with their big picture view if they can't get this shit straight. It's probably because the network and multimedia teams are separate and don't know what the others' doing.
Not just piracy!!! (Score:3, Funny)
P2P is also widely used to dowload free software! Mainly Linux distros! Oh... Nevermind.
Re:Failed engineering (Score:5, Insightful)
One thing I don't get is how he managed 41.61% CPU utuilization [technet.com] while transferring a file. Did he have the ethernet equivalent of a winmodem?
Re:Failed engineering (Score:5, Funny)
4.4% to draw the moving file animation (it re-reads it every time the anim loops).
3.8% to report to MS about the file you're copying.
2.1% is wasted on old code that constantly scans memory for pictures of rabbits (Balmer is scared of them)
1% is needed for WGA.
2.5% because Vista constantly swaps all application code in and out of the first 640k. Bill still believes its enough.
1.7% to actually copy the file.
the rest is just wasted to make CPU graphs look pretty.
Re:Failed engineering (Score:5, Funny)
Mod parent redundant ;-) (Score:3, Insightful)
So, basically the GP poster was right: 1% goes to WGA.
Re:Failed engineering (Score:5, Funny)
One thing I don't get is how he managed 41.61% CPU utuilization while transferring a file. Did he have the ethernet equivalent of a winmodem?
No, he had the OS equivalent of a Winmodem.
Re:Failed engineering (Score:4, Funny)
Re: (Score:2, Interesting)
Re: (Score:2)
Re:Failed engineering (Score:5, Informative)
If you're talking about MTU (which is 1500 bytes, not 1.5K), that's the maximum, not the standard.
The average packet size depends on type of network traffic. On most ethernet networks I've managed, average packet size was 700 to 800 bytes.
Re:Failed engineering (Score:5, Insightful)
Furthermore, in the workplace, many people listen to music and access large files on network shares. Clearly, Vista is *broken* for these uses. Not a good indication of Vista being business ready.
Frankly, I don't know why Windows is considered the best business OS. You're much better off with a unixy OS in any environment where gaming isn't important.
Re: (Score:3, Funny)
Uhm....
My *internet* link is 20Mb... So yeah. I won't notice...
Re:Failed engineering (Score:5, Informative)
Okay. But, math doesn't match up with the numbers you typed.
1,500 Bytes is not the average packet, it's the maximum on most ethernet segments. But, the subject original subject is a stressful network environment effecting music playback. 10,000 packets per second is REALLY cranking the data.. so this isn't simple WWW browsing, etc. This is bulk file transfer. So, a large average packet size becomes more realistic in that environment.. say 1400 Bytes.
1,400 Bytes * 10,000 Packets per second = 14,000,000 BYTES / sec = 112,000,000 bits/sec = 112 Mbps
Obviously, that's not even possible on most common home networks, which are 100Mbps. But, an increasing number of people are doing Gig-E at home, in which case 112Mbps is well within the norms for bulk file transfer.
On modern fast multi-core systems, enforcing a pre-set cutoff for packet rate seems like a poor choice. As the linked article showed, the system had plenty of CPU left and didn't need to be throttled at that low a rate. There are also NIC and Driver factors in there.. others might be more or less efficient than the author's equipment -- offload of parts of packet processing and interrupt minimizing techniques can make a big difference.
In any case.. It's easy to say "that's what you get for using MS / Vista". But, really.. that's true in this case. Windows gives you the lowest common denominator. It's designed to be usable with any hardware, by users of any experience level, and to avoid problems by assuming a worst case scenario. So, that's the kind of solution you get given the assumptions MS uses. As we've seen in the Linux world, the solution is to take great pains to build a scheduler that holds up to ridiculous stresses.
Re: (Score:2, Interesting)
Re:Failed engineering (Score:5, Insightful)
Vista's worst engineering decision is to make a system optimized for restrictions and money-farming, not for user experience. The WGA breakdown is the best example. The legitimate users who paid a ridiculous sum to use Vista's 'ultimate' features (you know, the ones which are free in Linux and at least standard in MacOSX) had their systems crippled, and the pirates who bypassed WGA were not even affected. The whole feature does exactly the opposite of what it was supposed to do. That's failed engineering, any way you look at it.
Re: (Score:2)
it's a far better user experience than windows XP. if they did put some DRM related stuff in there, I haven't noticed, nor will 99.99% of its userbase.
Re:Failed engineering (Score:4, Insightful)
it's a far better user experience than windows XP. if they did put some DRM related stuff in there, I haven't noticed, nor will 99.99% of its userbase.
Jesus, have *you* used vista? The user intended user experience could be orgasmic, but I'll be damned if I can get the thing stable given the state of drivers for my vista approved hardware.
In a year it may be better than XP ( and at best, marginally so ), but right now it's hit and miss.
Re:Failed engineering (Score:5, Interesting)
Linux isn't in strong shape on the desktop. It doesn't have the application support it needs, its drivers aren't able to perform as well as their Windows counterparts, which means it's constantly making excuses for not being able to use 100% of the computer its on. But then everyone knows this.
I'd rather have an OS that runs every bit of software I want (including games, video editing, office suites, open source apps, etc.) and may have occasional problems, than one that doesn't run everything I want, and still has occasional problems.
I admire your spirit, though
Re: (Score:2)
Re:Failed engineering (Score:5, Interesting)
Re: (Score:3, Interesting)
And when Microsoft found out people were listening to him... they bought it and hired him to be a Microsoft Advocate, but don't seem to have listened to him ?
The Windows kernel (NT Kernel) was designed by Dave Cutler (ex of DEC) who designe
Re: (Score:3, Insightful)
>you begin to appreciate what Microsoft has accomplished with windows
I've always assumed there's more to it than just "Windows sucks", but I've never had the time to learn about how Windows and Linux work more in-depth so I can meaningfully compare them (nor will I anytime soon).
Care to give an example or two of things Windows gets right?
Okay... (Score:4, Interesting)
Re: (Score:3, Funny)
Slower Network Cards.
Re:Okay... (Score:5, Interesting)
Slower Network Cards.
Re: (Score:2, Informative)
Re:Okay... (Score:5, Insightful)
Winodws XP -- can play an MP3 file and video file at the same time with no reduction in network speed.
Vista -- same computer, same hardware, -- major reduction in network speed.
In other words, Microsoft tried to "fix" something that wasn't broken.
Re: (Score:2, Informative)
In other words, Microsoft tried to "fix" something that wasn't broken
Well, on some machines and in some environments, heavy network traffic can cause an XP machine to slow down, particularly on older/slower hardware. Geeks tend to run stuff that, even if it's not the latest, is amongst the top performers for its generation.
My wife had an e-Machines 1.2 GHz Celeron machine (purchased before we were engaged) with an el cheapo Intel 810 chipset. When she was still running Windows XP, she'd complain all the time about audio dropouts -- I found that these occured during high p
Re:Okay... (Score:5, Informative)
No, in other words, Microsoft/**AA tried controlling something they weren't in control of before.
Where do you want to go today, indeed.
Re: (Score:2, Insightful)
Re: (Score:2)
I was of the impression that it happens with both types. But trying to remember what leads me to believe that I'll have to admit that I obviously only assumed it to be so. So is there anyone who can give us a definite answer to this?
Re: (Score:2)
Re: (Score:2)
Slower Network Cards.
Higher Bus bandwidth...
BillSF
(Troll warning: redundant)
Crappy software aside, 32bit x 33MHz is rather limiting compared to the 64bit x 133MHz standard that has been with us for awhile. IMO, it appears Microsoft has failed to keep up with the hardware. If MS had put less energy in the hoax call
Re: (Score:2)
What "DRM servicing overheads" are you talking about? You don't know really do you? You are just repeating an idiotic meme you heard elsewhere.
The issue occurs even when the user is playing unencrypted content. It has nothing to do with DRM.
Re: (Score:2, Insightful)
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:2)
Because they don't have the Multimedia Class Scheduler Service that Vista has, which ups the thread priority, which in turn causes a throttling of network traffic because heavy network traffic interrupts might disrupt playback to the end user? (Only a bug caused WAY too much throttling) You didn't read the article, did you?
Re:Okay... (Score:4, Interesting)
There are two possible scenarios now:
1. Vista is actually less performant and the inferior system.
2. We're just plain lucky that we get to play MMC on XP and 2k without interruption, and the system throttles network performance on a "just in case" basis. In this case it's a bug that should be fixed.
Re: (Score:2)
No, neither have I, but as the article states, tests by Microsoft showed that very heavy network traffic MIGHT in some circumstances affect media playback on Vista. You often have to decide if you wan
Re: (Score:2)
Oblig (Score:4, Funny)
No problems here (Score:4, Funny)
I can hardly wait (Score:2, Funny)
Dumb dumb dumb (Score:5, Insightful)
Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won't see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you'll see throughput drop to roughly 15%."
That is one of the dumbest things I have heard in a while. Let's see:
What an over-engineered non-solution to what should have been a non-problem in the first place. Microsoft is supposed to employ some of the smartest engineers in the world: can none of them optimise their code?
Not dumb (Score:2)
Do they have 15 year olds designing their operating system?
Re: (Score:2)
But shouldn't I, the user, get to decide what's more important?
Re: (Score:3, Funny)
Apparently not, if you use Microsoft products.
Re:Dumb dumb dumb (Score:5, Insightful)
And this seems like a strange conclusion to jump to...especially coming from Mark.
maybe I am just confused, but the NDIS driver handles sending and receiving of pkts, so is the pkt rate limited to 10,000 pps coming and going? (he mentions packets received by network adapter drivers, but I am still curious). if it is limited to 10,000 pps in either direction...then you the theoretical limit comes down by quite a bit.
Even at that, he is assuming full sized packets, which is a bit of stretch, there is a good chance that not all of them will be the full 1500 bytes, factor in broadcast traffic, and other crud which may be running...and you start seeing a noticable drop even on a 100mbit connection.
And, God forbid, 10Gb (Score:2)
10Gb is starting to get out there too. Granted, it's not likely to be hitting end-user boxes for a while, but you'd think somebody at Microsoft might've stopped to do the math on that one.
Re:Dumb dumb dumb (Score:5, Insightful)
Actually, this is 2007, with stupidly fast processing, memory levels, and network throughput. There's no reason whatsoever that either effect should be showing up when both activities are happening at the same time.
And it's not "slight network performance penalties". It's ridiculously harsh network performance penalties.
Re:Dumb dumb dumb (Score:4, Insightful)
They know why.. it's the kernel-mode encryption required to send audio to the card. There's two engineering failures here:
1. thread locking is accomplished by raising the interrupt level to DPC (KeAcquireSpinLock)
2. Requiring several steps/levels of encryption to interract with the audio card.
The real issue is a combination of utilizing DPC interrupts for basic thread locking (which thrashes the scheduler during long halts) and encryption (which requires long halts).
The real fallacy, IMHO, is thatMS thinks that because it's in kernel mode that it's immune or safer from attacks - so they created lots of "security features" in the kernel. In many ways this makes attacks much simpler - as you can simply move your code into kernel mode which has fewer limits than user mode!
Re: (Score:3, Insightful)
Wouldn't it be better to just let the music skip, as that's most likely the more unimportant and can easily be shut off? Heck, if not listening to music makes the difference between going home on 5 and working for two hours longer I'd happily choose silence.
That's also an assumption as well. Personally, I rarely have any heavy network usage, though it will spike. If I used Vista, I would rather the network be throttled than the music skip. I'm assuming a majority of computer users are not hardcore users and therefore they were the ones catered too. However, you're second point about not knowing its the media player causing the throttling is a valid point. It would have been better to make this throttling mechanism an option as opposed to forcing it on pe
Wow... (Score:5, Insightful)
It is these sorts of things and things like the teams and teams debating the "Shutdown Menu" in Vista that are really showing Microsoft needs to really change if they are going to survive. It amazes me how a bunch of open source developers with all their own agendas do a better job then a bunch of folks all paid by the same company. Of course then there is Apple of an example of a group that shows you can pull it off and still all look like the same organisation.
Funny you should mention that menu (Score:2)
The shut-down menu, that is...
"Bill, Bill! I'm sorry! I didn't mean it! Come back!"
*repents*
Re: (Score:3, Insightful)
Re: (Score:2)
I totally agree. For example, look at the cross-purposes of those working on DRM vs. those working on every other part of the OS.
Re: (Score:2)
no surprise (Score:2)
Russinovich (Score:2)
Re: (Score:3, Interesting)
Have you ever seen a talk by Mark?
While he might be on the Microsoft payroll, he is definitely NOT one to sell-out.
Re: (Score:3, Insightful)
Vista is a turd (Score:5, Interesting)
I gave it an honest chance, but Vista is a turd. If it can't play decades old MP3 technology MS should really give it up.
Completely Unfair Scheduler (Score:5, Funny)
New definition of a Kludge here, I think (Score:4, Insightful)
As goes without saying, arbitrarily throttling one particular task, at some arbitrary level, is the wrong thing.
Perhaps this could go in Wikipedia under "Kludge"?
never trust anyone over 40... (Score:2)
Have you heard anyone under the age of 40 who uses the word 'glitch' to describe undesirable behavior in machinery? Everyone over 40 remembers the modem days so bandwidth issues aren't a big surprise, either. Just another sign that Microsoft is truly the proverbial dinosaur.
(says the guy who just turned forty)
Re: (Score:3, Funny)
Hang on a minute... (Score:5, Insightful)
How short-sighted? (Score:3, Insightful)
Engineering for profit vs. for improvement (Score:3, Interesting)
Suggestion to the Engineers (Score:2)
In the interest of getting a fix out quickly, I have carefully considered the problem and suggest changing 10 to 11 or maybe 12.
Ya hurt yer what? (Score:2)
OK - well that was how those old 6v Beetles worked...
Comment removed (Score:3)
Vista media playback worse than 486 (Score:5, Interesting)
When my old XP HD crashed I was forced to use Vista exclusively for several weeks. It was like my computer was sick and in the hospital. No TV from my ATI x800 All-in-Wonder (though I did get the FM radio working after a week or two), sucky video game frame rates, unstable network card and sound card drivers and crap multimedia playback. P2P kept crashing the network stack.
Some people say that this isn't Microsoft's fault, it's those third party driver writers to blame. I say fuck that, these folks can write good drivers for the exact same computer in several other operating systems. It's Vista's fault.
MS fanboys will all come out and say their systems all work perfectly. Horseshit. I've now had hands on with more than two dozen Vista machines ranging from laptops to upgrades and in every single case, that's 100% MS fanboys, not 99%, not 80%, all of them had stuttering media playback.
There is no excuse for this sort of crap. My goodness it was such a relief to get an XP install back. My computer was perkier and all of a sudden everything worked again.
If Microsoft does not fix this with the mother of all service pack releases rewriting Vista from the core out then my next post-XP os will not be Windows. My best guess is Vista SP1 will be lipstick on a pig rather than the thorough cleaning out that poor excuse for a beta release really needs though.
R.I.P Mark Russinovich (Score:4, Interesting)
His "analysis" here is not much more than a series of rationalizations and excuses:
"Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet's protocol, updates the connection state, finds the receiving application, and copies the received data into the application's buffers." (emphasis mine)
The issue at hand is related to gigE NICs. Please find me a single gigE NIC that does not support TCP/IP checksum offload (even the lowly Realtek does).
His graph showing 40% CPU utilization during a file copy must be a joke or an admission of a dismally performing network stack. There are only 2 possible explanations for that number:
1. His file copy was saturating a 1gigE link - if you've saturated the link, 40% is not great but is decent. However, the test is not applicable to most people who've seen the issue. It also means there is another 60% of the CPU for processing audio - that should be plenty.
2. His file copy was nowhere near saturating the link and Vista's network stack is horribly inefficient. My experience with pervious incarnations of Windows (2K, 2K3 and XP) has shown that under ideal conditions a single file copy will max out (because of inefficiencies in CIFS but that's another story) at ~35MB/s (roughly 1/3 of a gigE link in one direction). If Vista performs at roughly the same rate, then 40% CPU for 35MB/s is terrible. No wonder there is a degradation problem that required network throttling.
Looking down further to the NDIS packet graph, it appears that it is indeed explanation 2 that is correct. Peak throughput through the system was 24.6MB/s (17215*1500). If this test was similar to the CPU test for the previous screenshot, we are seeing 40% for 24.6MB/s. It appears the system will saturate its CPU at 50MB/s half-duplex?!? That's horrible. Or Mark is showing different numbers from different tests. I'm not sure which I want to believe.
Something appears to be very wrong with the network stack in these experiments. I don't have Vista. Can anyone test this?
Disabling MMCSS fixes the issue (Score:3, Informative)
Re: (Score:2, Insightful)
But don't let logic or common sense get in your way.
Re:Well that's just not true (Score:5, Insightful)
However, this actually does make sense. In all honesty, they probably would have worked on a better answer than cutting back on networking, but with the time crunch on releasing it, they probably cut corners here and there (and by probably, i mean definitely and by here and there, i mean everywhere). They probably viewed this as an acceptable cut for the time being because for a majority of users, they use very little of their networking bandwidth. If its just a PC connected to the internet, they'd most likely never notice. The only time this would be an issue is for heavy network usage, which would normally only occur on work-related machines because let's face it, aside from geeks and techies, not many people have systems set up that max out their network bandwidth, so, if they were work-related machines, well, they probably wouldn't be playing that much music to begin with.
I'm not a MS shill, though I don't assume everything they do has evil intentions. We have to admit that they are great code writers, just not the best. Just because they do shady things here and there (mostly in business practices however) doesn't mean everything they do is evil. This was a problem they ran into and they made a workaround that would only affect a relatively small amount of their users. They were probably hoping no one would notice it at all until they either A) had a fix or B) just let it go because maybe no one would notice it.
Remember, this wouldn't really slow down your internet unless you have an *extremely* high bandwidth and even then, bottlenecks on the information before reaching you would probably still mask the problem. This is only an issue on system that have heavy network usage on some sort of intranet or other type of local area network, because these would account for the majority of networks that could even use a decent amount of your possible networking bandwidth.
Re: (Score:2, Interesting)
http://it.slashdot.org/comments.pl?sid=280101&cid= 20366549 [slashdot.org]
http://it.slashdot.org/article.pl?sid=07/08/26/162 8200 [slashdot.org]
http://it.slashdot.org/comments.pl?sid=280101&cid= 20377327 [slashdot.org]
If they are meant to be humourous then my sense of humour must be completely broken. I'm sure there are more comments of equal paranoia to be found in previous installments of this saga.
Re: (Score:2, Funny)
You mean, not at all?
Re: (Score:2)
If you RTFA it explains that it isn't DRM causing the problem.
Re: (Score:3)
Re: (Score:3, Funny)
Re: (Score:3, Insightful)
On Linux, with the CFS and/or SD schedulers, if your nice levels are set correctly, sound (MP3) will play just fine with your processor(s) pegged at 100. Heck, forget about sound; you can run multiple Quake 4s with high-speed LAN transfers in the background, and everything works just fine (network transfers slowdown slightly, Quake 4's FPS scales down linearly with the number of sessions running, but there a