Ars Analysis Calls Windows 7 Memory Usage Claims "Scaremongering" 334
Posted
by
timothy
from the nothing-but-dihydrogen-monoxide-to-drink dept.
from the nothing-but-dihydrogen-monoxide-to-drink dept.
Via newsycombinator comes a reaction at Ars Technica to the recently reported claims of excessive memory use on machines running Windows 7. From the article: "I installed the XPnet performance monitoring tool and waited for it to upload my data to see what it might be complaining about. The cause of the problem was immediately apparent. It's no secret that Windows 7, just like Windows Vista before it, includes aggressive disk caching. The SuperFetch technology causes Windows to preload certain data if the OS detects that it is used regularly, even if there is no specific need for it at any given moment. Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory."
Re:So (Score:3, Insightful)
It is basically.
You can even (trivially) roll your own in Linux. lsof occasionally to build some stats on commonly opened files. cat them to /dev/null to fill the filesystem buffer cache. I'm not sure anybody would even bother to give it a name like "SuperFetch", never mind a trademark like "Microsoft Windows SuperFetch®". It's kind of sad and depressing really.
Re:So (Score:5, Insightful)
More to the point, the company that wrote this little monitoring tool badly misunderstood basic principles of how the operating system works. At this point, I think we can move on and completely disregard any conclusion they came to. It either demonstrated profound ignorance or a deliberate attempt to mislead people it what turned out to be a slashvertisement of their products and company.
From the article:
One might almost think that this whole exercise was simply a cynical ploy. Allegations of Microsoft bloatware are, of course, nothing new, and oblique references to the old canard that what Intel gives, Microsoft takes away does nothing to dispel the impression that this is another case of Microsoft bashing.
What a surprise. Fortunately, people really didn't even let them get away with it even in the previous article. Microsoft deserves plenty of what slashdot slings its way, but let's stick try sticking to facts.
Re:It's a matter of definitions (Score:5, Insightful)
Re:It's a matter of definitions (Score:5, Insightful)
Windows 7 is sucking up your system memory to make Windows appear faster.
So windows has a feature which makes everything run faster and yet it only "makes Windows appear faster", instead of making it actually faster?
It seems to me that windows is only using hardware in a rather intelligent way: if it's not being otherwise used or needed, it uses it to boost performance.
Linux does the same thing, as far as I know, and you don't see anybody calling Linux a memory hogging OS.
Re:It's a matter of definitions (Score:0, Insightful)
I swear... you people are fucking retards.
Re:So (Score:3, Insightful)
No disk activity is required to flush read buffer/caches. A little might be needed to flush write cache but read cache should be flushed first if memory is required by programs.
Anyway, most of the cache is read cache since write cache are flushed at regular interval.
Short story, no, no disk activity is required.
You will only have a lot of swapping if you do not have enough memory to run your programs, not because of buffer/cache usage. Before you start to have a lot of swap activity, your buffer/cache usage will go to zero anyway ;-))
Basically, if you do not have any buffer/cache usage on your system, it is probably because your are tight on memory.
Re:It's a matter of definitions (Score:3, Insightful)
This is one of those great mysteries of life (Score:5, Insightful)
if everyone is so afraid of their computer memory being used to the fullest, why do these people install so much of it?
I've got 8GB of ram in the machine I'm on at the moment, and I want the OS and applications to use it to the fullest and most efficient extent possible at all times. I didn't install a 64-bit OS and 8GB of ram so that I can see 6GB free at all times.
Re:Secondary problem - spinning up HDD (Score:1, Insightful)
There's so much s**t going on in the background on every Windows machine that the HDD will never spin down even with SuperFetch disabled. Practically every program writes to the registry every few minutes...
Too much? (Score:3, Insightful)
If Windows refused to use your RAM that you had installed, now that would be an issue. But fully using RAM? This, on its own, is not something to complain about.
You're missing the point (Score:2, Insightful)
Re:So (Score:3, Insightful)
"those who fail to understand Unix are doomed to re-implement it...poorly."
lol.. like Linux "re-implemented" UNIX? Unless ofcource they used the same code, then the SCO lawsuit could be revived again ;)
They understand UNIX and re implemented it appropriately, not poorly.
Re:Tits on a bull (Score:3, Insightful)
fflush() just flushes stdio's buffer, so that any data written to the file is sent to the operating system (via write() on *nix), making it visible to other programs. It does not flush anything in the operating system's own buffers/caches. Also, fclose() calls fflush() automatically, so calling both is redundant.
You're probably thinking of fsync(), a system call that does actually force data to disk. And should almost never be used, unless you enjoy waiting through several seconds of disk grinding and general unresponsiveness.
In theory (Score:2, Insightful)
A good OS uses all the RAM, and allocates available free blocks of RAM to the programs as required.
However using the greater part of a gigabyte plus paging to the hard drive just to display the desktop and run the low level functions is inexcusable and points to either a) memory leak b) the OS is doing something legitimate you are unaware of, like indexing files, etc c) the OS is doing something illegitimate like sending the contents of your hard drive to someone in Redmond, the NSA/FBI or the RIAA/MPAA or d) your system has been compromised by a virus. I can't really think of any other possibility.
Re:So (Score:5, Insightful)
The vast, vast, vast, vast majority of that cached memory is read-only caches (like DLL caching and superfetch) which doesn't need to be "dumped". Some small, very small, portion of it is read/write disk cache, but that portion is never going to be dumped unless you're *completely* out of memory otherwise. And that's basically a "last resort failure mode" at that point.
You're as bad as the guys who wrote that article in the first place. If you don't know how Windows works, please don't talk about it.
Re:So (Score:5, Insightful)
You have no idea what you're talking about. Superfetch is much more than stats on commonly opened files. It takes into account the times of the day, weekends etc. too among other advanced stats. Anyway if it's trivial to roll your own, why doesn't such a thing run by default in Ubuntu? That's the thing that's sad and depressing, not giving names to technology.
Re:So (Score:5, Insightful)
To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).
The problem with previous versions of windows (I haven't used anything newer than XP) is in how the OS decides that you do not "need, use or have a purpose for" certain types of memory.
The pathological, and yet all too common case with XP is the OS's decision that text pages should be dumped in favor of disk cache far too soon. The result being that if you have multiple apps open and a few that you haven't touched for roughly 10 minutes and then go to copy a couple of gigabytes of files around the text pages for those 'idle' applications are flushed out and the disk cache loaded with parts of those copied files (which you are unlikely to ever need). When you click on the iconbar to bring one of those formerly idle apps back to the foreground the system grinds away for a long time (obviously machine dependent but never instantly and frequently way beyond the point of annoying) as it reloads those text pages from disk before the application even starts to redraw itself much less starts becoming fully interactive again.
The worst part about that behavior is that, to the best of my knowledge, there are no knobs to tweak it. I can't specify how long a text page needs to be idle before it should be a candidate for flushing or even if it should be pinned down permanently so that is never paged out. I once went looking to see if there was a way to do it from within the application code itself - something like mlock()/mlockall() in posix - and I couldn't find an equivalent, which may just be a reflection of my own inexperience with the Windows API but I figured I would throw that out there anyway.
Re:So (Score:3, Insightful)
One of the main tricks, which you're completely glossing over, is the algorithm that determines what files are likely to be wanted/needed at any particular time of day. Sure, lsof will give you the raw data on what's open when, but making something useful out of that it a touch trickier than you suggest.
Additionally - and forgive my lack of Linux kernel expertise; I never thought to check this - is there a limit on the size of the filesystem cache? If it's less than a couple gigabytes, you're not going to be able to pre-load some of the files that most need it. If it is that large, presumably the ability to re-allocate it to a user-space program near instantly is also present, but that would be my next question. Finally, is the cache FiFo? If so, you've got a problem - unless your users are literally as regular as clockwork, they aren't going to want program X, and nothing but program X, at exactly time Y every day. Therefore, to ensure it's pre-loaded for them, you have to cache it in advance. However, if there's a lot of filesystem access - even small everyday stuff, like browsing the web or looking through a photo collection - the constant disk I/O will shove a lot of your pre-cached data out of the buffer.
Superfetch doesn't *sound* terribly impressive, but more thought went into it than you might realize.I'm certainly not saying it can't be duplicated on Linux, but there's obviously a reason it hasn't been, and it's obviously not because the feature is useless. I'm not sure how relevant it is to servers, but from a user perspective, the difference between five seconds and under one second to open a particularly large program is extremely noticeable. Given enough RAM (4GB on my system, of which Superfetch typically is using upwards of 3) you can easily achieve speedups like that.
Re:It's a matter of definitions (Score:3, Insightful)
Actually, even XP had a disk cashe (like all the other OSes you mention, it was post-caching - storing data in case you need it again soon, rather than pre-caching things that you are probably about to use). However, as I understand it, the disk cache in XP was relatively small, and XP was (at release) used extensively on machines that really didn't have the kind of RAM it wanted. Therefore, it implemented an extremely aggressive paging algorithm, constantly writing memory pages to disk but not recovering them until it absolutely had to (even if there was un-allocated RAM). Not only did this increase I/O, but it meant that switching out of one long-running application (which has been allocating and freeing memory for so long that *everything* else got paged out) takes ages.
The up-side of this is that the main running program still looks pretty fast, even on a low-memory computer. That was the goal for XP. Unfortunately, it also makes XP hopelessly bad at managing systems with the kindof RAM you see today - it will still super-aggressively swap out memory pages for inactive applications, even if it could leave stuff in main memory and just re-use a portion of the RAM. Once that RAM is freed - perhaps because the program is closed - it lacks the sense to fill it up again with cached data that you might want soon (quite possibly including chunks of the pagefile holding data for one of the background processes the user is just about to click on and would rather see activate without 4 seconds of delay to pull paged memory in again).
That said, your point stands. XP's memory management is terrible for modern systems, and aside from the difference between pre- and post-caching, all other major OSes are "memory hogs" in this manner too. Unused RAM is wasted RAM.
No simple answers (Score:3, Insightful)
The issue is not a simple one.
On the one hand it's potentially a good idea to use "unneeded" RAM to pre-fetch possibly useful in the future disk data.
But a whole lot of apps were written without that "feature", so a lot of them already pre-fetch data. Now you have twice the amount of RAM tied up, for no benefit.
And nobody can predict the future, so the pre-fetching is speculative at best, and has no way to compensate for other tasks the user may double-click into competion with ones being pre-fetched.
Worse yet, a lot of apps look at the amount of free RAM when they decide whether to prune their working set or make other cache/purge decisions.
If all of a sudden the OS reports less free RAM, all those apps are going to make the wrong decision.
Re:So (Score:3, Insightful)
Re:So (Score:2, Insightful)
You never found a use for wasting less time? Seriously?
Re:So (Score:2, Insightful)
Re:30ms? (Score:4, Insightful)
I didn't make a snide remark. I pointed out that windows maintains a pool of zeroed ram, and will constantly fill it when it's able to.
Windows will allocate from that pool first, and then top it back up when it hits a threshold of low free memory from supercache.
It then has the luxury of handling this in the background, so you're not waiting for the system to zero memory out, unless you somehow manage to totally max out the memory you've got in use.
It's at this point when you've got bigger problems, like the fact that the disk will probably become your bottleneck, not the speed at which you can zero out ram.
I didn't miss your point, I'm suggesting that you're not thinking like an operating system designer, and saying "how can i shorten the critical path of giving a process the memory it requests?" The answer is: Pre-zero memory, and fill the zeroed pool from the cache as needed when you get low. There's no "wait" for zeroed out memory, at this point.
Re:So (Score:1, Insightful)
If they don't have to worry about it, then it's not a problem.