Forgot your password?
typodupeerror
Wine Software Operating Systems Windows

Wine vs Windows Benchmarks 286

Posted by ScuttleMonkey
from the coming-right-along dept.
PeterBrett writes "Tom Wickline recently posted to the Wine development list announcing that he'd done some benchmarks comparing Windows XP to Wine. They should be taken with the requisite dose of salt, but Wine has certainly come a long way."
This discussion has been archived. No new comments can be posted.

Wine vs Windows Benchmarks

Comments Filter:
  • by creepynut (933825) * <teddy(slashdot).teddybrown@ca> on Tuesday January 31, 2006 @01:37AM (#14604819) Homepage
    We need real benchmarks! Get some Windows worms/viruses/trojans running on WINE and then we'll have some real-world benchmarks!

    I say good day to you sir!
  • on a dev list (Score:5, Insightful)

    by mrcdeckard (810717) on Tuesday January 31, 2006 @01:37AM (#14604826) Homepage
    while i realise that postings to a dev list shouldn't be taken as gospel, why would a dev list posting of benchmarks be assumed to be doctored? of course i would expect this from a marketing dept, but a dev list?

    chris
    • Re:on a dev list (Score:5, Insightful)

      by i kan reed (749298) on Tuesday January 31, 2006 @01:55AM (#14604908) Homepage Journal
      pride. i'm a developer of software, and I don't allow myself to test my own code beyond a certain point because i'll be too proud of my accomplishments to accept mistakes or failures.
      • Re:on a dev list (Score:3, Insightful)

        by njh (24312)
        pride. i'm a developer of software, and I don't allow myself to test my own code beyond a certain point because i'll be too proud of my accomplishments to accept mistakes or failures.

        This is only half the story. Our research group tries to get our bleeding edge algorithms into existing software (e.g. text algorithms in scribus, connector routing and graph layout in inkscape). One thing we've found is that when you are developing some code it's easy to get trained into only trying certain pathways through
        • Re:on a dev list (Score:5, Interesting)

          by hardburn (141468) <hardburn@wumpus- ... OWnet minus city> on Tuesday January 31, 2006 @02:57AM (#14605105)

          Read the article. These aren't dev-created benchmarks, but standard benchmark suites like 3DMark and Quake 3.

          Some of the tests look really weird. For instance, in the 3DMark2000 Fill Rate test, Single Texture on Wine gets 2,402.8 MTexels/s and 11% behind Windows, but on the Multi-Texture test it soars to 6,695.1 MTexels/s and 74.5% in front of Windows. There's got to be some freaky driver code or something implemented oddly or some background process that wasn't noticed.

          I don't think these benchmarks were run rigoriously enough to say anything, except that Wine is capable of running 3DMark.

          • Re:on a dev list (Score:5, Insightful)

            by leuk_he (194174) on Tuesday January 31, 2006 @07:05AM (#14605726) Homepage Journal
            Some of the tests look really weird.

            That is the key phrase(but you have to look literally). The point is that you need to test the output of the benchmarks, not just look at the frame rates. You can create a REAL FAST benchmark by not implementing some api functions. The output might look reasonable, but if you zoom into some edges you might find additional oddities. That is the main point what is mising in this benchmark.

            Rememeber driver writers made some unacceptable shortcuts in the past to increase performance.
    • by SuperBanana (662181) on Tuesday January 31, 2006 @02:55AM (#14605100)
      while i realise that postings to a dev list shouldn't be taken as gospel, why would a dev list posting of benchmarks be assumed to be doctored?

      Nobody said they were doctored; the slashdot editor said "take it with a grain of salt". I see a lot of reasons to do so:

      • there is no sample size (ie, was each benchmark on each platform run 10 times, or just once?) or variance (if it WAS run 10 times- how much did the results vary?)
      • The benchmarks all have wildly different results. Either the benchmarks are that way normally, or WINE (or Linux) is inconsistent. The data is presented such that, again, we have no clue as to the consistency of the results.
      • In a number of the benchmark categories for PC Mark 2004, Linux is less than 1% faster. Usually that kind of difference is thrown in the "statistical anomaly" bucket, but the developer happily gave it the "green" mark, when it should have received a "grey" (ie, "not clear"). If the sub-1% wins had been thrown out, Windows would have won by at least an equal margin.
      • Equal weight was given to the insignificant "wins", as was the massive failures.
      • The developer breaks down the number of Wine failures into 4 categories, but groups Wine successes into one. As a result, it appears Wine is the overall winner, when in fact Wine was slower in 63 cases, and faster in 67.

      Honestly? The results probably aren't manipulated, but the presentation is very clearly set up with a number of tricks (perhaps without him/her realizing it) to give the impression that Wine "kicked some serious ass", when for the most part, it did horribly.

      • As a result, it appears Wine is the overall winner, when in fact Wine was slower in 63 cases, and faster in 67

        Whoops. That should read "overwhelming", not "overall".

      • by VGPowerlord (621254) on Tuesday January 31, 2006 @03:58AM (#14605276)
        Not only are the marks of less than 1% thrown into the green category, so are the 0 difference marks. That's right, Wine is marked as a winner if they perform exactly the same.
        • by mcvos (645701) on Tuesday January 31, 2006 @07:22AM (#14605786)
          That's probably because this is aimed at developers. It shows in which areas they need to improve. Once Wine is equal to XP in a test, they're done, and should focus on other areas. From that point of view, it makes sense to lump all successes in one category, and distinguish between levels of failures.

          This benchmark isn't a Wine vs. XP contest, it's a test to see if Wine is at least as good as XP, and it failed in 81 categories, which means there's still some work to be done.
      • by Anonymous Coward on Tuesday January 31, 2006 @04:05AM (#14605296)
        I think the main reason the 1% less is given a green is because this is targeted at developers of WINE, and seeing that wine is 1% as close as real windows means that that area is "done" being optimized. The areas where wine needs to focus are on the cases where WINE is significantly slower. WINE really only needs to be as fast as windows, not faster.
      • The benchmarks all have wildly different results. Either the benchmarks are that way normally, or WINE (or Linux) is inconsistent. The data is presented such that, again, we have no clue as to the consistency of the results.
        My first guess would be that WINE is inconsistent. Especially in the areas where it falls behind. After all, it is still a beta and has not achieved 100% compatibility yet, so the developers might not care too much about optimization at this point.
        But Linux or even Windows are also possi
      • Don't be silly (Score:5, Insightful)

        by something_wicked_thi (918168) on Tuesday January 31, 2006 @04:54AM (#14605401)
        These results aren't presented to try to make Wine look better, nor is the author, consciously or unconsciously, trying to make it appear faster. These results simply were not meant to be used to say that Wine is better than Windows, and only Slashdot would try to make it appear that way. The real point of these comparisons, as is apparent if you read the Wine weekly summaries, is to give the Wine developers an idea of what areas need to be improved, and what areas are adequate. Green obviously means "at least as fast as Windows", which means that it's good. There is no point in grouping them any other way, since they don't care if they are 50% better or 1% better. Also, your criticisms of why this benchmark doesn't give a good idea of the relative speeds of Wine and Windows are quite wide of the mark (though they are valid complaints). The real reason why this benchmark cannot be used to gauge relative speeds is that it doesn't cover real world work loads. They measure a very small number of very specific things, mostly related to gaming and 3D performance. The benchmarks they ran that weren't related to that were designed to test the *hardware* speed, not the speed of the API. The Wine developers know this, and that's what the comment about taking it with a grain of salt means. It's probably adequate to give a rough idea of what parts of Wine need to be improved, but it is nowhere close to a comprehensive comparison of the speed of Windows and Wine, and was never meant as such.
      • by Savantissimo (893682) * on Tuesday January 31, 2006 @05:11AM (#14605455) Journal
        You forgot the really important issue: in 18 of the tests, some pretty important, Wine didn't complete the test at all.
        • You forgot the really important issue: in 18 of the tests, some pretty important, Wine didn't complete the test at all.


          If I read it right, Wine didn't finish 15 of the tests, and Windows XP didn't finish 3, leading to 18 "no-comparsion" blue results...

          But yeah - that Wine should crash out on a DivX compression or a Web Page Rendering(??!!?) test is ... strange.
      • Some of those benchmarks are not good because wine is good, but because the underlying platform is good - ej virus scanning, I guess that those are good because linux I/O subsystem is good (unless the guy who did the benchmark didn't told the antivirus to scan the same amount of files)

        Then there's basic stuff that you can't explain - why the "CPU speed" benchmark is better under wine? A CPU test will, uh, do things with the CPU, it will be CPU bound and the windows api shouldn't involved in that code path.

        A
        • by Haeleth (414428) on Tuesday January 31, 2006 @08:27AM (#14605968) Journal
          Anyway, I do not care how fast wine is. I care about API compliance. This is 2006, Microsoft has rewritten half of the OS with longhorn and I continue without being able to run many windows apps created years ago. Wine is far from being a true windows replacement for windows apps today....

          I quite agree. Last time I tried Wine it didn't run any of my favourite Windows applications. I'm not talking crappy shareware utilities that I can learn to live without - I'm talking showstoppers like OpenOffice.org, Firefox, and Cygwin, all the really critical tools I use every day.

          Until Wine can adequately run programs like that, I'm sadly going to be stuck using Windows. :(
  • Very Impressive! (Score:5, Informative)

    by gasmonso (929871) on Tuesday January 31, 2006 @01:39AM (#14604833) Homepage

    I've quite impressed with the performance of WINE, however these stats can be a little deceiving. These stats are based on a game that works. Getting the game to work in the first place can be quite a challenge. But for the part-time gamer that doesn't wanna be chained to Windows, this is a great alternative indeed!

    http://religiousfreaks.com/ [religiousfreaks.com]
    • Re:Very Impressive! (Score:3, Interesting)

      by MichaelSmith (789609)
      These stats are based on a game that works

      My sister in law runs ubuntu and I have had a go at getting some windows games running under wine for her son. What I would like to see is a windows environment which she can use to install these things herself.

      As it is I have to mount the CD, find the installer executable and run it under wine. This is a bit difficult to explain to a non technical person.

      • Re:Very Impressive! (Score:2, Informative)

        by harryman100 (631145)
        While you'd have to pay for it - Point2Play (from Transgaming) does exactly that, it allows you to have completely seperated environments and settings for each game/application. I used to use it when I was still finishing off the windows games I had been playing when I switched to linux. Now I only buy linux games.
  • by strider44 (650833) on Tuesday January 31, 2006 @01:41AM (#14604843)
    The results aren't exactly surprising - Wine is excelling in what Linux is generally better than Windows in doing - memory management, hard drive speed, and related matters (stressing generally there, because of course different apps give different results). This is Gentoo after all, it's built for speed. Then the heavier the load on the video drivers the more the superiority of the Windows drivers takes hold, so for the graphical stuff things don't work as fast.

    Congrats to the Wine devs!
  • by shoelace_822695 (789021) on Tuesday January 31, 2006 @01:46AM (#14604869)
    To me this seems to be more a test of the linux implementation of teh video card drivers.. and NOT the wine system itself.

    i think a wider suite of tests would be required.. and not just the preformance/gaming orinted stuff.
    • The fact that the drivers are better is certainly interesting by itself.
    • To me this seems to be more a test of the linux implementation of teh video card drivers.. and NOT the wine system itself.

      Well that is implying that the Nvidia drivers for Windows are much better than for Linux, which is essentially the opposite of my experience with video.

      Obviously graphics is the bottleneck, so perhaps it's a case of WINE not translating the video instructions as well as it does for other instructions.
  • by aussersterne (212916) on Tuesday January 31, 2006 @01:48AM (#14604874) Homepage
    Speaking as someone who used to be a Wine-hater, Wine has definitely come a long way. My impression of Wine for years was that it a) was impossible to install and configure, b) didn't run anything other than solitaire, and c) caused major instability to desktops.

    Then I tried Codeweavers' "Crossover Office," essentially a pre-configured Wine with graphical configuration and installation tools, and everything changed. I currently use all of the following under Fedora Core 4:

    - Microsoft Office XP
    - Wordperfect 12 (word processor only)
    - Photoshop 6
    - Framemaker 7

    They all installed using the standard CD install, without my having to jump through any crazy hoops or type a single command, and they all run flawlessly and are great for serious work. They sit right in my KDE menu like all other applications and it's a real head-turner to be able to show up to work with my laptop running Linux and then pop into Word XP and Framemaker.

    Wine works incredibly well after all, it's just more "raw material" than "finished product." Get someone to write a user-friendly front end for it (ala Codeweavers' Crossover Office) and it offers a very high level of Windows compatibility to Linux users.
  • Notice, however, that the 60 some tests that Wine leads on are synthetic through and through... and when you get to actual games it's XP all the way. While Wine's performance is impressive, the requisite dose of salt may be several kg for this article.
    • Windows XP (and 2k) are better for gaming becuase of all the third part addon's and apps that only work on windows. Technically yes you can can run a game in linux but you cant run many of other programs such as Ventrilo. Ventrilo is a MUST becuase I am in a WoW raiding guild. Right now there is no Ventrilo client for linux.

      Before you go and say "well run Ventrilo under wine!" let me tell you that i have tried and it did not work. I dont know why it did not work and i dont care. Even if it was a relati
      • Again before you guys go off about how Windows doesn't "just work" let me tell you it DOES for me on my gaming machine. My spare time these days is spent raiding instead of tweaking my computer and i prefer it that way.

        I'd much prefer a couple days of tweaks to get a Linux/BSD system up and working properly, rather than spending hour after hour of my life putting up with an operating system that is incredibly slow for absolutely no reason, is terribly unstable, needs to be scanned constantly for viruses and

  • by Anonymous Coward on Tuesday January 31, 2006 @02:12AM (#14604965)
    ...it makes you feel relaxed, slightly fogged and, in sufficient quantities, happily drunk. Windows, on the other hand, just makes you feel angry and frustrated. Give me wine!

    oh, wait, you were discussing software?
  • by Sathias (884801) on Tuesday January 31, 2006 @02:13AM (#14604969)
    Seems a bit strange to me to do a current comparison by using a version of 3d Mark that is 5 years old. If you were going to test out a 6800 on Windows alone you would use 2003 or 2005, the fact they didn't use that one in their Wine comparison suggests to me it couldn't run the later versions at all. The fact that 2000 ran better than under XP, but 2001 ran considerably worse suggests this as well.

    If this is the case, the results in regard to game performance are out-dated at best.
  • Funny statistics (Score:5, Interesting)

    by cd_smith (106365) on Tuesday January 31, 2006 @02:16AM (#14604979) Homepage
    Anyone else notice the funny stuff going on with the statistics? All the statistics are reported as percentages of the XP value, with higher = better. That means that if wine is "+ 90%", it's performing less than twice as fast as XP. But if it's "- 90%", XP is performing ten times faster!

    So whatever this is measuring (and I concur that it seems to be mostly Linux graphics drivers), it's not reporting the results particularly well.
    • by typical (886006)
      That's not unusual -- choosing the incumbent as a baseline is hardly an unreasonable choice if you are trying to do performance comparisons against a challenger.
    • All the statistics are reported as percentages of the XP value, with higher = better. That means that if wine is "+ 90%", it's performing less than twice as fast as XP. But if it's "- 90%", XP is performing ten times faster!

      That sort of confusion occurs whenever percentages are used. The solution would be to take that same data, and compare the log of the ratios. Or if we follow what's typical in audio and electronics, take ten times the log of the ratio. That's the sort of comparison that's going on whe
  • Wine is significantly slower in nearly half of the tests. And getting faster results during memory and CPU tests don't make any sense. The OS shouldn't have anything to do with the results of these tests. Maybe the results are skewed by the Wine's timer implementations?
    • Presumably, the memory tests deal with the various OS-level Alloc's (HeapAlloc, GlobalAlloc, LocalAlloc, VirtualAlloc, etc...), which include fault protection checking, SACL checking, and other safety features. The reason that Wine performs better is that either they have implemented a faster version of the WinXX memory management APIs, or that the underlying Linux memory management is faster and the cost of the Wine wrapping calls is negligible. Same for the CPU-related tests... Just as memory is a managed
  • by phorm (591458) on Tuesday January 31, 2006 @02:23AM (#14605003) Journal
    Well, my experience with Wine/Cedega has been that for the games and applications that work, the disk-access tends to be faster. Not necessarily because the actual disk is being accessed faster, but because the filesystem (in my case reiserfs) is speedier to read.

    The other wonderful thing I've found about Wine is the graphics abstraction layer. My laptop has a GeForce FX5600 (mobile) card in it. It's actually rather spiffy for most games still, but sucked ass at Battlefield 2 in windows, popping up the warning that my graphics drivers were out of date. Well, it seems that the drivers are tied to the laptop in windows to co-habitate with the power-saving etc etc... so I couldn't update from the official NVidia ones. And of course, my laptop vendor doesn't offer updates for anything over a year old it seems.

    In linux, however, the normal NVidia accelerated driver works. The game runs on that faster than in windows, and with better detail levels. I don't know if it's just that the Cedega HAL does a better emulation for the software bits, or if it's due to the more-up-to-date driver, but it's a much less painful experience in Linux.

    Lastly, my soundcard. SB Live 5.1. Abit dated, but with livedrive still a very nice functional card, except that the windows drivers will eventually/randomly freeze in most directX intensive games. Running in linux... no problemo. That's actually why I switched to Cedega/Debian almost completely (too many losses in Warcraft from lockups).
  • by Chrax (782154) on Tuesday January 31, 2006 @02:28AM (#14605018)
    The real question on everybody's mind is: how well does it run Counter-Strike 1.5? I didn't see that test on there.
    • The real question on everybody's mind is: how well does it run Counter-Strike 1.5? I didn't see that test on there.

      I agree. It's the only reason I have a Windows partition. Can people please post any Wine vs XP benchmarks they have, along with their machine specs! Also for CS:Source, if anyone has it working. Thanks.

      Phillip.
  • DirectLinux (Score:2, Insightful)

    by MrNybbles (618800)
    The Linux Kernel has some graphics support but neither the Kernel nor the X Window System are geared for fast 3D graphics. DirectX is good at getting around using the slower Windows GDI. DirectX is one of the few things Microsoft does somewhat well. (Insert joke here about Directx 9 and taking nine trys to get DirectX right.)

    I have a feeling that unless some major changes are made to the X Window System (and maybe Linux drivers) that WINE will not catch up with WindowsXP and DirectX, but that just means I w
    • DirectX was built for two purposes in the first place: vendor-lock-in and to avoid paying to SGI. Microsoft stopped OpenGL development by interfering with the OpenGL ARB, in order to catch up with their own solution, DirectX.

      Yes. Vendor-lock-in is what Microsoft generally does well.

      • by TheNetAvenger (624455) on Tuesday January 31, 2006 @07:08AM (#14605735)
        SGI. Microsoft stopped OpenGL development by interfering with the OpenGL ARB, in order to catch up with their own solution, DirectX.

        Love the 'personal' theories...

        Microsoft did not even have an alternative to OpenGL in development when Microsoft pulled out of OpenGL. Microsoft pressed for OpenGL to enhance low level hardware support with intention of doing more than cad/engineering and supporting 3D rendering environment conducive to gaming and directly access video card hardware for gaming.

        OpenGL told Microsoft to go pound sand, and that OpenGL was not for games or going to support direct hardware features for gaming.

        Microsoft started stringing together a set of technologies that were called WinG, mainly a 2D form of rendering with plans for a new model that was a 3D rendering solution with direct video access on par of what the current DOS based games were used to, but in the Windows environment.

        If OpenGL would have not 'played catch-up' to DirectX, and instead took Microsoft's recommendations at the time Microsoft was a big OpenGL proponent, there would never have been a DirectX, as OpenGL would be what Microsoft would be using, and contributing to instead.

        The vendor lock in, was just a bonus in the long run, it was others involved in OpenGL that made the choice to not go for gaming.

        But you can say it was about paying to SGI or a diabolical plan to take control of the gaming industry, but the facts don't support it.

        The second part of this topic is that DirectX evolved to be more than an alternative to OpenGL, as it encapsulates everything from input devices, networking, to sound and voice.

        When DirectX first existed it was the only game in town for any standardized interface to video for accelerated graphics in gaming. Now it is more than just Video...

  • What's more impressive is that he managed to get 3DMark 2001 working at all under Wine!

  • transgaming (Score:2, Interesting)

    As a longtime transgaming subscriber, I can tell you that wine really does work as well as pictured. However, it uses an absolutely offensive amount of ram while it does so. I don't knwo how closely related the branches are though.
  • by Anonymous Coward on Tuesday January 31, 2006 @03:50AM (#14605251)
    It does not take a second look to figure out the stats made up.

    1. Most of Wine's wins are in the 0-2% mark. Means nothing except _inconclusive_; otherwise where is the variance, num tests to justify this?
    2. Wine's perf is bad in the tests it lost
    3. Old test suites were used
    4. As some one said, If Wine is 90% faster it means it is 90% faster. If it is 90% slower it means it is 10 times slower!!!

    BUT, what is really impressive is that Wine actually managed to run all the tests. The compatibility is indeed impressive. This benchmark would have been very credible had it not played with the numbers and colors.

    Maybe a troll, but here is my argument against Wine:
    Windows is moving to WinFX. Then it makes more sense to emulate WinFX's API than Win32 API. (WinFX does use Win32 extensively underneath, but why emulate 2 API's??). In the longer term, the answer to Windows compatibility is not Wine, it is MONO [mono-project.com].
    • BUT, what is really impressive is that Wine actually managed to run all the tests. The compatibility is indeed impressive. This benchmark would have been very credible had it not played with the numbers and colors.

      It didn't manage to run all the tests. FTA:

      Wine or XP aborted on 18 tests

      The breakdown for that is 3 for Windows and 15 for Wine.

    • by n0dalus (807994) on Tuesday January 31, 2006 @10:45AM (#14606624) Journal
      This benchmark would have been very credible had it not played with the numbers and colors.

      This test was never intended to show up on sites like Slashdot. The page was made with Wine's developers in mind to have a place to watch performance differences between wine versions. Nobody is trying to say Wine is better than Windows. It's not supposed to be a 'credible benchmark' for the purposes some of you are using it. The main idea behind it is so that in future versions of wine we can run these tests again and see how the results changed. How we represent the numbers is not important. What's important to us is how the numbers change over time.

      To reiterate, this benchmark is really for comparing versions of wine against other versions of wine; it is not intended to be a good or thorough comparison between wine and Windows.
  • what drivers where used? and what driver and OS configurations where used? and what about features? since a product has less features, it should have less to process, hence it running faster. how fast can windows 95 paint a window to the screen compared to windows xp? ya, i know, its a lame example, but its true.
  • The tests are ok. Sure we need more (like later versions of software, assuming that it runs of course), but at least it's a start. I look at all the green's as a "these bits work ok" and anything worse than yellow as "this needs a lot of work". Beyond that, I wouldn't read anything else into it.

    We also need to know what version of the NVidia driver was used, on XP and on Linux, as this will make a huge difference. It'd also be good to know some other stuff about each setup (eg: DirectX version and patch lev
  • On related news... (Score:3, Informative)

    by Anonymous Coward on Tuesday January 31, 2006 @04:54AM (#14605399)
    The latest LugRadio show ( http://lugradio.org/episodes/43 [lugradio.org] ) features a very interesting interview of Jeremy White about Wine.
  • almost classic (Score:2, Insightful)

    by dchallender (877575)
    Looking at the results - "Grammer Check"
    Shame it was not "Spelling Check" but on still quite amusing.

    In all seriousness, interesting and makes me want to revist Wine as it looks a lot better than when I last tried it (given I run pretty low spec hardware, performance is key rather than stability).

    Though I do think "Wine or XP aborted on 18 tests" was a bit cheeky as it was 3 XP aborts, 15 Wine aborts...

The reason why worry kills more people than work is that more people worry than work.

Working...