Speed Test 2: Comparing C++ Compilers On WIndows 132
Nerval's Lobster writes "In a previous posting, developer and programmer Jeff Cogswell compared a few C++ compilers on Linux. Now he's going to perform a similar set of tests for Windows. "Like all things Windows, it can get costly doing C++ development in this environment," he writes. "However, there are a couple notable exceptions" such as free and open-source cygwin, mingW, Express Versions of Visual Studio, and Embacadero. He also matched up the Intel C++ Compiler, Microsoft C++ Compiler, and the Embarcadero C++ 6.70 Compiler. He found some interesting things — for example, Intel's compiler is pretty fast, but its annoying habit of occasionally "calling home" to check licensing information kept throwing off the rests. Read on to see how the compilers matched up in his testing."
Calling home (Score:1, Insightful)
>> its annoying habit of occasionally "calling home" to check licensing information
Calling home for the latest NSA exploits to inject in to your application? /tinfoil-hat-no-so-paranoid-these-days-dept
Re: (Score:2)
Oh, they meant the tests.
Re:Calling home (Score:5, Interesting)
A simple "We sell to corporations, so we don't expect cracks but we do expect optimists trying to install a few extra seats" licensing system really just needs to (ideally over SSL, hear that, LG?) phone home the serial number at intervals and get a 'yes/no' in response.
The more paranoid systems, designed on the assumption that the client will be under active attack, may be considerably murkier. Something like 'BD+', for instance, for Blu-ray DRM, mandates a proprietary, blackbox virtual machine (with access to the client hardware's memory and the ability to run native code, including applying persistent patches to the client system, in addition to what it does internally) that executes whatever blob of BD+ code burned into the disk you try to play.
A system like that, or an AV-like 'signature' system that is designed to have one or more parts of the client DRM continually interrogating one another in novel ways to detect compromises in the authentication mechanism would be more or less indistinguishable from a binary, because it would be one.
For something like a compiler, rather than a game, or some precious 'premium content', I'd expect the former; but I've underestimated human stupidity before.
Re:Calling home (Score:4, Informative)
You might think so, but no. If the licensing check is done in the clear, one could spoof the reply and thus bypass the licensing effortlessly. More likely it's encoded/encrypted in some way that you can't (easily) fake a green light. So you may be able to identify the transmission of "very large binary" or equivalent, but differentiating between a "license check" and "tiny binary" would be tricky since you won't be reading the stream.
Re: (Score:1)
There was a company that sold (or still sells) a blob of software that calculated a registration key based on the configuration of the hardware; serial numbers of the CPU(s), motherboards, GPU boards, installation date of the software, time zone. If any of those changed, you would have to apply for a new registry key.
Re: (Score:2)
That seems like a terrible idea. You'd have many legit claims for new reg keys, how would you know when to deny the requests from people pirating the software? Ugh.
Re: (Score:2)
PKI can fix that for you... allowing the message to be open, but incapable of being spoofed.
Re: (Score:1, Troll)
Who mods this garbage up?
Re: (Score:1)
Sure, it's nonsense. Then Stuxnet happened. When "someone" is investing that much effort into injecting malicious binary code into specific industrial equipment, I think anything is possible if there was sufficient motivation behind it. Ordinary people as targets? Obviously not worth the trouble, so for most of us it's something we can joke about and forget. But let's say someone is compiling programs to run some important industrial equipment along similar lines as Stuxnet's target. Maybe they're doi
Re: (Score:2, Offtopic)
Who mods this garbage up?
You can bet it's not me. I almost get sex more often than Slashdot mod points.
Re: (Score:1)
Who mods this garbage up?
You can bet it's not me. I almost get sex more often than Slashdot mod points.
Damn. I get 5 mod points almost every day. It can be quite exhausting.
A couple of years ago, I was getting 15 mod points daily for a few weeks. Couldn't have taken it much longer...
Re: (Score:2)
>> its annoying habit of occasionally "calling home" to check licensing information
Calling home for the latest NSA exploits to inject in to your application? /tinfoil-hat-no-so-paranoid-these-days-dept
They are also trying to ram down the users' throats a sign-in feature with VS2013 too [wordpress.com].
Crickets... (Score:1)
I do believe that no one on slashdot cares about this.
Re:Crickets... (Score:4, Insightful)
Re: (Score:3)
In the 20+ years I've been coding, I've never once picked a compiler on the basis of how long it takes to spit out a binary.
Then you don't compile big projects. Neither do I, but there are projects where a full build can take hours. Sure build farms and stuff help, but so does a 2x faster compiler.
Re: (Score:2)
I do work on large projects in the multi-hour range for a full rebuild, and the compile time is still pretty much the lowest priority in selecting a compiler. All things being equal, of course I'd like to have the fastest compiles possible. But more important than that is that I can write the code I need to write without dodging compiler bugs / shortcomings all day, and deliver a binary which is optimized well for the target.
You can adapt to slow compiles. Breaking the project up into libraries, for example
Re: (Score:2)
Re: (Score:3)
Iterate twice as fast? Only if all you do is compile. Some of us compile, test some, think, make changes, and then recompile. I heartily recommend the "test" and "think" ideas if you're not used to them.
Re: (Score:1)
Dude, lay off the meth.
Calling home threw off the results? (Score:5, Insightful)
Did calling home really throw off the results? Since that is something that ordinary users would have to put up with, I would think it should be part of the test. It might be difficult to get an average, but testing Intel's compiler only when it is at its fastest doesn't seem fair.
Re: (Score:2)
Does the speed of compiling matter that much compared to the speed of the resulting code? I know that C++ compilers can be extremely slow to run but it shouldn't be a major concern unless it's so slow that an incremental build is wasting enough time that you're off getting another coffee or donut while waiting.
does the Intel one still slow down on AMD systems? (Score:4, Interesting)
does the Intel one still slow down on AMD systems and or trun out code with AMD slow down blocks?
Re:does the Intel one still slow down on AMD syste (Score:5, Interesting)
Re: (Score:3)
My understanding is that they never explicitly 'slowed down' AMD systems; but that the binaries produced by their compiler refused to honor the capabilities flags of non-intel processors
Oh, my. Just how many major non-Intel x86-64 CPU vendors are there? AMD, and...? It's suspiciously similar to the ACPI [slated.org] and SecureBoot affairs, don't you think?
Re: (Score:2)
Re:does the Intel one still slow down on AMD syste (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
Naturally, they wouldn't. It's just liars lying to people not technical enough to catch it.
Re: (Score:1)
The reason there are these feature flags is that, they can be used to identify the capabilities of the processor. It's not supporting processors that aren't theirs, it's supporting the same features in all processors that support the features.
Re: (Score:2)
It's not really, but it is their job not to sandbag them. The ICC isn't a freebie you get with the CPU.
Re: (Score:3)
VIA was also one that was affected by Intels compiler behaviour.
Re: (Score:2)
Back before Intel got (slightly) serious about cheap, with 'atom' and AMD got slightly serious about low-power, with some of their APUs, they made more sense, (in particular, a number of rather interesting x86 embedded specialty boards were VIA based, for situations too low-power or cost constrained for a p3/p4); but lately they've been a much tougher sell. Still some interesting specialty stuff; but 'Unichrome' graphics are such a c
Re: (Score:2, Interesting)
Long, long ago some review site ran a Via CPU based system while spoofing the CPU ID to appear as an Intel CPU of similar capabilities.
They expected a few percent gain in the FP and INT benches, but oddly got an 8-fold increase in reported memory bandwidth. The other benchmarks appeared to reflect a real increase in memory performance.
Don't wipe your arse with Intel, they're so dirty you'll end up shittier.
Re: (Score:1)
Re: (Score:2)
With modest time investment you get almost optimal implementation for multiple instruction sets on any compiler you use.
I'm using ClozureCL and SBCL. I don't think that this is going to work. :-)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
A lot.
Typing this in a AND Phemom II black edition which is very fast and not that far off from an i7 back in 2010 when I purchased this. True the newer ones are slower per ghz sadly.
But what if AMD's next chip kicked ass! Remember the Athlon and the later AthlonXPs were the fastest x86 chip you could buy a decade ago?
Tomshardware would include Skyrim and other Intel compiled apps and whine how slow their inferior AMD chips are and intel fan boys would gleam ... but regardless I have a problem with Intel.I
Re: (Score:2)
But what if AMD's next chip kicked ass! Remember the Athlon and the later AthlonXPs were the fastest x86 chip you could buy a decade ago?
It could theoretically happen, but the Athlon's success was as much about AMD coming up with a decent architecture as it was Intel simultaneously dropping the ball with the Netburst architecture.
Re: (Score:2)
My understanding is that they never explicitly 'slowed down' AMD systems
You are wrong:
"Overview of CPU dispatching in Intel software"
http://www.agner.org/optimize/blog/read.php?i=49#121 [agner.org]
Re: (Score:2)
That is a form of explicitly slowing down and a rather blatant one. Like if someone decides to 'run' the 100 meter by hopping on one foot.
The only way to win is not to play (Score:2)
How good would the Intel compiler have to be at optimizing on AMD processors to avoid accusations that they were deliberately slowing things down?
Re: (Score:2)
You should know that you could get an instant (and large) performance boost by patching the produced binary to always assume it was running on Intel. So I would say not actively sandbagging the performance would be a reasonable start. That is, use the same code path for the same set of relevant feature flags.
Representative benchmarks? (Score:5, Interesting)
Based on his description, he is using a very synthetic benchmark:
The code I’m testing contains no #include directives, and makes use of only standard C++ code. It starts with one class, and then is followed by 6084 small classes derived from various instantiations of the template classes. (So these 6084 classes are technically not templates themselves.) Then I create 6084 instantiations of the original template class, using each of the 6084 classes. The end result is 6084 different template instantiations. Now, obviously in real life we wouldn’t write like that (at least I hope you don’t).
So in his own words, the code does not reflect realistic compiles. There is no reason to assume that the result generalise to any programs that anyone actually cares about.
Also, there are no error bars of any kind listed.
In other words, I have no reason to assign any meaning to these numbers.
Re: (Score:2)
In other words, I have no reason to assign any meaning to these numbers.
Given the reaction to the previous article I don't know what this guy is even trying to do.
And why 6084? What is so special about that number?
Re:Representative benchmarks? (Score:4, Insightful)
In other words, I have no reason to assign any meaning to these numbers.
Given the reaction to the previous article I don't know what this guy is even trying to do.
And why 6084? What is so special about that number?
6084 / 2 % 100 == 42
That is meaning enough.
Re:Representative benchmarks? (Score:4, Insightful)
The article is alright but not one I would use to pick a compiler. IMHO the resulting EXE is more important than the compiler processing time. I've dealt with large sized applications and if structured properly, your build times on a modern computer should not be an issue.
Re: (Score:1)
I think you never had to use cygwin...
Build time on modern computer could be an issue sometimes, but your statement that the performance of the binary is more important keeps valid.
Not that the benchmark of the article makes any sense...
Re: (Score:2)
Its like your average Moronix linux kernel benchmark, where they have bars and numbers with no scale. Basically, look at me, I'm an attention-whore nerd!
Re: (Score:2)
Re: (Score:1)
You build race cars, we build family sedans, mmkay?
Re: (Score:3)
Oh and one minor detail: did you see the final compiled code sizes and how much smaller the optimized versions are (esp. clang!). I'm willing to bet the entire benchmark just code "optimized away" by dead code elimination; and that's an entirely unrealistic situation... Also, where's the code? Is this reproducible?
The benchmark isn't worth anything.
Embarcadero cheap? (Score:3)
I took a quick took at their website. It looks quite scammy, they only talk about how much you will save, not about how much it will cost.
After clicking through the buy-now buttons twice, I found the C++ version was $4000.
Re: (Score:1)
Hmm, I looked at their website and see prices from $999 to $3999. The 64 bit compiler is included at the $999. Looks like all kinds of enterprise database stuff is in the $3999 version.
Re: (Score:2)
Well there were lots of options. I clicked on the one with C++ in it.
Either way, $999 or $3999 is a barrier to me using their products. I could use it in production, but in production I'm going to use the tools that I'm fluent in because they're free and so I get to use them everywhere.
I was questioning TFA, because it implied that Embarcadero was cheap/free. It isn't.
Re: (Score:1)
Quoting a lower ranked answer because I use 64 bit Embarcaderos compiler and find it quite nice.
"Hmm, I looked at their website and see prices from $999 to $3999. The 64 bit compiler is included at the $999. Looks like all kinds of enterprise database stuff is in the $3999 version."
Executable size (Score:2)
LLVM has got to be dynamically linking and stripped by default. There are switches on the other compilers that will let you do that, and it looks like they're being ignored.
Re: (Score:2)
On VC, all debugging information goes into a separate file (.pdb).
Generated code speed (Score:5, Insightful)
This doesn't test the speed of generated code. I like to know which compiler produces faster code when looking at benchmarks.
Inaccurate test. (Score:5, Insightful)
Re:Inaccurate test. (Score:4, Insightful)
while absolutely correct, and not just we put up with it.... if the license check is what the compiler does, then it is what it does. To leave those out is to be measuring something other than the real behaviour of the compiler in real situations.
Hell if this is the case, can you really call the testing complete if he didn't simulate network conditions like, the licensing server being unreachable, or having really high latency?
Re: (Score:2)
There also doesn't seem to be anything about how good the executables the compiler produces are. Y'know, the whole reason for the existence of compilers.
Re: (Score:1)
Compiling millions of lines of code only takes hours if you're using a computer made in the 90s... and that's not even taking partial compiles into account.
Compile time hasn't been a relevant metric for many years. After all, you're only going to compile once, but your code might run millions of times. I'd rather shave 100ms off an execution than 100s of a compile.
Re: (Score:2)
I've worked at places where everything gets built every night to ensure nobody has broken anything.
So, in some cases, you may well compile many many times.
I'm not sure it's how I'd primarily choose a compiler, but in some places it could certainly be a factor.
Re: (Score:1)
And we use continuous integration such that we're always recompiling... but the compile speed is still not a major concern...
Compile speed might batter when you're talking about JavaScript or Java or any other JIT language, but c++? Not so much.
Re: (Score:1)
In the real world, if the number of times it's compiled isn't miniscule compared to the number of times it's run you're doing it wrong.
Re: (Score:1)
Speed is always nice but... (Score:3)
Re: (Score:2, Troll)
Forget C++11 - switch to D. No, I can't do it either, but I can dream. C++11, for all that it has some nice features, continues the endless quest to make C++ ever more baroque, and to give it a syntax that makes the result of an obfuscated code contest look the same as any other code. It can be done so much more cleanly. In fact Walter Bright and Andrei Alexandrescu already have.
One of the interesting things about D is that both Bright and Alexandrescu are serious C++ experts. I don't think Bright decided t
Re: (Score:2)
Re: (Score:2)
Embarcadero's 64 bit compiler uses CLANG 3.1, so it has most of C++11
Re: (Score:2)
Clang 3.1 is prehistoric. Clang is at 3.3 now and really *is* pretty complete C++11. Screw Embarcadero.
Gcc 4.8, which is current, is also pretty complete C++11.
Re: (Score:2)
Wow, a bit of hate on for Embarcadero?
Re: (Score:2)
I don't waste any energy actively hating them. They are irrelevant to me. They tack on a GUI which many find helpful for certain task scenarios to an absolutely free compiler, which isn't even necessarily up to date, and then charge an absurd premium. It's an option.
If I need to do any work in Windows I just use free Cygwin or MinGW.
Re: (Score:1)
VS 2013 which just came out supports C++ 11. At least that is what MS is saying.
Re: (Score:2)
Re: (Score:2)
Did you somehow miss the 'most' in my comment. It has what CLANG 3.1 supports which is quite a chunk.
Re: (Score:2)
I suppose it depends on the importance that one weights those features.
There are three distinctive languages features of C++11 that sold me on using it once and for all: lambdas, constexpr, and user-defined literals. Of these three, Visual Studio 2013 has only one. One out of three isn't "most"
Re: (Score:2)
They have just released a "tech preview" of the compiler that (per the Microsoft provided info) supports constexpr and inheriting constructors with a clear roadmap to supporting the rest of C++11/C++14 (including user defined literals and attributes)
http://blogs.msdn.com/b/vcblog/archive/2013/11/18/announcing-the-visual-c-compiler-november-2013-ctp.aspx [msdn.com] is the announcement from the Visual C++ guys about it.
Invalid Benchmark - Who Cares (Score:4, Insightful)
There are so many things that can affect compile time more than the compiler - and the end customer really doesn't care anyway. Frankly, if you want a 3-5x speedup, just put the whole thing on an SSD and let it fly.
Re: (Score:2)
benchmarking cars based on how long it takes to fill the gas tank.
Electric cars have made that an extremely relevant benchmark... and marketing stunts involving battery swaps have indeed benchmarked how long it takes to fill a tank.
Re: (Score:2)
Electric cars fill their gas tanks very quickly!
Compiler++ (Score:2)
Intel compiler does not phone home for licensing (Score:5, Informative)
The Intel compilers do NOT "phone home" for licensing. What they do "phone home" for is to send anonymous usage data. When you install, you're asked if you want to opt in to this - it is not enabled by default. Licensing is done entirely locally for single-user licenses. See http://software.intel.com/en-us/articles/software-improvement-program [intel.com] for more information.
Posting to cancel moderation. (Score:2)
Posting to cancel moderation.
He measured single-threaded compiles?!!! (Score:2)
Visual C++ has this handy /MP option which tells the compiler to do multi-threaded compiles. On some of our build machines (with 16 cores) this gives an almost linear increase in build speeds. It's obvious from the author's discussion of multi-core that he is not aware of this option and did not use it.
A performance benchmark which doesn't turn on the go-fast option is not going to produce meaningful results.
The author also doesn't discuss debug symbols. VC++ generates debug symbols by default, whereas the
Re: (Score:2)