The Problem With the Top500 Supercomputer List 175
angry tapir writes "The Top500 list of supercomputers is dutifully watched by high-performance computing participants and observers, even as they vocally doubt its fidelity to excellence. Many question the use of a single metric — Linpack — to rank the performance of something as mind-bogglingly complex as a supercomputer. During a panel at the SC2010 conference this week in New Orleans, one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number, thereby revealing the 'stupidity' of Linpack. While grumbling about Linpack is nothing new, the discontent was pronounced this year as more systems, such as the Tianhe-1A, used GPUs to boost Linpack ratings, in effect gaming the Top500 list."
Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks. In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package. Several readers have also pointed out that IBM's Blue Gene/Q has taken the top spot in the Green500 for energy efficient supercomputing, while a team of students built the third-place system.
Re:New Benchmark (Score:3, Informative)
Right. (Less than symbol didn't show up because I didn't choose plain text! Derr)
Good to hear (Score:5, Informative)
The Top500 has the problem in that many of the systems on there aren't super computers, they are clusters. Now clusters are all well and good. There's lots of shit clusters do well, and if your application is one of them then by all means build and use a cluster. However they aren't supercomputers. What makes supercomputers "super" is their unified memory. A real supercomputer has high speed interconnects that allow direct memory access (non-uniform with respect to time but still) by CPUs to all the memory in the system. This is needed in situation where you have calculations that are highly interdependent, like particle physics simulations.
So while you might find a $10,000,000 cluster gives you similar performance to a $50,000,000 supercomputer on Linpack, or other benchmark that is very distributed and doesn't rely on a lot of inter-node communication, you would find it falls flat when given certain tasks.
If we want to have a cluster rating as well that's cool, but a supercomputer benchmark should be better focused on the tasks that make owning an actual supercomputer worth it. They are out there, that's why people continue to buy them.
You missed the point too, btw (Score:3, Informative)
The actual benchmark does stress interconnects (Score:5, Informative)
Yes, noticed that.
Here's the actual benchmark used for Top500: [netlib.org] "HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers". It solves linear equations spread across a cluster. The clustered machines have to communicate at a high rate, using MPI 1.1 message passing, to run this program. See this discussion of how the algorithm is parallelized. [netlib.org] You can't run this on a set of machines that don't talk much, like "Folding@home" or like cryptanalysis problems.
Linpack is a reasonable approximation of computational fluid dynamics and structural analysis performance. Those are problems that are broken up into cells, with communication between machines about what's happening at the cell boundaries. Those are also the problems for which governments spend money on supercomputers. (The private market for supercomputers is very small.)
So, quit whining. China built the biggest one. Why not? They have more cash right now.
Re:Good to hear (Score:1, Informative)
What year did you last look at clusters? The high speed interconnects used now are sufficient for particle simulations; I know, I've written and on one of those clusters near the top of the list. Contrary to your comment, NUMA has pretty much become the standard for high end supercomputers whereas shared memory supercomputers are relatively rare. This is for several reasons, not least of which are: top-ranked supercomputers are rarely dedicated to a single customer, single customers of them rarely have unchanging resource needs, and that customers of supercomputing facilities don't want to rewrite their algorithms if at all possible and therefore stick with the standards they know (e.g., OpenMP or MPI). Finally, even many single-machine supercomputers are NUMA because after a certain point, DMA across the entire machine stops making sense for almost any application/use case.
Re:Well, there's a non-notable point! (Score:2, Informative)
In this case it was the benchmark software that was buggy, not the OS.
Yeah that's right, the LINPACK benchmark software that Microsoft strangely got to rewrite themselves. Yep that's apparently where the bug was.
I wonder why MS was given the task of rewriting the benchmark in the first place. I guess it will always be a mystery... oh hold on, no wait... "It should be noted that Microsoft has helped fund the Tokyo Institute of Technology's supercomputing programs." Guess it helps to read the sentences they stick near the end of unrelated paragraphs at the end of the articles.
Re:Quelle surprise! (Score:4, Informative)
That makes a nice headline, but everything the article is based on has been proven to be untrue and sensationalist. My 8 year old son, when he lost, used to also accuse others of cheating as well. Usually he was wrong as well, but I didn't take his word for it and then try to pass off a news article on it.
It was a Microsoft bug in the supercomputer (Score:2, Informative)
That is incorrect. It was not in the software designed to do the rating. It was in the Microsoft designed software that ran on their OS, and that was rated. If I need to perform a task, and I pay a company to provide the hardware and software to achieve that task, I don't really care if it was a bug in their OS or their application that reduces my productivity.
It is true that the bug wasn't in Windows OS, but it was still a bug in the supercomputer implementation, and therefore a Microsoft flaw in the supercomputers performance, and not a flaw in the benchmarking tools or methodology.
Re:New Benchmark (Score:2, Informative)
Also, I have seen cases where compiler optimization is smart enough to remove
the entire loop if there are no side effects to incrementing i, and it's not
used outside the loop.
Most compilers should be doing this. Hell, even IE9 is supposed to do it for JavaScript now. It gets great scores on SunSpider because of it (the JIT can throw away entire tests).
Hey mods, I think the parent was going for funny.
The IE9 Javascript "optimization" is completely invalid because it eliminates branches that have side effects hidden by valueof().
For example, you can define a global valueof() immediately prior to calling the function. If the branch was previously eliminated by an optimizer, then the valueof() calls will not be made; therefore the final result of the computation will be incorrect.
GPUs don't work well (Score:1, Informative)
The Chinese computer with GPUs works with around 40% efficiency because of communication overhead and wasted SIMD ways, whereas most winners historically worked with around 90% (The inefficiency of 100k handsets would be even higher than the lameness of the joke). The silicon power-wall is the limit in scaling of the supercomputers that are built with GPUs.
So, giving examples on GPUs and handsets is not a good way to promote the use of other benchmarks because even this simple benchmark is enough to fail them (assuming your concern power in year 2010 and existence in year 2020). However, it would be a motivation to change this benchmark if there was proof that it is not a representative set for the use of these machines.
Re:Why is being on the the Top500 important? (Score:3, Informative)
Codes? Is that what internets are programmed with?
I know you're just being sarcastic, but the HPC equivalent of a program or an application is a code. Google "hpc codes" or "multiphysics codes" for some examples. And for some trivia: the input script for a code is typically called a deck, a term that's been around since the days when the input was handed over to computer operators as decks of punch cards.
Re:Quelle surprise! (Score:2, Informative)
Blurry is frustrating, not relaxing. Unless you're talking about the relaxing man-made waterfalls of semen.
What I would like to have is some Japanese porn where the actresses don't sound like a cat was set on fire. What the fuck is wrong with these people that they make those kinds of sex noises?