The Problem With the Top500 Supercomputer List 175
angry tapir writes "The Top500 list of supercomputers is dutifully watched by high-performance computing participants and observers, even as they vocally doubt its fidelity to excellence. Many question the use of a single metric — Linpack — to rank the performance of something as mind-bogglingly complex as a supercomputer. During a panel at the SC2010 conference this week in New Orleans, one high-performance-computing vendor executive joked about stringing together 100,000 Android smartphones to get the largest Linpack number, thereby revealing the 'stupidity' of Linpack. While grumbling about Linpack is nothing new, the discontent was pronounced this year as more systems, such as the Tianhe-1A, used GPUs to boost Linpack ratings, in effect gaming the Top500 list."
Fortunately, Sandia National Laboratories is heading an effort to develop a new set of benchmarks. In other supercomputer news, it turns out the Windows-based cluster that lost out to Linux stumbled because of a bug in Microsoft's software package. Several readers have also pointed out that IBM's Blue Gene/Q has taken the top spot in the Green500 for energy efficient supercomputing, while a team of students built the third-place system.
LINPACK isn't so bad (Score:1, Interesting)
How exactly would one perform a sparse matrix product on 100,000 Android cell phones? Linpack isn't great, but it isn't terrible, either. Many supercomputing processor cycles go into large matrix operations--FFT, products, SVD/PCA/Eigenvectors. These can be run in parallel, but they don't break down into trivially parallel operations and the machines on the list really do perform well on them.
A bigger issue is that the rules allow operators to essentially optimize for the specific problem sizes and input, and that the machine doesn't have to be running stably or in the same configuration for all of the runs. It is OK if your machine is overclocked and melts one minute after it executes the benchmark. That isn't useful. The Green 500 is a better metric because it considers the power cost of the computations--that's what most supercomputing centers care about--but even it needs much more stringent rules.
Re:New Benchmark (Score:1, Interesting)
Whatever computer finishes first is clearly the fastest supercomputer
Or has the representation of "infinite" that is the lowest.
Also, I have seen cases where compiler optimization is smart enough to remove the entire loop if there are no side effects to incrementing i, and it's not used outside the loop.
Re:LINPACK isn't so bad (Score:3, Interesting)
I wonder why they don't use EISPACK?
That is for solving Eigen systems.
I remember in the early 1980's writing a program to check my linear algebra homework using Fortran and EISPACK.
This is why I love the fact that Bender likes to drink "Old Fortran" malt liquor.
I have to admit I don't know much about benchmarking but I remember using LINPACK and EISPACK on the VAX and later the Cray YMP.
Re:New Benchmark (Score:5, Interesting)
Also, I have seen cases where compiler optimization is smart enough to remove the entire loop if there are no side effects to incrementing i, and it's not used outside the loop.
Most compilers should be doing this. Hell, even IE9 is supposed to do it for JavaScript now. It gets great scores on SunSpider because of it (the JIT can throw away entire tests).
Re:Heard at Microsoft headquarters: (Score:3, Interesting)
Leela: How is that good news, Professor?
Professor Farnsworth: I still charge enough per seat to be feared.
100,000 smartphones (Score:1, Interesting)
If this phone cluster actually did outperform the fastest supercomputers, why would that make the benchmark stupid?
I mean, the idea of using 100,000 smartphones might be stupid when examined pragmatically, but I don't see how that affects the validity of the performance.
And nobody, so far, is claiming that using GPUs is an inherently stupid idea for any reason, so that should have no bearing on the Tianhe's victory.
When a foreign computer wins, the benchmark needs to be changed? Now that is gaming the system, American style.
Re:Why is being on the the Top500 important? (Score:5, Interesting)
The advantage is that, contrary to the arguments of TFA, the test is very representative of scientific and engeneering problems.
No, it really isn't. I work in HPC at a national lab, and our bureaucrats buy these computers based on these benchmark numbers and then expect us to adapt our codes to fit these machines, rather than buying machines that are better suited to the problems we are solving. For example, one of our machines peaked at #2 on the Top500 list, and was essentially useless for real codes. Another machine of ours held the #1 spot for quite a while, and worked well for a small class of problems, but was so limited in functionality that it couldn't even run many of our codes. I've heard similar stories from people using other machines near the top of the Top500.
Real science codes often do not look anything like LINPACK, and the computers that run these benchmarks fast aren't necessarily good for true HPC.