Cell-based "Roadrunner" Tops Elusive Petaflop Mark

prunedude writes "The NY times is reporting that an American military supercomputer, assembled from components originally designed for video game machines, is more than twice as fast as the previous fastest supercomputer, the I.B.M. BlueGene/L. To put the performance of the machine in perspective, Thomas P. D'Agostino, the administrator of the National Nuclear Security Administration, said that if all six billion people on earth used hand calculators and performed calculations 24 hours a day and seven days a week, it would take them 46 years to do what the Roadrunner can in one day."

Summary should have a shout out

[HolyCoitus]

1350 IBM Linux cluster team. xCAT for pwning.

Re:Question

[avalys]

It will be used for nuclear weapons simulations - primarily for investigating issues related to how warheads will perform as they age.

Re:Question

[Anthony Rosequist]

Before it does weapons simulations, it will first work on some scientific problems, like model testing to predict climate change.

After it's done with that (I wonder how they will determine what done is...), it will go classified and do nuke simulations.

Change in paradigm

[karvind]

If one looks at http://www.top500.org/ [top500.org] list and compare the CPU frequencies of the top supercomputers - all BlueGene CPUs were running at less than a GHz. And it seemed those low power cores were key to HPC (high performance computing). Cell and opteron - both run at multiple GHz and (presumably consume more power). IBM still has next generation of BlueGene/Q in works and is also for +Petaflop computation.

Re:exaflop, zettaflop, the yottaflop and the xeraf

[DogDude]

Military taking the lead on computing as usual. Why is the military so much more progressive (with practical results) than any other institution of government?

Are you kidding? [warresisters.org]

Re:err

[Anonymous Coward]

It was designed originally for the PS3. But not solely for it.

Cell was the brainchild of Ken Kutagari of Sony and Peter Hofstee of IBM.

Re:The future

[Thalin]

This is actually based on Cell 2 or as IBM marketing likes to say it "Cell eXtreme"!

Cell 1 (the Playstation chip) didn't have the double precision floating performance to achieve the petaflop mark; Cell 2 is far better on that front.

Yes, It Does Run Linux

[Doc Ruby]

As I posted the last time this story was reported (in IBM Touts Supercomputers for Enterprise [slashdot.org]") in "Yes, It Does Run Linux" [slashdot.org]:

From IBM's detailed press release [ibm.com]:

the QS22 boasts an open environment, utilizing the flexibility of Red Hat Enterprise Linux as the primary operating system and the open development environment of Eclipse.

That means that a PS3 running Linux [psubuntu.com], even with its ridiculously low 512MB RAM, can be used as a $500 development platform for these CellBE BladeServers.

And, in turn, some QS22 SW might be usable on the PS3, if it can be ported to use the tiny RAM. Or if someone hooks an i-RAM bank to the SATA port as swap/ramdisk, using perhaps iSCSI over its Gb-e for storage.

Now get out there and supercompute!

Old News

[MBHkewl]

This was covered last year, and the Los Alamos website [lanl.gov] had a few interviews with some people involved on what the uses of Roadrunner are. They had a time-line of what phases are to be done, and as far as memory serves me, they were going with Opterons for the first phase, then performance assessment, then add the Cell processors in the third phase.

From these pictures [lanl.gov], it clearly shows they're using IBM Blades (4 chassis in each rack), and IBM already offers BladeQ [ibm.com] servers which use Cell processors for HPC applications. The IBM BladeQ servers pack double the CPUs of a PS3.

If you take a look at the Folding@Home project statistics [stanford.edu], you can see the performance of PS3 boxes, and almost relate...

Re:Not in perspective - this is a media number

[jareds]

How fast it could break the 128 bit encryption used when you log onto your bank's web page to pay your bills (this might also be understandable and would probably be a bit scary)

No, not at all scary. It's apparently twice is fast as the BlueGene/L, which apparently set a record of 478.2 teraFLOPS. Let's assume it takes 1 floating-point operation to test a single key, which is a gross underestimate. We'll thus assume the Roadrunner can test 10^15 keys per second. Testing 2^128 keys would then take about 10^16 years.

wikipedia

[Anonymous Coward]

Each Compute or I/O node is a single ASIC with associated DRAM memory chips. The ASIC integrates two 700 MHz PowerPC 440 embedded processors, each with a double-pipeline-double-precision Floating Point Unit (FPU), a cache sub-system with built-in DRAM controller and the logic to support multiple communication sub-systems. The dual FPUs give each BlueGene/L node a theoretical peak performance of 5.6 GFLOPS (gigaFLOPS). Node CPUs are not cache coherent with one another.

Re:Question

[Anonymous Coward]

The military didn't build Roadrunner. The U.S. Department of Energy built it, one in a long line of supercomputers used for (in addition to many other things) simulations to evaluate the reliability of the nation's aging nuclear weapons stockpile.

This CELL is not single precision

[raftpeople]

The CELL processor is single precision, which translates into wrong answers most of the time. Me guess is that for problems requireing double-precision numerics, you should divide CELL based supercomputer by 10 to 100 (software emulation of double precision is MASSIVELY SLOW), so this is really a teraflop machine. No big whoop...NEXT!!!

Things move fast in technology Jethro, including this 2nd gen of the CELL proc, this is what you missed:

Double Precision FP - 190TFLOPS (5 times faster than 1st CELL)
Memory: Expanded to 32gb
Memory: DDR2 instead of Rambus
65nm (I know, I know, but it's better than 90nm)

Re:exaflop, zettaflop, the yottaflop and the xeraf

[Anonymous Coward]

So, it also has Cell-based processors AND Opterons. I wonder what the functional division between the two chip types is?

Each node has two Opterons and 4 PowerXCell 8 processors (an upgrade to the PS3's Cell processor). This allows a developer writing code for the platform to run in a number of different modes: all Opteron, all Cell, or something in between. The first of these (all Opteron) may constitute a significant amount of the early work on the machine by practitioners, as they can simply compile legacy codes to the platform and ignore the Cell processors. Of course, to reap the full benefit of the machine, developers will exploit both the Cell chips and the Opteron chips.

Re:Question

[Anonymous Coward]

Perhaps, but that's not what we have. In fact, funding bodies for the most part try to "encourage" skewing the data downward, particularly in the corporate reich of america.

And don't forget it's pretty much the same data as goes towards the weather forecasts. Those things on TV? Sure, they're not always right. But they're pretty good nowadays. You can't have it both ways, like some creationist happily microwaving his dinner.

Re:Question

[Anonymous Coward]

If a dozen nukes dropped off their loading racks and exploded before being launched, the world would become a much better place.

Re:Change in paradigm

[TubeSteak]

all BlueGene CPUs were running at less than a GHz. And it seemed those low power cores were key to HPC (high performance computing).

Supercomputing is on its way to a water cooled infrastrucure.
IBM is already selling a product under the name bluefire
http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=207100873 [eetimes.com]
http://www.eweek.com/c/a/Infrastructure/IBM-Ships-First-WaterCooled-Supercomputer/ [eweek.com]

I hope we see more water & less air in the future

Re:This CELL is not single precision

[faragon]

The difference between the two Cells it is not "abysmal", let's do some comparisons:

a) PS3 "Classic Cell" 1 PPC64 w/ 2 threads and 7 SPEs (8, but one disabled, defective or not):

GFLOPS 32-bit (float): 3.2GHz * 8 FLOPS/Hz * 7 SPEs = 179.2 GFLOPS
GFLOPS 64-bit (double): 3.2GHz * 1 FLOPS/Hz * 7 SPEs = 22.4 GFLOPS (huge penalty, because of simulation via unoptimized simple precission operation)
GFLOPS 64-bit (double) a optimized 32-bit operation [netlib.org]: 3.2GHz * 3.9 FLOPS/Hz * 7 SPEs = 87.36 GFLOPS

b) Roadrunner "New Cell" 1 PPC64 w/ 2 threads and 8 SPEs:

GFLOPS 32-bit (float): 4GHz * 8 FLOPS/Hz * 8 SPEs = 256 GFLOPS
GFLOPS 64-bit (double): 4GHz * 4 FLOPS/Hz * 8 SPEs = 128 GFLOPS

P.S. ad hoc rewrite, based on my own Journal at Barrapunto (spanish /.) [barrapunto.com].

Re:But can it run....

[Anonymous Coward]

The DoD Like all large corporate entities refuses to install Vista.

Re:Cell processor

[BiggerIsBetter]

One thing to remember is that there is various iterations of the cell processor. The Xbox is a 3 core version The Playstation. I believe the Playstation is a 6 core processor. The roadrunner will use a 8 core processor. IBM originally discussed having a 16 core processor. There was not much talk about it afterwards. My guess is that there was significant bus contention issues. The original Power4's shut down one of their cores while running at full speed to avoid contention. The Power6 was designed to overcome these issues.

No, the XBox 360 has a three core PowerPC processor, not a Cell BE processor. The Cell BE in the PS3 has 1 PPE (Power core) and 7 SPEs (that "other" CPU core), while IBM apparently gets to use the fully functional Cells (PPE + 8 SPE) in their more expensive hardware. Those Cells with even fewer functioning SPEs might end up in HDTV TVs or similar.

Re:Cell processor

[bestinshow]

XBox360 has a tri-core in-order PowerPC - each core is actually very similar to the single general purpose PPU in the PS3's Cell.

Cell in addition has 8 SPUs. 1 is disabled in the PS3 for yield reasons, and another is reserved, so there are 6 available for general purpose computing.

Both run at 3.2GHz. I think Cell has at least 3x the vector/streaming power of the XBox 360 CPU, but only 1/3rd of the general purpose capability. Figures pulled from thin air, etc.

Re:Summary should have a shout out

[chrysrobyn]

From what I've heard (I'm no expert), these processors are good at certain types of calculations, but horrible at others. Ask a Cell to run Folding@Home, and it'll be blazing. Ask it to do general-purpose computing, though, and you'll quickly see the strength of other processors.

You're precisely correct. Cell's strength is in very predictable workloads (ones it can perform without branch mispredict penalties), very parallelizable workloads (ones that can be distributed over 6-8 SPU's / SPC's) that fit within 256 KB of local storage per SPU (manually managed cache, mapped to main memory). The non-double precision floating point enhanced version's (the version in the PS3) strength is further limited to integer and single precision floating point workloads. Roadrunner's Cell-DP eliminates that last limitation. While video games, encryption, nuke simulations and anything else that involves matrix manipulation can really stretch their legs on such a beast, general purpose computing won't find a benefit.