$208 Million Petascale Computer Gets Green Light

coondoggie writes "The 200,000 processor core system known as Blue Waters got the green light recently as the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications (NCSA) said it has finalized the contract with IBM to build the world's first sustained petascale computational system. Blue Waters is expected to deliver sustained performance of more than one petaflop on many real-world scientific and engineering applications. A petaflop equals about 1 quadrillion calculations per second. They will be coupled to more than a petabyte of memory and more than 10 petabytes of disk storage. All of that memory and storage will be globally addressable, meaning that processors will be able to share data from a single pool exceptionally quickly, researchers said. Blue Waters, is supported by a $208 million grant from the National Science Foundation and will come online in 2011."

imagine...

nah, nevermind

$208 Million Petascale Computer Gets Green Light

I'm glad they've given it a green light.

Imagine having all that computer power, and not even knowing if it was switched on!

Naive question...

Yes, I know this is probably a very naive question, but has anyone here actually had the privilege of working on one of these things? I mean, what do they actually use this for?

I think it's awesome, but are there any concrete advancements that can be attributed to having access to all this computing power?

Just wondering...

Re:Naive question...

I don't use one myself, but I know people involved with supercomptuers. They are used for large simulations. Often this comes down to solving large systems of linear equations, since at the inner step finite elements need solutions to these large equation systems. The point is, the larger the computer the larger the grid you can have. This involves simulating a larger volume, or simulating the same volume in more detail (think, for example of weather systems).

As for concrete advancemants? I'm not in the biz, so I don't know, but I expect so. Apparently they're also used for stellar simulations, so I expect the knowledge of the universe has been advanced. I would be suprised if they haven't seen duty in global warming simulation too.

Re:Naive question...

Weather modeling comes to mind, both terrestrial and space.

rj

Re:Naive question...

These machines are used to work on simulations that involve aerodynamics and hydrodynamics, quantum electrodynamics (QED), or electromagnetohydrodynamics. All of these simulations require that a mathematical model is constructed from a high density mesh of data points (2048 ^ 3). Blocks of such points are allocated to individual processors. Because of this, each processor must be able to communicate at a high speed with its neighbours (up to 26 neighbours with a cubic mesh).

Usually, the actual individual calculations per element will be take up less than a page of mathematical equations, but require high precision, so the data values will be 64-bit floating point quantities. A single element might require 20 or more variables. Thus the need for some many processors and high clock speed.

Re:Naive question...

electromagnetohydrodynamics

And quantum electroptical tomographics. See, I can make shit up, too...

Re:Naive question...

Do you notice neither USA or Russia blows a portion of planet to test nuclear weapons anymore? It is because the planet is so peaceful so further research is not required? Unfortunately no.

These monsters can simulate a gigantic nuclear explosion in molecular level.

Or for peace purposes, they can actually simulate that New Orleans storm based on real World data and pinpoint exactly what would happen.

Re:

Or for peace purposes, they can actually simulate that New Orleans storm based on real World data and pinpoint exactly what would happen.

Right.

That's why the City of New Orleans evacuated to Baton Rouge.

Re:Naive question...

I'm working on a PhD in chemical engineering, and I do simulations. I occasionally use Lonestar and Ranger, which are clusters at TACC, the U. of Texas' supercomputing center. Lonestar is capable of around 60 TFLOPS and Ranger can do around 500-600 TFLOPS. A few users run really large jobs using thousands of cores for days at a stretch, but the majority of people use 128 or fewer cores for a few hours at a time.

My research group does materials research using density function theory, which is an approximate way of solving the Schroedinger equation. Each of our jobs usually uses 16 or 32 cores, and takes anywhere from 5 minutes to a couple of days to finish. Usually we are interested in looking at lots of slightly different cases, so we run dozens of jobs simultaneously.

The applications are pretty varied. Some topics we are working on -
1) Si nanowire growth
2) Si self-interstitial defects
3) Au cluster morphology
4) Catalysis by metal clusters
5) Properties of strained semiconductors

Re:Naive question...

Come on now. Let's be serious. They're trying to play Crysis.

Re:Naive question...

Did you know that a very credible FAQ mentions Apple purchased a Cray for manufacturing/design and someone actually saw them emulate MacOS on that monster?

http://www.spikynorman.dsl.pipex.com/CrayWWWStuff/Cfaqp3.html#TOC23 [pipex.com]

I bet they tried some games too :)

Re:Naive question...

Actually, no, they're future proofing their computer for Duke Nukem Forever :)

More crap code

Cool thing about the globally addressable petabyte. That way people writing really crappy code that don't bother thinking about their memory storage can just thrash away. And who cares about pipeline stalls.

I find it funny how the people who have never been formally trained with writing in a language (Mathematics, and just science in general) write the best codes while the majority of the IT people I see write the most appalling code I've ever seen. I think it has something to do with the fact that the science people don't pretend to know everything and are much more willing to learn something new while the IT people already know everything.

Star Trek "Data" rated at 60 Teraflops

I just saw The Measure of a Man episode on the Star Trek Labor Day marathon. Data has a speed of 60 Teraflops and 100 petabytes of storage. That used to seem large in the late 1980s. (Episode were Data goes on trial whether he is a machine or sentient.)

Re:

About a decade or so ago, I remember someone very crudely trying to ballpark the amount of storage that would be needed to contain the raw data of the entire human brain complete with a lifetime of experience at around 10 terabytes. Needless to say, that seems incredibly unlikely by today's standards.

Even if something like this were possible (storage not withstanding), the data itself would likely be unusable until we sufficiently understood just how our brains work with their own data enough to create a cr

It's said...

...Apple used to use a Cray to design their new computers, whereas Seymoure Cray used an Apple to design his.

More compute power is nice, but only if the programs are making efficient use of it. MPI is not a particularly efficient method of message passing, and many implementations (such as MPICH) are horribly inefficient implementations. Operating systems aren't exactly well-designed for parallelism on this scale, with many benchtests putting TCP/IP-based communications ahead of shared memory on the same fripping node! TCP stacks are not exactly lightweight, and shared memory implies zero copy, so what's the problem?

Network topologies and network architectures are also far more important than raw CPU power, as that is the critical point in any high-performance computing operation. Dolphinics is quoting 2.5 microsecond latencies, Infiniband is about 8 microseconds, and frankly these are far far too slow for modern CPUs. That's before you take into account that most of the benchmarks are based on ping-pong tests (minimal stack usage, no data) and not real-world usage. I know of no network architecture that provides hardware native reliable multicast, for example, despite the fact that most problem-spaces are single-data, most networks already provide multicast, and software-based reliable multicast has existed for a long time. If you want to slash latencies, you've also got to look at hypercube or butterfly topologies, fat-tree is vulnerable to congestion and cascading failures - it also has the worst-possible number of hops to a destination of almost any network. Fat-tree is also about the only one people use.

There is a reason you're seeing Beowulf-like machines in the Top 500 - it's not because PCs are catching up to vector processors, it's because CPU count isn't the big bottleneck and superior designs will outperform merely larger designs. Even with the superior designs out there, though, I would consider them to be nowhere even remotely close to potential. They're superior only with respect to what's been there before, not with respect to where skillful and clueful engineers could take them. If these alternatives are so much better, then why is nobody using them? Firstly, most supercomputers go to the DoD and other Big Agencies, who have lots of money where their brains used to be. Secondly, nobody ever made headlines off having the world's most effective supercomputer. Thirdly, what vendor is going to supply Big Iron that will take longer to replace and won't generate the profit margins?

(Me? Cynical?)

Re:It's said...

You could not be more wrong.

Considering that we've got SDR IB with under 2 microseconds latency for the shortest hops (and ~3 for the longest), I think you need to go update your anti-cluster argument. :) The problems with congestion in fat trees have virtually nothing to do with latency. Yes massive congestion will kill your latency numbers, but given that you don't get cascades and other failures causing congestion without fairly large bandwidth utilization, latency is the least of your worries that that point. Furthermore, the cascades you talk about also aren't common except in extremely oversubscribed networks or in the presence of malfunctioning hardware. We do our best to use properly functioning hardware and to have no more that 2:1 oversubscription (with our largest machine not being oversubscribed at all).

MPICH ain't that bad (heck, MPICH2, even just it's MPI-1 parts might be considered to be pretty good by some). MPI as standard for message-passing is fine. I'd love to hear what you think is wrong with MPI and see some examples where another portable message passing standard does consistently better. Though it's a bit like C or C++ or Perl in that there are lots of really bad ways to accomplish things in MPI and a handful of good ones. It's low-level enough that you need to know what you're doing. But if you believe anyone that tells you they have a way to make massively parallel programming easy, I've got a bridge you might be interested in.

Finally, I don't know of much in the way of a "supercomputer" that's using TCP for it's MPI traffic these days, so you can put that old saw out to pasture as well.

Don't worry

in 40 years some kid will laugh at your pathetic attempt at geek coolness when you mention the Bluewater and say "wow your old, Im amazed anyone needed a warehouse just for one petaflop even my Wango-matic game cube has 50 petaflops"

Can't take another 40

in 40 years some kid will laugh at your pathetic attempt at geek coolness when you mention the Bluewater and say "wow your old..."

Forty more years of the kids saying "your"? Kill me now! :)

Re:

No. This is Urbana,Illinois. HAL 9000 [wikipedia.org] would be more appropriate.

Re:

I think you meant tea. [wikipedia.org]

Re:Yes, but the article doesn't address a few ques

It will not run 32 bit linux, so of course, the admins in charge are going to bitch about the lack of adobe flash support.

Re:

Yeah, that was my thought. Roadrunner at Los Alamos sits at the top of the 500 list [top500.org] with Rmax 1,026,000. I don't know enough about benchmarks to distinguish between "Rmax" and "sustained petascale," but it is achieving over a petaflop. Maybe someone here can tell us more about linpack [top500.org] vs. whatever they're using for this new one. I notice the article linked in the story mentions Roadrunner at the end, but without saying how it compares in speed. It doesn't seem to say by what specific measure this new comput

Re:

2020 seems unlikely. A reasonably accurate real-time synaptic simulation can run maybe 100 neurons on a high end pc today, probably less. A human brain has about 100 billion neurons, so we're 1 billion times short in computation. Last time I checked, GPUs had not yet been used in neuron simulation, so I'll even give you that we may be 1000 times better off. That's still 1 million X improvement needed to match the brain, or roughly 20 more generations of computer hardware, at a generous 18 months, that l