A Look Inside Oak Ridge Lab's Supercomputing Facility 59
1sockchuck writes "Three of the world's most powerful supercomputers live in adjacent aisles within a single data center at Oak Ridge National Laboratory in Tennessee. Inside this facility, technicians are busy installing new GPUs into the Jaguar supercomputer, the final step in its transformation into a more powerful system that will be known as Titan. The Oak Ridge team expects the GPU-accelerated machine to reach 20 petaflops, which should make it the fastest supercomputer in the Top 500. Data Center Knowledge has a story and photos looking at this unique facility, which also houses the Kraken machine from the University of Tennessee and NOAA's Gaea supercomputer."
Can we build ... (Score:4, Funny)
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Re: (Score:3)
With all due respects, Dr. Cooper, are you on crack?
-- George Smoot
(Oh, for God's sake, I click the "post anonymously" button 25 minutes after the last comment I post and it says I didn't wait long enough. Good way to spoil a joke... No, I'm not really Dr. Smoot. Posting under my real name is the only way to post this. Are you on slashdot staff, Dr. Cooper?
Oh, BTW, BAZINGA!
Re: (Score:2)
Linux has been on top in supercomputer usage since the mid 2000's.
http://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Supercomputers [wikipedia.org]
Re: (Score:2)
"UNITE with the Campaign for a Free Internet because today, our future begins with tomorrow!"
Freedom for more posts from people like you? I might need to reconsider this whole net neutrality thing...
Re: (Score:2)
Re: (Score:3)
Yeah, WTF does that mean, "...which should make it the fastest supercomputer in the Top 500..."
Not the capitalisation of T in Top 500, which you quoted correctly! It's the name of the list of fastest supercomputers.
Now, please hand in your nerd card on the way out.
Re:What? (Score:4, Informative)
The Top 500 is a specific list: http://top500.org/ [top500.org]
It's more correct to say it's the fastest on the list, than the fastest in the world. There are any number of metrics you can use to compare supercomputers. Top 500 just uses the most popular metric. Another machine could easily be the fastest on a different list, like http://www.graph500.org/ [graph500.org].
Re:What? (Score:4, Informative)
The Top 500 is a specific list: http://top500.org/ [top500.org]
It's more correct to say it's the fastest on the list, than the fastest in the world. There are any number of metrics you can use to compare supercomputers. Top 500 just uses the most popular metric. Another machine could easily be the fastest on a different list, like http://www.graph500.org/ [graph500.org].
The other specific consideration is that the list is ONLY for those that volunteer to run the Linpack benchmark and wish to publicize the results. It is presumed that governments with classified computing facilities withhold this information, for obvious reasons, so there are likely many "supercomputers" (perhaps even a "fastest") that will never be part of the Top 500. The US NSA, for example, is widely believed to operate facilities at or near the top of the list, but they are nowhere in sight for obvious reasons.
Re: (Score:3)
Is it Symbiotic? (Score:2)
the final step in its transformation into a more powerful system that will be known as Titan.
Oh, that is not even its final form.
Re: (Score:2)
No, the final version will be called Colossus!
"This is the voice of World Control" (Score:2)
Ah, one of my favorite films. Still stands up even today.
Re: (Score:2)
Redundant? (Score:2)
" which should make it the fastest supercomputer in the Top 500"
At first I thought this was redundant, but then I wondered if there are faster supercomputers that simply are not independently verified to be in the top 500 supercomputers. Anyone have any more info, or am I just overthinking this?
Re: (Score:1)
Obviously a number of classified systems don't share their benchmark results, for one....
Re: (Score:1)
You know what they say about hiding things in plain sight....
Re: (Score:2)
Different benchmarks will produce a different fastest super computer list. 'Top 500' is a specific list that uses a specific benchmark, a benchmark that this particular machine is currently at the top of. Using a different benchmark could be just as valid and produce a completely different list.
Re: (Score:2)
I've been there. (Score:2)
I went there my Sophomore year to check out Oak Ridge. I didn't go for computing ,but since my guide knew that I like computing, he took me to look at the supercomputers. It's this huge room that was visible through glass windows which looked essentially like a huge clean, white, office floor with all the cubicles removed and with the supercomputers in place instead.
At that time (2009?) I heard it wasn't really the fastest supercomputer but it's awesome to hear they're revving it up to that. If I didn't hat
Re: (Score:2)
Trufax. I have to read a section and do an outline on it that's due in two hours but yeah...lols.
Topology matters more than GFLOPS (Score:5, Insightful)
I really, really wish articles would stop saying that computer X has Y GFLOPS. It's almost meaningless, because when you're dealing with that much CPU power, the real challenge is to make the communications topology match the computational topology. That is, you need the physical structure of the computer to be very similar to the structure of the problem you are working on. If you're doing parallel processing (and of course you are, for systems like this), then you need to be able to break your problem into chunks, and map each chunk to a processor. Some problems are more easily divided into chunks than other problems. (Go read up on the "parallel dwarves" for a description of how things can be divided up, if you're curious.)
I'll drill into an example. If you're doing a problem that can be spatially decomposed (fluid dynamics, molecular dynamics, etc.), then you can map regions of space to different processors. Then you run your simulation by having all the processors run for X time period (on your simulated timescale). At the end of the time period, each processor sends its results to its neighbors, and possibly to "far" neighbors if the forces exceed some threshold. In the worst case, every processor has to send a message to every other processor. Then, you run the simulation for the next time chunk. Depending on your data set, you may spend *FAR* more time sending the intermediate results between all the different processors than you do actually running the simulation. That's what I mean by matching the physical topology to the computational topology. In a system where the communications cost dominates the computation cost, then adding more processors usually doesn't help you *at all*, or can even slow down the entire system even more. So it's really meaningless to say "my cluster can do 500 GFLOPS", unless you are talking about the time that is actually spent doing productive simulation, not just time wasted waiting for communication.
Here's a (somewhat dumb) analogy. Let's say a Formula 1 race car can do a nominal 250 MPH. (The real number doesn't matter.) If you had 1000 F1 cars lined up, side by side, then how fast can you go? You're not going 250,000 MPH, that's for sure.
I'm not saying that this is not a real advance in supercomputing. What I am saying, is that you cannot measure the performance of any supercomputer with a single GFLOPS number. It's not an apples-to-apples comparison, unless you really are working on the exact same problem (like molecular dynamics). And in that case, you need some unit of measurement that is specific to that kind of problem. Maybe for molecular dynamics you could quantify the number of atoms being simulated, the average bond count, the length of time in every "tick" (the simulation time unit). THEN you could talk about how many of that unit your system can do, per second, rather than a meaningless number like GFLOPS.
Re: (Score:1)
Here's a (somewhat dumb) analogy. Let's say a Formula 1 race car can do a nominal 250 MPH. (The real number doesn't matter.) If you had 1000 F1 cars lined up, side by side, then how fast can you go? You're not going 250,000 MPH, that's for sure.
No... but collectively you cover the same distance, right?
Re: (Score:1)
I don't remember reading any stipulation about the nature of where the cars are placed.
I guess my point is that with ideal conditions, i.e. an infinitely wide and straight track (plus no mechanical failures, infinite tyres, come on we all understand the meaning of the words "ideal conditions"), then collectively 1000 cars cover the same distance as one car going 1000 times the speed.
Similarly with ideal conditions, i.e. perfect data parallelism, then collectively 1000 processors can process the sa
Re: (Score:1)
Comparing LINPACK numbers makes sense. But GFLOPS (or TFLOPS or PFLOPS or whatever), by itself, is a meaningless and misleading number. Most people just stop thinking when they see a single metric like GFLOPS, and then they compare GFLOPS in one system to GFLOPS in another system. If those systems *are* comparable, then fine. But often enough, they are not comparable.
Also, it wasn't a rant. A rant would have involved caps lock.
Re: (Score:1)
Earth escape velocity is ~25,000 MPH. But 1000 F1 cars cannot reach orbit. Adding up numbers does not always give you a meaningful number.
Re: (Score:3, Informative)
I'll drill into an example. If you're doing a problem that can be spatially decomposed (fluid dynamics, molecular dynamics, etc.), then you can map regions of space to different processors. Then you run your simulation by having all the processors run for X time period (on your simulated timescale). At the end of the time period, each processor sends its results to its neighbors, and possibly to "far" neighbors if the forces exceed some threshold. In the worst case, every processor has to send a message to every other processor. Then, you run the simulation for the next time chunk. Depending on your data set, you may spend *FAR* more time sending the intermediate results between all the different processors than you do actually running the simulation. That's what I mean by matching the physical topology to the computational topology. In a system where the communications cost dominates the computation cost, then adding more processors usually doesn't help you *at all*, or can even slow down the entire system even more. So it's really meaningless to say "my cluster can do 500 GFLOPS", unless you are talking about the time that is actually spent doing productive simulation, not just time wasted waiting for communication.
Considering that computational fluid dynamics, molecular dynamics, etc., break down into linear algebra operations, I'd say that the FLOPS count on a LINPACK benchmark is probably the best single metric available. In massively parallel CFD, we don't match the physical topology to the computational topology, because we don't (usually) build the physical topology. But I can and do match the computational topology to the physical one.
Re: (Score:1)
Yes, many problems can be expressed as dense linear algebra, and so measuring and comparing LINPACK perf for these makes sense for those problems. However, many problems don't map well to dense linear algebra. The Berkeley "parallel dwarfs" paper expresses this idea better than I ever could: http://view.eecs.berkeley.edu/wiki/Dwarfs [berkeley.edu]
Re: (Score:1)
Yes, many problems can be expressed as dense linear algebra, and so measuring and comparing LINPACK perf for these makes sense for those problems. However, many problems don't map well to dense linear algebra.
Sure, but as far as I've seen, linear algebra problems dominate the runtime of these very large systems. That's what I use them for.
At least the first 6 on that dwarfs list are done daily on top500 machines. I write parallel spectral methods, and use structured and unstructured grids. Achieving high scaling on these on massively parallel machines is not at all what I would call an open problem (as far as correctly using a given network for a problem, or designing a network for a given problem). For any give
Re: (Score:1)
You're doing it wrong. Line them up end-to-end, not side-by-side.
The nuclear establishment in the post-nuke era (Score:2)
The US still has these Big Science centers left over from the glory years. There's Oak Ridge, Los Alamos, and the Lawrence Livermore Senior Activity Center (er, "stockpile stewardship"), plus the NASA centers. Their original missions (designing bombs, sending people to the Moon) are long gone, but nobody turned off the money, so they keep looking for something, anything, to justify the pork.
The atomic centers are all located in the middle of nowhere. This was originally done for good reasons - their exist
Re: (Score:1)
Re: (Score:2)
Please do your homework first. While the supercomputers at Lawrence Livermore, Los Alamos, and Sandia National Laboratories are primarily used for nuclear weapons work, the work of keeping the country's huge stockpile safe and reliable is a gigantic job, especially if you don't want to actually detonate any of the warheads. Yep, that's the trick. Simulate the ENTIRE weapon, from high explosive initiation all the way to final weapon delivery. With all of the hydrodynamics, chemistry, materials science, nucle
Pictures (Score:2)
Nuke'em from orbit (Score:2)
While it's certainly fascinating to hear about the machine itself, it's easy to forget part of why it exists: simulating destruction. The Manhattan Project also came from Oak Ridge, if you recall.
As someone who lives in the region, nobody is particularly keen on what possibly goes on at these places. There are various "secret" military installations scattered around here, from Oak Ridge to Holston Army Ammunition. Between what we factually know is buried under and developed at these places, and what is r