NCSA To Build $53 Million, 13-Teraflop Facility 162
Quite a few readers submitted news of a distributed system to be built by four U.S. institutions (mostly) out of IBM computers, and paid for with a whopping grant. DoctorWho and november writes: "'The National Science Foundation has awarded $53 million to four U.S. research institutions to build and deploy a distributed terascale facility...' A link to the press release is here." An anonymous reader contributed a link to coverage on Wired, and GreazyMF to one of this story at the New York Times.
Intel's article (Score:1)
Linux is for kids (Score:2)
13 teraflops is a pretty big toy.
Re:Linux is for kids (Score:1)
They can try all they want, but it is a moot effort. The recent Linux GUI desktops are gorgeous. Take away the GUI and you have a hard-core workhorse. Either way these "toy" OSs are pretty damn serious.
What scientists and engineers appreciate about Linux and always have about UNIX in general is the sheer flexibility and modularity offered. I have never felt Windows offers such flexibility.
Re:Linux is for kids (Score:1)
Re:Linux is for kids (Score:1, Troll)
It wouldn't look bad in a teletubbies episode.
2003 (Score:1)
My Prediction (Score:1, Funny)
This project will flop most terably
sorry
Space enough for your lifetime? (Score:1)
Let us examine:
450 Terabytes
4.5mb per mp3 (average)
Thats 100,000,000 Mp3's
Lets take the average length of music = 3.5 min.
Thats:
350000000 Minutes
5833333.33 Hours
243055.55 Days
34722.22 Weeks
667.73 Years
Is there any venture capitalists interested in this idea? I think that this could be one great consumer service!!
You know what's good about this... (Score:1)
It's all linux baby!
13 teraflops? No problem. (Score:1)
IBM... (Score:1)
Now my only question is... where can i get a beowulf cluster of these babies? That would be sweet...
Linux (Score:2, Insightful)
On crack, are you? (Score:1)
Saying Linux isn't tested nor built for clusters this big is a little like saying that sand isn't meant to go in car windows.
Linux has ten years and millions of manhours' worth of development and refinement that has gone into it. You wanna do WHAT from scratch?? PASS!
A cluster is still a machine-by-machine entity, which is the level that the OS is working at; it's the "hooks" you create that facilitate cluster behavior. If you want to write an actual "cluster OS," i.e., one that does not have a context on a single machine, then by all means, go for it, but don't blame these guys for building something by integrating mostly pre-existing parts in order to get the behavior they seek.
Forgive me for the harsh subject line; it's been a long week!
Re:Linux (Score:1)
Linux doesn't have to scale (Score:1, Informative)
This is not a big SMP machine - the kernel does not have to manage all 1300 CPU's at once. Instead, there will be 1300 copies of Linux running (in the long run, you don't really want the OS involved much anyway)
It totally depends on exactly what they'll run on it, but based on what's currently running on the NCSA machines the concerns will be a high speed, low latency network (which they got in Myrinet - note that I didn't say cheap) and a good MPI implemenation to take advantage of it. Both LAM and MPICH have Myrinet-aware implementations, and they're both pretty fast.
Re:Linux doesn't have to scale (Score:1)
Myrinet (Score:1)
Re:Linux (Score:2, Informative)
You seemed to be slightly confused about how such clusters work. Linux is more than just a good choice, it is the definitive best choice in the supercomputing industry for clusters. If you ever goto the SuperComputing conferences, you would notice how there are many dozens of cluster companies, and they all use linux. Clustering is what supercomputing is all about now.
Linux does not need to efficiently utilize 1300+ itanium processors. This isnt a singular machine, it's a cluster. The linux kernel needs to be able to handle its individual node (consiting of a couple processors or so) efficiently, not all the processors. The distribution and parallelization is handled by other software, such as message passing interfaces like MPI. To be honest, linux is tested on many clusters with this many processors and whatnot, and it has been customized and hardened for use in large magnatude clusters. But like I said, it really isnt a kernel thing, its the other software in the package that controls distribution of processing payloads to the individual nodes.
Building an operating system for scracth is just a bad idea for something like this. They are not exactly something that can be built a couple weeks.Look at all the other OS projects out there besides Linux. Even with a few dozen constributors, alot of been years in the making, and are not any where near the level of linux, or an OS that could be used in such a fashion. Basically, it would take a very long time to build an OS from scratch that would do all the things necessary, and have the stability requirements for such a project.
Time and Space don't like you either (Score:5, Funny)
to eliminate the tyranny of time and space limitations.
This time and space flaming has got to end. Granted, time and space have a monopoly on time and space, but it is a *benevolent* monopoly, which is ok with every legislative body in the world except the EU. Time and space have prevailed as the primary purveyors of time and space through quality, perseverance, and generous donations to any political party that would take their money. So, lay off, slashdot!
See ya (Score:1)
karb: I'm going to kuro5hin!!
Re:Time and Space don't like you either (Score:2)
No kidding. If this keeps up, it could spell the end of innovation as we know it. Next we'll have the EU hauling time and space off to court, saying that they tried to extend their monopolies into other markets through predatory practices and hiding their API's. (They'll be saying things like, "We didn't even KNOW about relativity until the 20th century, for crying out loud! Where were those API's?")
On the other hand, with a little competition, I might finally have enough time to finish my work and space on my desk to keep all this idiotic paperwork...
yeah but can it... (Score:2, Funny)
It could be built for less than half that cost. (Score:1)
hardware only quarter of the cost (Score:2)
scale up of a commercial system- always have
very high overheads.
First, you have a dedicated hardware and software
support crew. A production system ammortises
this over multiple deliveries.
Second, you are pushing the envelope. Though it
looks possible on paper, you don't always know
what won't scale up properly in a cutting edge
system.
Third, educational institutions (U of I) charge
large overheads (@50%) for existing buildings/staff.
The largest systems just don't get built
unless the government subsdizes some of the costs.
If you are lucky, the contracting company learns
new things to help its commercial side.
Re:hardware only quarter of the cost (Score:1)
It works! (Score:2)
"Quite a few readers submitted news of a distributed system to be built by four U.S. institutions..."
Looks like our "Slashdot Distributed Story Submission" (SDSS) is working quite nicely.
I wonder... (Score:1)
Terraflop..... (Score:1)
Can you? (Score:1, Offtopic)
Aww, shit. nevermind.
Why waste money with Bleeding edge? (Score:2, Insightful)
--Mike--
Re:Why waste money with Bleeding edge? (Score:1)
Re:Why waste money with Bleeding edge? (Score:1)
Re:Why waste money with Bleeding edge? (Score:1)
Beyond that, you really need as high a processing-power to memory-transfer cost ratio as possible. When you are dealing with highly coupled simulations (such as wireless simulations) you pay dearly for cross-processor memory IO.
Re:Why waste money with Bleeding edge? (Score:1)
Re:Why waste money with Bleeding edge? (Score:1)
So why complain?
Re:Why waste money with Bleeding edge? (Score:1)
When you attempt to distribute a problem (a non-trivial problem, one that isn't embarassingly parallel), you have to strike a balance between the load on a single processor (or single SMP machine), and the overhead associated with message passing. Many research groups who build their own clusters go through extensive analysis of their particular problems to find the appropriate "sweet spot". For this machine, which will no doubt be used for many dissimilar projects, I don't know how they determined how much per-node power they needed. With a $53million grant, I bet they just went with the simple solution: as much as they can get
My point is just that the assumption that the CPU is the biggest expense is erroneous when dealing with specialized networking equipment like Myrinet, and that trading processor power for numbers isn't always a good bet.
They should use SETI@Home (Score:1)
OS/software (Score:1, Troll)
Re:OS/software (Score:4, Informative)
Re:OS/software (Score:1)
You mean the Globe is not an OS? Think about it for a while - you can set your own enviroment in which you operate and it is a complex system.
AIX (Score:2, Funny)
Re: when I don't understand AIX I say 'When you d (Score:1)
Re: when I don't understand AIX I say 'When you d (Score:2)
Re: when I don't understand AIX I say 'When you d (Score:1)
Re: when I don't understand AIX I say 'When you d (Score:1)
No, I'm not a zealot. AIX does, however, provide me a nice comfortable living... it can't be all bad.
Re: when I don't understand AIX I say 'When you d (Score:2)
I think any OS can be an entreprise-level OS in the hands of the right person (even M$ Windows and OS/2). A former co-worker of mine could make an NT server scream. It had uptimes of a year or more. Very stable, very reliable---in his hands. We had a similar box for in-house purposes. Almost the same hardware. It went up and down like a damned yo-yo----in our hands. A similarly gifted AIX person can do similar things. The average Joe can't though. The average Joe can't make termcaps work right in AIX let along secure the box. I'd love to run PPC Linux on our 6k's. It would really make those boxes scream. Anything is faster than AIX on those boxes.
I would personally love to have the time to get really good Solaris experience. Sure I probably wouldn't use it in the end unless I became the admin of a number of Solaris boxes but still I'd like the experience. I'd like to shadow a good Solaris admin for a couple weeks.
BTW, the original post was 90% humor and 10% sarcasm.
Re: when I don't understand AIX (Score:1)
I bet he just used VMWare to run Linux on top of it
Don't be so hasty to slam AIX (Score:1)
Re:AIX (Score:1)
Looks like Someone just won the RSA award (Score:1)
Spare CPU cycles for a sociopathic meglomaniac? (Score:3, Funny)
Hmm (Score:1)
Not just NCSA (Score:1)
NCSA is certainly an important part of this partnership, but they're neither the only part nor the lead site.
Globus/gnutella (Score:2)
Weather forecast (Score:2, Interesting)
users, big money & perspective (Score:2, Interesting)
Every cluster I know of (around 20 systems, 14 sites) is not for want of cycles, they need programmers to write the codes to eat the cycles. There are not enough small 'education' clusters to allow everyone the education & experience.
Even just $1m of that could be much better spent in education instead of feeding the 0.0001% of computer problems that currently need this class of hardware.
-- Multics
Re:users, big money & perspective (Score:1)
One could also calc PI now on a beowolf now. So yes, I would like to see a beowolf programming class in the college courses. Have is as intro, hardware setup, software setup, programming, advanced topics (weather).
Re:users, big money & perspective (Score:2)
We need more programmers to program the machines? Maybe. This is an important but niche market, and throwing billions into education so that kids with bachelor's can call themselves super-computer programmers isn't the answer. The systems are already programmed by brilliant people researching these problems, doctorates all around. This isn't work for your average 15 year old 3r33t haXor, you know?
Re:users, big money & perspective (Score:1)
So since we don't have faster processors (relatively) we will have more and more processors.
I do not advocate spending Billions on teaching how parallel programming works and how to use PVM and MPI effectively, but I do think it is time that it become a standard theme at the college level CS world. That means that the professors learn how it works and then have access to equipment that allows everyone to have the experience.
-- Multics
This is the future of the Internet (Score:2, Interesting)
Re:This is the future of the Internet (Score:1)
Re:This is the future of the Internet (Score:3, Informative)
Unless, maybe, it's controlled by MS... Take a look at these two articles on The Register [theregister.co.uk]:
- MS poised to switch Windows file systems with Blackcomb [theregister.co.uk]
- How Microsoft's file system caper could wrongfoot the DoJ [theregister.co.uk]
for comparision (Score:4, Informative)
This is impressive, but the nasa machine will blow it out of the water.
Re:for comparision (Score:2)
A much better comparison (Score:1)
Re:A much better comparison (Score:2)
Most people can visualize a hundred or so boxen a lot easier than a thousand or so. It gets a little unreal. So the Brit site [dur.ac.uk] with pretty pictures [dur.ac.uk] of the system is a good site for those not familiar with the larger systems.
They have other pretty pictures from their work [dur.ac.uk] as well.
Re:for comparision (Score:2, Informative)
Re:for comparision (Score:1)
Re:for comparision (Score:2, Informative)
Re:for comparision (Score:1)
and that's our old machine...
Wasted (Score:1)
I wonder how much of that power goes wasted into the regular administration of the site, idle time, everyone's web and email traffic, and storing employees' pr0n pix and mp3s, instead of the science it is intended for. It is my experience in a corporate environment that no one ever cleans up disk or mail boxes and they don't consider the impact of running non-essential processes on compute servers.
Also, what are they doing to protect and backup that much data?
Re:Wasted (Score:1)
If you look at the details of the system it doesn't handle email or web traffic, just physics programs which will be submitted through a single node which then distributes it out to the 128 processors so there wont be any user data on the machines just temp files from the data being run on each processor.
Backing up data is likely to occur through the huge amount of storage currently being purchased for the UK-GRID and tape. What is there to protect? Monte Carlo simulations of cosmology experiments? this isn't personal or corporate data, one bogus result is unlikely to throw the experiment off.
Anyway, this is only one of a few new systems in the UK which are getting announced at the moment, so although they aren't as large as the ones being *talked* about in the states they're here, now and working while it'll take 2 years before the american ones come online.
From the front line.. (Score:1)
Actually, I didn't stick around for as much of the press conference, 'cuz I had WORK to do! Many press releases on the DTF make it sound like it's one cluster in one lab's basement, and that ain't right. As importantly as looking at the distributed nature of the project, look at what each institution is contributing- this isn't a homogeneous wide-area cluster. I don't have a big part in it, and my internship is almost over, but I'd like to think that what I've been working on for over a year may become well-known soon. So yeah, while the press conference was going on I was in the next room working on enhancing a visualization library to work on tiled displays, (which has been news on /. recently. Too bad few managed to find our work here- We gots neet stuph).
Now an obligatory Oh, puh-leeze! RC-5 cracking? Quake? We've already seen Quake3 in the CAVE [visbox.com]. Listening to conversations at the reception, there are much cooler things coming..
Cryptomancer, working the magic on code
Yes, but... (Score:2, Funny)
I'm just glad it's not the NSA... (Score:2)
Re:I'm just glad it's not the NSA... (Score:5, Interesting)
While I'm sure many members of the audience would like to see NSA's hand in here somewhere the processing power is needed since CERN's sending out data from the experiments at 40TB a second (ok, ok I know it gets filtered down to only 100MB/sec)
Which is the problem, while these 4 systems make a nice addition to the GRID we need more supercomputers!!!
Re:I'm just glad it's not the NSA... (Score:2, Funny)
if the NSA would build such a computer, you think they would announce it to the world ?
it may already be out there
</paranoid>
Re:I'm just glad it's not the NSA... (Score:1)
cracking power? (Score:1)
Do the math... (Score:2)
13 teraflops = 1e13 instructions per second
Assume 1 trial decryption per instruction
which is of course unrealistically low.
You still need 3.4e25 seconds or about 1e18 years to search that keyspace.
Sorry, no cigar...
From the wired article (Score:5, Insightful)
"The only way to do this project is open source," project director Stevens said.
Interesting that researches know that open source projects are the only way they can control all the variables. After all, if you don't control the OS, you can't be sure some little bug in the code is screwing with your data. Universities have long understood this principle, which is why Unix is so popular. Now our millions of tax payer dollars will be spent on research rather then licensing costs, plus the research is controlled, scalable, and open to peer review. Always nice to see professionals understand the benefits of open source that no closed source movement could possibly replicate.
Re:From the wired article (Score:3, Insightful)
While I am in broad agreement, do not take the announcement of this machine as another blast in the direction of Micro$haft, or another nail in their corporate coffin. If a closed-source system is built correctly, and presents consistent and well-documented interfaces to the outside world, then it can be just as effective.
Business didn't employ Unix because they could get the source code, they bought it because it followed interface standards, and it was thus easier to get your Unix boxes to talk to your S390s and your Unisys 2200s and your VAXs etc etc etc
If Microsoft had offered common external interfaces in the first release of NT, and not those bloated buggy propriety standards years later, they might actually have managed to produce a useable OS that enterprises could then integrate into their existing data centres, rather than boxes that perform tasks in independant installations.
Re:From the wired article (Score:2)
Ah, but then there would be no incentive in the future to replace those machines. Microsoft, as the subscription based licenses show, cannot merely sell a product and live off the income. That's not how you maximize profit. You keep them paying, and make sure they can't pay anyone else. That's how a monopoly works - you don't play nice with anyone else.
Re:From the wired article (Score:3)
The GPL seriously undermines the commercial viability of software.
Re:From the wired article (Score:1)
Yes - using linux is all very fine and well but it has some nasty suprises. For example on RedHat 6 upgrading to the next version of Sun's JDK (in this case 1.3) requires an upgrade to a new version of certain libraries and the recompiling of most of the software on the system.
While this is fine on a home hobbyist machine it is not very good if you have multiple users and especially not if you are selling computer time to companies. And why do you need Java 1.3 you ask? You need it because the Globus CoG toolkit [globus.org] needs it.
Re:Does this have anything (Score:2, Funny)
I (gasp) -- LOVE (huuugh) -- THIS (aaarrr) -- COMPANY (shhhhlop)
Re:Just to get on with it (Score:1, Offtopic)
Re:Just to get on with it (Score:2, Funny)
Re:13-Teraflops.... (Score:1)
Re:horsepower (Score:1)
Re:horsepower (Score:1)
Re:13-Terflops of... (Score:1)
Yeah that's why they're making all the details so public. >SMACK<
On the other hand, think Contact [imdb.com]: government philosophy: "Why buy one when you can get two at twice the price."
Re:Deep Thought ? (Score:1)
it's 42.