New 640-Node Apple Xserve Cluster at UIUC

frostyboy writes "At the University of Illiois at Urbana-Champaign's Department of Computational Science and Engineering , a new high-performance computing cluster comprising 640 Dual G5 Xserves has completed benchmarking runs for the top500 list. The New Turing Cluster is a replacement for an old 208-node linux cluster. Preliminary results have it at about 4.6 teraflops, not too shabby. Slide Presentation and Photo Album"

Re:Linux?

[keeleysam]

I would assume that since it replaced a Linux cluster, it probobly runs close to the same form that it replaced so all their old apps will continue to work without much tweaking.

Re:Linux?

[geoffspear]

Ah, Slashdot. Where people reply to people who haven't read the article with assumptions based on nothing in particular, but make it clear that the person replying hasn't read the article, either.

It's running OS X.

Re:Linux?

[Spock the Baptist]

"Where people reply to people who haven't read the article"

Not reading the article is often due to the fact that the site that the article gets /.ed.

Re:Linux?

[geoffspear]

UIUC has pretty reliable web servers, thanks.

Very few sites invented as much of the technology behind the web as they did.

Re:Linux?

[SA Stevens]

Did they really 'invent' that much of the technology, or did they just implement the stuff invented at CERN?

Re:Linux?

[Enrique1218]

Read the slide presentation. On pg 9, it shows a picture of Mac OSX at the terminal and on pg 16 it lists OSX under the software. However, linux will still operate the NFS server because Apple's implementation of NFS server only allows fifty clients. I am not sure whether a XServe or an old computer from the previous cluster will run it.

Re:Linux?

[SunFan]

However, linux will still operate the NFS server because Apple's implementation of NFS server only allows fifty clients.

This is interesting, because Solaris 10 would be the best choice for an NFS server (especially for a super-computer, where these things matter).

Re:Linux?

[peragrin]

Of course Slowaris 10 only runs on Opterons, and Sparcs. not PowerPC.

Advantage Linux. It runs on more platforms than you can remember.

Note x86 support is there but i wouldn't bet your hardware is supported completely. Between that and Sun's management telling the F/OSS community that the GPL developers don't matter they should get's lots of help.

Re:Linux?

[FidelCatsro]

As much as i dislike the way suns managment acts and as much as i love to use linux and GPL code , he has a point that linux is still not on par with solaris for an nfs server of this magnitude . Its great to use F/OSS whenever we can and if linux handels the NFS well enough then sure use it , that does not change the fact (that in my experiance) solaris still has better handling of NFS.
other that that i agree with you

Re:Linux?

[SunFan]

Slowaris

This name is long dead. People using it just look silly.

Also, the cost of a super-computer would accomodate a new Opteron server plus attached storage for NFS. Solaris 10's NFS implementation is the best in the industry, so why not use it? Various places, I've seen that Solaris 10 on a good 2-way server can easily saturate a gigabit network (meaning we need even faster networking to accomodate it).

Re:Linux?

[SunFan]

Spoke a bit too soon. It looks like they're already pushing 10Gbit [sun.com] for regular traffic and getting >200MB/sec on NFS.

Re:Linux?

[node 3]

does it run Linux

No, it runs UNIX.

<ducks!>

Re:Linux?

[SA Stevens]

It runs a branded UNIX?

When did MacOS 10 become a branded UNIX?

Re:Linux?

[pianophile]

When did MacOS 10 become a branded UNIX?

They talk about the "UNIX based" OS X here [apple.com], but don't actually say "OS X is a UNIX" per se.

Notice the lack of (C) and TM symbols on that page; isn't the word UNIX (C) the Open Group or whatever?

Re:Linux?

[menace3society]

OS X has an X in its name. Hey, it was a good enough reason for Xbox linux!

NFS only

[BSDimwit]

The slide show shows that the only thing Linux about the cluster is the NFS server. Seems that OS X 10.3.5 nfs server maxes out at 50 clients.

Re:NFS only

[oudzeeman]

I had serious issues with a 256 compute-node XServe G5 cluster and a NFS server running OS X... we eventually had to go to linux on our NFS server as well...

Re:NFS only

[citog]

University of Maine, I believe.

Sweet Dreams Steve

[intmainvoid]

Surely Steve Jobs dreams of scenes like this [uiuc.edu].

Anyone know what sort of crazy discount they get? It must be pretty good PR for apple.

Re:Sweet Dreams Steve

[UnknowingFool]

Anyone know what sort of crazy discount they get? It must be pretty good PR for apple.

While it might be good for PR to give UIUC a discount so that they would buy the system, remember that the cost per performance is already low. Virginia Tech built the X cluster for around $5 million whereas the computers next to it on the top 500 list cost $20+ million. There may have been a deal, but given the price tag is already low, it may not have been much.

Re:Sweet Dreams Steve

[prockcore]

Virginia Tech built the X cluster for around $5 million whereas the computers next to it on the top 500 list cost $20+ million.

Mellanox donated 1100 Infiniband adapters to Virginia Tech. At $3000 a piece, that's $3.3 million of interconnect hardware that Virginia Tech got for free.

Virginia tech can never be used as a cost comparison because they got so much stuff for free or discounted.

Re:Sweet Dreams Steve

[UnknowingFool]

True, free or discounted parts do lower the pricing but as far as Apple is concerned, the cost per performance for a clustered Apple grid is lower than a customer supercomputer built by IBM even if you count the adapters.

Re:Sweet Dreams Steve

[Stubtify]

Or he could have dreams of this [uiuc.edu] guy. I love the shirt... its almost too funny to be true.

Re:Sweet Dreams Steve

[Anonymous Coward]

Anyone know what sort of crazy discount they get?

Cluster node: $3,000 list, c. $2300 educational.

hmm ... are you sure?

[Triumph The Insult C]

Forbidden

Available bandwidth quota for this filesystem has been exceeded. (/benoc/cluster/album/cluster album-Pages/Image0.html)

Please, try again later.

Re:hmm ... are you sure?

[intmainvoid]

Forbidden

Well, I bet he hopes that it wasn't hosted on the cluster!

So is this the...

[ciroknight]

Small Mac? Perhaps the Double-Cheeseburger Mac?

Re:So is this the...

[yardbird]

640 nodes should be enough for anybody.

yeah, but can it play doom?

[acomj]

Oh, wait ... it can..

4.6 seems low

[Andy_R]

Virgina Tech are claiming 12.25 teraflops for their 1100 xserves [apple.com] ...and Sony are claiming 1 teraflops for a single PS3, so if you believe the hype we'll all be able to outrank these guys at a lan-party in a year or so.

Re:4.6 seems low

[CXI]

Tech's machines each have dual 2.3GHz processors as opposed to UIUC's dual 2.0GHz. Tech also appears to have a much faster network, and I imagine custom software for developing and running simulations. I guess it all adds up and Tech did a better overall integration job.

Re:4.6 seems low

[Rhys]

Worth a note: we don't have a 640-node cluster, it's really a 512 and 128 node clusters. Theoretical peak was I think 8 Tflop, and we got 4.6 or about 57%. VT's first entry only hit 54% if I remember what our parallel software expert told me right, so with more time to tune we'll probably get higher again for SC05's list. (VT's up to what, 61% or so now?)

Re:4.6 seems low

[CXI]

All true. VT also started out with G5 Desktops, not the servers, and upgraded. I think the reason for that was the desktop machines were easier to get a hold of at the time, in order to make the top 500 list entry deadline. :)

Re:4.6 seems low

[Game Genie]

Actually the desktops were the only ones for sale at the time. The G5 Xserve was not announced until after VT's system was installed.

What? No HAL-9000 jokes?

[joetheappleguy]

Are we forgetting the other supercomputer born in Urbana, Illinois?

Re:What? No HAL-9000 jokes?

[Rhys]

Computer Science already has HAL as a cluster name. Having the new one be HAL would only confuse matters. Especially since HAL contains the old turing machines!

Re:What? No HAL-9000 jokes?

[daeley]

Well, they could just change the name slightly. You know, shift the letters one place in the alphabet....

Of course, "GZK" is kinda hard to pronounce. ;)

Re:What? No HAL-9000 jokes?

[AcornWeb]

"You like your Macintosh better than me, ... don't you Dave?"

Awesome ad [psu.edu] :-)

Pictures inaccessible now

[Anonymous Coward]

Well, we didn't REALLY slashdot them. They simply went over the bandwidth quota.

Re:Pictures inaccessible now

[Rhys]

Netfiles will un-quota then every... hour I think Ben said, so try back in a bit. The PDF's kinda big.

The FAQ is wrong

[blamanj]

Why no Xwindows Support for Emacs?
Mac OS X does not come with Xwindows, but it does come with emacs. So, by default, emacs is not built with Xwindows support. On Turing, we use xemacs for windowed, x-enabled emacs. You can find it in /usr/local/bin, which should now be in the default path on all head nodes.

Mac OS X does in fact [apple.com] come with Xwindows. It may be that the default version of emacs doesns't use it, but to say it's not there is simply wrong.

Re:The FAQ is wrong

[FidelCatsro]

I suppose they mean it dosn't come as standard as you have to conciously decide to install it , and since they dont need the overheads of two windowing systems..

Re:The FAQ is wrong

[ThousandStars]

Mac OS X also has a Carbon-native Emacs binary. See http://www.apple.com/downloads/macosx/unix_open_so urce/carbonemacspackage.html [apple.com].

Re:The FAQ is wrong

[stoney27]

Yea but who cares about emacs, this is UIUC it was the home for Xemacs from about 1995 - 1997 there about, Emacs versions 19.11 through 19.14. And the programmer who worked on it is still at UIUC. So I think they are a bit Xemacs bias.

Of course there was also a Prof that was using Gosling Emacs druing that time :)

-S

Re:The FAQ is wrong

[stoney27]

Doh! That should read "XEmacs versions 19.11 through 19.14." darn I should have prof-read quicker.

I'm impressed.

[suitepotato]

No, really. This should be almost powerful enough to run next years (cr)applications from Macrodobe [slashdot.org].

Okay, seriously, it is impressive and another good illustration of the strengths of OSX over every horrible thing Apple called the Macintosh OS before they went over to the *nix base.

Oh yeah, that Star Wars t-shirt guy reminds me of what I looked like before I became older, grayer, and crustier. I'm so glad I stopped wearing shirts like that.

Nice Shirt...

[FlyingPostman]

Where did that Uber-Geek get that Star Wars T-Shirt? That guy is everything I picture when I think "nerd"

Re:Nice Shirt...

[NuclearBovineBoy]

Hey, don't diss him! I can personally attest to his superior social skills and sense of humor. And if that makes me an Ueber-Geek too (you could at least spell Ueber right ;) , so be it.

Re:Nice Shirt...

[Lars T.]

Try Über (I just hope the Umlaut doesn't only work in Preview but also in the actual post or I'll look really dumb;-)

Re:Nice Shirt...

[NuclearBovineBoy]

Fear not, the Umlaut works -- now I have to figure out how to do it...

Anyone else...

[Electroly]

Anyone else getting sick of seeing all these stories with pictures of people assembling hundreds of Xserves into massive clusters?

Yeah, me neither.

Re:Anyone else...

[izzo nizzo]

What I want to know is whether it could be profitable to lease computing time on one of these clusters - because that could be a cool job if you ask me.

I wonder how you'd manage to...

[jen729w]

A quote from UIUC's site:

The cluster is shared by many users, so users should be respectful of each other and not monopolize resources...

I wonder how a user would manage to monopolise (sic. - I'm English) this little baby? Suggestions on the back of a postcard to Please Just Send Me One Of These Puppies, You Really Won't Miss It, PO Box 67, ...

Re:I wonder how you'd manage to...

[menace3society]

Easy:

int main ()
{
for(;;)
fork();
return 0;
}

15 yrs and will be out desktop

[cylcyl]

I like the final slide.

Number of years when then entire cluster will be on your desktop: 15

Many other clusters

[drc1]

There are an increasing number of xserve clusters (http://www.macinchem.fsnet.co.uk/clusters.htm) why is this thought to be news worthy?

But this one replaces Linux

[AHumbleOpinion]

There are an increasing number of xserve clusters (http://www.macinchem.fsnet.co.uk/clusters.htm) why is this thought to be news worthy?

Perhaps because it replaces a Linux cluster? I haven't been following clusters but this is the first story where I recall Linux being replaced rather than Linux being adopted. Just a wild-ass guess.