Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Supercomputing PlayStation (Games) Hardware

How To Build a Homebrew PS3 Cluster Supercomputer 211

Posted by timothy from the slot-a-tab-b dept.
eldavojohn writes "UMass Dartmouth Physics Professor Gaurav Khanna and UMass Dartmouth Principal Investigator Chris Poulin have created a step-by-step guide designed to show you how to build your own supercomputer for about $4,000. They are also hoping that by publishing this guide they will bring about a new kind of software development targeting this architecture & grid (I know a few failed NLP projects of my own that could use some new hardware). If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike (where they make the most profit)."
This discussion has been archived. No new comments can be posted.

How To Build a Homebrew PS3 Cluster Supercomputer More

How To Build a Homebrew PS3 Cluster Supercomputer

Comments Filter:

  • Oblig. OMG... (Score:2, Funny)

    by Anonymous Coward
    something to finally run Vista?

    • Re: (Score:2, Funny)

      by windsurfer619 ( 958212 )

      At first I laughed... But then I realized that, no, Vista won't be able to run on this.

      Vista doesn't support the PowerPC architecture.

  • ibm (Score:2, Interesting)

    by sreid ( 650203 )
    why would ibm be involved in this if it means they will sell less servers?
    • FEWER servers! FEWER! Aauughhhh!

      • Re: (Score:3, Funny)

        by Anonymous Coward

        This isn't really the place to start criticizing grammar and spelling, unless you REALLY want to live a life full of frustration and torment....?

        (Though it could be worse, I suppose - you could go to digg etc. intsead...:p)

    • It's showcasing the usefulness of IBM's cell processors for exactly this kind of thing. They have very good reason to be involved as it may mean that there is interest in using their processors for smaller computers at a higher volume to do modeling and research.

    • Re:ibm (Score:5, Insightful)

      by fuzzyfuzzyfungus ( 1223518 ) on Wednesday December 17, 2008 @07:37PM (#26153317) Journal
      IBM is probably banking on the existence of people who want Cell processors in systems with more than 256megs of RAM. Other IBM value-adds would presumably include rack mountability, support for netbooting and other convenient management stuff, and so forth.

      If your application leans almost entirely on the CPU with very little need for RAM, and you have an army of screwdriver monkeys(or grad students) to do all the legwork, the PS3 is an excellent deal. If you need something with RAM capacity that wasn't a joke in 2001, and/or management features that won't have you tearing your eyes out when you have 10,000 of them, then IBM smells opportunity.

      • Re: (Score:3, Informative)

        by dbIII ( 701233 )
        If you wait out the two weeks of used car salesman tactics from a place like Mercury that can sell you cell processors in systems with more than 256megs of RAM you'll find out that unless you have an endless budget you are probably better off with the ten quad core Xeon systems you could get for the same price.

        That is why a system made of game consoles makes a lot more sense than very similar hardware in a rackmount case. Other cell hardware has been priced into complete irrelevance by salesfolk having too

    • IBM is the first company to see the writing on the wall and invest in new markets, even if those markets invalidate their current holdings. They went from a typewriter company, to a mainframe company, and an operating system company, to a PC company, to a server company, to a virtualization company, and now a SaaS company.

  • Does Sony make any money on PS3 hardware sales? Last I heard they were selling them at around $100 under the cost of production.

    • Re: (Score:2)

      by Shados ( 741919 )

      I dont know about the newer models without the backward compatibility and stuff, but previously, they were definately net loss. The -last- thing Sony wants is to sell a million PS3 with 0 attach rate. Of course, those numbers would still count to impress developers, and may be a catalyst, but....

      • Re:Subsidized Supercomputers (Score:5, Insightful)

        by timeOday ( 582209 ) on Wednesday December 17, 2008 @07:21PM (#26153149)
        The number of PS3's sold will never be enough to hurt Sony's bottom line, but will boost the image of the console. Having credible scientists call your product a "supercomputer" is worth something. Does Ferrari's Forumla 1 racing team pay for itself? Nah, it's an investment to promote an image.

    • Re: (Score:3, Informative)

      by makapuf ( 412290 )

      AGAIN, revenue of console sales is not N*const (positive or negative), but const1+N*const2 where const2 is negative (it's a gain per console) but upfront costs=const1(R&D, licences ...) are big. So the fact that the total is negative implies const is negative, but in fact it's mostly that N*const2 is still less than const1. (I hope this makes sense to some at least)

      • Re: (Score:2)

        by nasch ( 598556 )

        When someone mentions a product as losing money, they're usually talking about negative marginal revenue. After all, a brand-new fantastically profitable product will often be "losing money" at the beginning if you count fixed costs, but nobody would describe it so.

    • The Wii is the only current generation console sold at a profit.

  • Why use PS3s? (Score:4, Insightful)

    by JanusFury ( 452699 ) <.moc.liamg. .ta. .ddag.nivek.> on Wednesday December 17, 2008 @06:35PM (#26152685) Homepage Journal

    Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either. It seems like it would be cheaper to build a cluster out of commodity PC parts, and maybe use GPUs+CUDA to get more muscle without having to completely hand-roll your own accelerated computation code (since CUDA is roughly C). I can't imagine that the PS3 would end up cheaper for these purposes, considering it includes a Blu-Ray player along with a bunch of other things you're not going to be using.

    • ....About the cheapest computer you can build/buy that would be of any use as a supercomputer would be $200, add in a $150 GPU, and thats $350, about the price of a used PS3. Most supercomputers need fast CPUs, not a ton of RAM (though, the more RAM the better), and so it becomes that a PS3 is about the same as doing it with commodity computers only the PS3 has a much faster CPU.

      • Re: (Score:2)

        by JDevers ( 83155 )

        Well, that is all true except most simulations eat a ton of RAM (we aren't comparing 20 GB to 25GB here, the PS3 has 256MB of memory) and the PS3 does NOT have a faster CPU. If you think it does, look at the folding@home stats for a PS3 versus a mid/high end GPU. The GPUs are really what are interesting here, the main CPUs of both systems are slow enough to make them of no interest when only talking 8-20 units...

        From the folding@home Wiki:
        as of August 24, 2008, GPU clients accounted for the majority of ent

        • Yes, the GPU client is faster, but it's limited to the kinds of WU's it can do, compared to the PS3 client, as the FAH site says:

          What type of calculations the PS3 client is capable of running?

          The PS3 right now runs what are called implicit solvation calculations, including some simple ones (sigmodal dependent dielectric) and some more sophisticated ones (AGBNP, a type of Generalized Born method from Prof. Ron Levy's group at Rutgers). In this respect, the PS3 client is much like our GPU client. However, the

      • Re: (Score:3, Interesting)

        by afidel ( 530433 )
        The best MIPS/watt for CUDA is probably either the 9600 GSO or the GTX280 depending on whether you're memory or processor constrained. The 9600 can be had for about $75 for 768MB variety (forget the 512/1024 parts they much lower performing) and has 96 stream processors running at up to 650Mhz. The GTX280 costs about $400 and has 240 650Mhz stream processors (though I believe they might be slightly more advanced then the ones on the 9600 I'm not sure how much of that is exposed by CUDA). Power usage is 46W

    • Re:Why use PS3s? (Score:5, Informative)

      by drinkypoo ( 153816 ) <drink@hyperlogos.org> on Wednesday December 17, 2008 @06:55PM (#26152875) Homepage Journal

      Why would you want to use PS3s for a homebrew supercomputing cluster if it means you have to write and optimize code for the SPEs to get benefit out of it? The PS3's linux environment doesn't let you utilize the GPU or all of the built-in SPEs and it doesn't have a lot of RAM available either.

      Well, I'll bite; if the cell is the fastest processor for your workload, the PS3 is the cheapest way to get one, even at only six usable SPEs and no GPU. Doesn't the PS3 have GigE? That's plenty fast enough to shovel data in and out of the system.

      • Well, I'll bite; if the cell is the fastest processor for your workload, the PS3 is the cheapest way to get one, even at only six usable SPEs and no GPU. Doesn't the PS3 have GigE? That's plenty fast enough to shovel data in and out of the system.

        What workload is actually faster on a Cell than on a modern quad-core CPU or video card? I mean - it's possible that such a workload exists, but the niche between a general purpose CPU and the hundreds of FPUs in a video card has got to be pretty damn small.

    • Re:Why use PS3s? (Score:4, Interesting)

      by ookabooka ( 731013 ) on Wednesday December 17, 2008 @07:06PM (#26153001)
      If I recall correctly, Sony sells hardware either at-cost or at a slight loss because they make their money on the games. I know this was true for the original xbox as modded xbox clusters were demoed as extremely cost efficient compared to making the computers yourself. I used a moded xbox as an early TiVO as it was way cheaper than making a similar setup myself.

      • Re: (Score:2)

        by rm999 ( 775449 )

        Yeah, but a large part of the cost is the bluray player, which is useless in a supercomputer. I guess you could probably sell the drive/laser for 100 bucks to offset your costs.

    • The Blu-Ray drive and the controller are the only things in the system you won't really use, at least much. The rest of the system is a computer, even if it is an unusual architecture. I don't know if the system can install an OS over a USB drive or CF card vs. optical disc, I've never tried to install Linux.

      I'd say it's a very powerful computer for $400, assuming you can program for it.

    • Re:Why use PS3s? (Score:5, Informative)

      by ASBands ( 1087159 ) on Wednesday December 17, 2008 @07:32PM (#26153267) Homepage

      since CUDA is roughly C

      Not quite. CUDA looks a lot like C in that it has C-family syntax but the biggest limitation it has is that there is no application stack - which means no recursion. CUDA also lacks the idea of a pointer, although you can bypass this by doing number to address translation (as in, the number 78 means look up tex2D(tex, 0.7, 0.8)). The GPU also has other shortcomings, in that most architectures like to have all their shaders running the same instruction at the same time. For this code

      if (pixel.r < pixel.g){
      //do stuff A
      }else if (pixel.g < pixel.b){
      //do stuff B
      }else{
      //do stuff C
      }

      The GPU will slow down a ton if the pixel color causes different pixels to branch in different directions. Basically, the three sets of shaders following different branches of that code will be inactive 2/3 of the time.

      In the Cell, you really do just program in C with a number of extensions added onto it like the SPE SIMD intrinsics and the DMA transfer commands (check it out [ibm.com]). The Cell really is 9 (10 logical) processors all working together in a single chip (except in PS3, where there are only 7 working SPEs). Furthermore, your 8 SPEs can be running completely different programs -- they're just little processors. Granted, you have to be smart when you program them to deal with race conditions and all the other crap you have to deal with for multithreaded programming. The Cell takes about 14 times longer to calculate a double precision floating point than a single (and there aren't SPE commands to do four at once like you can with singles).

      So which is more powerful? It really depends what you're doing. If your task is ridiculously parallellizable and doesn't require the use of recursion, pointers or multiple branches, the GPU is most likely your best bet. If your program falls into any of those categories, use a Cell.

      • Not quite. CUDA looks a lot like C in that it has C-family syntax but the biggest limitation it has is that there is no application stack - which means no recursion. CUDA also lacks the idea of a pointer, although you can bypass this by doing number to address translation (as in, the number 78 means look up tex2D(tex, 0.7, 0.8)). The GPU also has other shortcomings, in that most architectures like to have all their shaders running the same instruction at the same time. For this code

        if (pixel.r

        Yeah, you win

    • Not that I am, but if I was some home/small business artist/modeller who needed some serious render time to generate the frames of a computer animated movie/demo, I'd be making one of these clusters ... It would be perfect for this kind of thing.

      • Re: (Score:2)

        by afidel ( 530433 )
        Nope, not nearly enough ram for even a moderately complex scene's geometry let alone the textures unless you want your output looking like a game (IE most graphic artists will want photorealistic output which is more than a game console is capable of).

        • Re: (Score:2)

          by J05H ( 5625 )

          Depends on the specific code - to my knowledge no one has written/ported a 3d renderer to the PS3Cluster architecture yet - so you should get to work on it. 8) To get realistic textures does require a lot of RAM or a lot of swapping. One thing that could help in this context is to have a big block of NAS on the same network - and treat part of it as a RAM disk or texture buffer. Not necessarily efficient but could work following any of several weird render schemes.

          I only suggest the block of external storag

    • Re: (Score:2)

      by J05H ( 5625 )

      When I installed linux+MPI on the test PS3 it recognized all the processors - pretty cool seeing 8 little penguins pop up. From what Chris said the programming is fairly generic C/C++ to utilize the whole console. It's apparently not that hard and PS3s are dirt cheap (compared to supercomputers or even blade servers).

      Josh

    • Feel free not to believe it, but actually doing your research might be smarter.

      And which part of the GPU not being fully exposed to Linux is relevant to supercomputing exactly?

    • Why would you want to use PS3s for a homebrew supercomputing cluster

      IIRC, a general purpose CPU has a small data cache and a large instruction cache, coz you can never be 100% sure which instruction is likely to be executed next. PS3s have large data caches and small instruction caches, because they spend much of their time executing a small number of instructions over a large set of data, that is, graphics rendering.

      If you are doing any sort of mathematical simulation, you can likely express your numeric

  • Why PS3s? (Score:2, Interesting)

    by whyloginwhysubscribe ( 993688 )
    I don't understand why this isn't possible with normal PC hardware - what is special about the PS3 - or is it just because it is better value for money?

    • Re:Why PS3s? (Score:5, Informative)

      by Darkness404 ( 1287218 ) on Wednesday December 17, 2008 @07:01PM (#26152959)
      A) Although the cell is a pain to code for, it is much better than whatever PC you can get for ~$400 which will probably contain a mid-to-low-range dual core x86 CPU, whereas the PS3 gives you a Cell CPU which is much, much, faster than the x86 CPU.

      B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

      C) PS3s are the new fad. It isn't going to be hard to set up a supercomputer cluster with PS3s compared to using a mismatch of older computers because again, the PS3 is uniform.

      • B) PS3s are uniform. Other than HD differences, a PS3 built in 2008 will be the same PS3 built in 2012 (assuming the PS3 lasts that long) this allows for a uniform cluster without worrying about differing parts (for example, the Core i7 built in 2008 will not be the same as the Core i7 built in 2012 and getting a 2008 Core i7 is going to be a pain)

        Don't rely on this - there are large hardware differences between early PS2 and later PS2 models as manufacturing tweaks and cost reduction packages were applied to the production process, to the extent where some games refused to run and some features were changed. I don't expect Sony to act any differently with the PS3.

  • I wonder whether this how-to will enable me build a cluster consisting of PIIs. I have 11 lying around.

    • Re: (Score:2)

      by armanox ( 826486 )
      Yes and no. The concepts are the same, but I'd use a more lightweight distrobution then Fedora to do it on P3's (I've got a stack of P3 1GHz laying around too)
    • I wonder whether this how-to will enable me build a cluster consisting of PIIs. I have 11 lying around.

      Imagine a Beowulf cluster of those.

      Then build one.

      I doubt the performance will be awfully impressive by today's standards - you'll be outdone by pretty much any 2008 desktop machine - but it'll be an interesting project anyway. Let us know how you get on.

  • Limited use (Score:5, Insightful)

    by idiot900 ( 166952 ) * on Wednesday December 17, 2008 @06:44PM (#26152765)

    Couple issues with this as an alternative to the garden-variety x86 cluster connected with InfiniBand:

    Slow network interconnect. For problems that are not trivially parallel, network latency is usually a big deal. Ethernet doesn't cut it.
    Lack of RAM. 'Nuff said.
    Have to care about Cell and PS3 architecture. The codes ("codes" has a slightly different meaning in the context of supercomputing) have to be modified to take advantage of this very specific architecture. Software always outlives hardware, so in the long run the effort may not be worth it.

    That said, it's really cheap. If your application isn't held back too much by these issues then enjoy your insanely cheap cluster!

    • Comment removed based on user account deletion

    • Re: (Score:2)

      by thaig ( 415462 )

      Well there are 7 SPEs on a chip right? So the latency right there must be pretty low?

      His problem probably *is* "trivially" parallel so perhaps he was right to do what he did?

      Later he can "upgrade" to an IBM PowerXCell 8i based blade.

  • Imagine a beowulf cluster.. (Score:2, Funny)

    by Anonymous Coward

    Imagine a beowulf cluster of those!

    • Imagine the ratio of expense of "one" of these to a beowulf cluster of x86s created to provide equivalent computing power. Just imagine!! Hmmm...on second thought it's probably only like 1 to 2.5 or 3...

  • If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike...

    Why "might not"? Are you implying that people may be building PS3 clusters just so that they can sneak into the lab at night and have big gaming parties? Because I can totally see that.

  • At any given time a supercomputer is one order of magnitude world fastest computers. This may have been a Year 2000 supercomputer, but far from one now.

  • Power and maintenance? (Score:3, Interesting)

    by lemaymd ( 801076 ) on Wednesday December 17, 2008 @08:17PM (#26153625) Homepage
    Researchers pay not only for the initial capital outlay required to install a supercomputer, but also for its power, cooling, the building it resides in, and its maintenance. This PS3 cluster might be cheap from the researchers' standpoint if they don't pay for any of these things directly, but I imagine their departments won't be real thrilled if a bunch of researchers start building their own individual "cheap" supercomputers! Those issues aside, it sounds like they're doing pretty cool stuff with those machines, so maybe more supercomputers should be cell-based!

  • I wish I had one (Score:3, Informative)

    by uassholes ( 1179143 ) on Wednesday December 17, 2008 @08:23PM (#26153689)
    For the dick licks that say it's useless, I guess you missed all the previous articles about scientists who have been doing the same thing:

    http://www.physorg.com/news92674403.html [physorg.com]

    http://dgl.com/itinfo/2003/it030528.html [dgl.com]

    http://www.lbl.gov/Science-Articles/Archive/sabl/2006/Jul/06.html [lbl.gov]

    http://folding.stanford.edu/English/FAQ-PS3 [stanford.edu]

  • What a ripoff! (Score:2, Insightful)

    by Anonymous Coward

    What a complete farce! Here I was all excited to go see this PS3 cluster "guide". From TFA:

    "Found at www.ps3cluster.org, the resource fully illustrates how to create a fully functioning and high performance supercomputer with the Sony Playstation 3."

    And what is actually *on* the site?? How to install Linux on a PS3 (as if there weren't any guides for that out there already). Then, they show the magical touch where they download the stock Fedora Open MPI implementation, and configure it using all *TWO THREAD

  • Games (Score:3, Funny)

    by daybot ( 911557 ) * on Wednesday December 17, 2008 @09:42PM (#26154435)

    If this catches on for research institutions it may increase Sony's sales, but they might not be seeing the corresponding sale of games spike

    Come on - that's the whole point. This is what you'll need to run the PS3 version of Crysis!

  • I've thought about Folding @Home and I've always wondered why can't there be a diy distributed computing server that could be setup. Something like this PS3 cluster but could be replicated with any home pc.

  • My setup for PS3Cluster (Score:3, Interesting)

    by J05H ( 5625 ) on Wednesday December 17, 2008 @10:27PM (#26154933)

    I helped Chris with the documentation, testing and image capture on this project. I see some "it doesn't do this!" comments above - please remember this is a young project that started out of one researcher's need to solve a specific type of problem. If you want to see this advance, it's all open source so start hacking.

    So my setup:
    1 40Gb Playstation3 w/ HDMI cable out and keyboard
    Hauppauge HDPVR digitizer
    PC running Windoze and Photoshop
    TV hanging off the HDPVR for reference

    Software as described on PS3Cluster.org including Geoff's Cell libraries, boot image on USB and Fedora 8 for PPC.

    Plugged everything together, installed Fedora 6 the first time around since we knew that worked, then redid it with Fedora 8. Added the MPI libraries and ran the little Pi test code. Digitized the whole install as video, proofed out the process in terms of instructions. Did frame grabs from the video, cropped etc in Photoshop. Lots of work, totally worth it seeing the project posted here.

    Oh, and it runs X - kinda cool having Firefox running on a game deck.

    Enjoy,
    Josh

    • Course it's cool having Firefox on a game console, though I had X and Firefox on my PS2 5 years ago, though it was Firefox was still Phoenix then. :-)

  • This is probably a silly question (Score:3, Interesting)

    by Chrisq ( 894406 ) on Thursday December 18, 2008 @04:07AM (#26157347)
    This is probably a silly question but why/how are they running PPC Linux (which is presumably for the Power PC) on PS/3s which have cell processors?

    I guess that either the PS3 has a PPC chip as well, or it runs some sort of emulation mode. I can't find either documented.

  • Since we are talking about this,does any one is using or have any newer news on the molecular simulator NAMD on the CELL Processor [uiuc.edu]? The official development stalled two years ago as its maintainer sinked into other projects, but I do actually help a team with a PS3 cluster which would be very interested in getting NAMD working under full load there.

  • ...a beowulf cluster of that. Oh, wait!
  • IBM BladeCenter QS22 [ibm.com]

    If you want your cell system without the PS3's, get a couple of these. Each comes with two Cell 8i CPUs in a 1U case. Upgradeable dedicated processor memory slots and general use RAM slots. A bit more expensive than the PS3's, but might be easier to get the institutions to pay for...
    • A bit more info I found on the Cell Wikipedia Page

      In 2008, IBM announced a revised variant of the Cell called the PowerXCell 8i, which is available in QS22 Blade Servers from IBM. The PowerXCell is manufactured on a 65 nm process, and adds support for up to 32GB of slotted DDR2 memory, as well as dramatically improving double-precision floating-point performance on the SPEs from a peak of about 14 GFLOPS to 102 GFLOPS total for 8 SPEs.

      So, I configured 4 QS22's with 32GB RAM each and it came out to $4

Slashdot Top Deals

No spitting on the Bus! Thank you, The Mgt.

Close