Cool/Weird Stuff To Do On a Cluster? 608
Gori writes "I'm a researcher at a university. Our group mainly does Agent Based Modeling of interdisciplinary problems (think massive simulations where technology, policy, and economics meet). Recently, we managed to get a bunch of money for a High Performance Cluster to run our stuff on. The code is mostly written in Java. Our IT support people are very capable of setting up a stable cluster that will run Java perfectly. But where's the fun in that? What I'm trying to figure out are other, more far-out and interesting things to do with this machine — think 500+ Opteron cores, 2 GB RAM per core, a gigabit interconnect with some badass switches, a massive storage array, plus a bunch of UltraSPARC boxes. So at times when there's no stuff to crunch, I'd like to boot the thing up with a 'weird' system image and geek around in the name of science. Try fancy ways of building models, dynamically adding all sorts of hardware to it, etc. Have different schedulers compete for resources. Imagine a Matlab vs. Boinc vs. ProActive shootout. Maybe run plan9 on it? Most of us are not CE/CS people, but we are geeky enough. So, what would be the coolest and most far out thing you would do with this kind of hardware ?"
Not far out but.. (Score:5, Interesting)
Provide access to registered user projects (Score:2, Interesting)
Artificial Intelegence (Score:2, Interesting)
Being massively parallel it could be interesting to do some AI research. Like chess algorithms or something more fun perhaps a good automated tech support system via chat. Or trying to decode, capatacha. Map optimizations... a bunch of fun stuff. At least it would be for me.
I can tell you what I do with them :-) (Score:1, Interesting)
I run distributed answer set and SAT solvers on them and use these to do things like compose music, optimise machine code, (hopefully soon) find integer overflow bugs in code. Drop me a line if you're interested (mjb in the computer science department, University of Bath).
Or you could run an Eternity 2 ( http://www.eternityii.com/ ) solver...
Run Crysis on high (Score:3, Interesting)
Re:You can hope (Score:4, Interesting)
Although that was the easiest meme, It gave me an idea. Perhaps you could try some very-very-very high resolution ray tracing. I hear that such a process scales well with parallelism.
side note: captcha = shading
Brute Force Programming and Image Generation (Score:2, Interesting)
Instead of random, how about have algorithm of the hour and show what seeding or using it can do.
Run protein folding (Score:5, Interesting)
Give me an account...I'm being serious (Score:5, Interesting)
Why not... (Score:5, Interesting)
as some have said, donate the time to people would like to buy it but can't, or on odd days turn up 1000 SETI@home processes and see how far it gets?
Create a reverse Google bomb - Index every link to say 'george bush' from Google, read each page into memory hash the words, assigning value by count amongst all the pages, and then post the top ten words on GW that are not 'the' 'a' 'was' etc.
Perhaps comparing this to the same process on Paris Hilton would be a report that sparks SkyNet to life. I don't know. Seriously, if those two have 6 of the top ten words in common it would have to mean something.
If you have all that cpu sitting idle, that is the kind of weird shit nobody else would do, but also couldn't not read the report either. Perhaps you'll start something new for Google Trends? :)
Rendering (Score:5, Interesting)
I do burn-in testing on a lot of the machines we get at work, using hardware I can't personally afford.
My favorite test is to find scenes to render with a raytracer.. yafray is my favorite, runs on all major platforms. But not just any scene, it has to have all the details turned up to 11, contain extremely high detail (polygon counts drive up memory usage), and write out an absurdly large image.
Kind of whimsical but it's hard to not be impressed by an image 20,000 pixels square with perfectly accurate reflections. Who cares if I can only fit a fraction of it on my monitor. ;)
better yet (Score:5, Interesting)
Rosetta @ home or fold.it
Or you could try to thermally load them in patterns that produced different tones on the fans (or maybe an AM radio) record it then speed it up. Maybe you could make it sound like a baluga whale.
Or maybe you could implement a virtual machine cluster of 250 cores. The repeat the process till you see how many virtual machines you can stack on top of each other till it has the same speed as a single processor.
While this might sound stupid this would give us a rough estimate of how many watts per virtual world it takes and from that we could figure out how many layers deep in the simulation we actually live assuming the top level one is powered by something less than 1 sol of power.
or work out all possible moves in N-space tic-tac-toe. The only smart move is not to play.
Re:Coolest? (Score:4, Interesting)
Prime Numbers (Score:2, Interesting)
Open Science Grid (Score:3, Interesting)
http://www.opensciencegrid.org/ [opensciencegrid.org]
That way you can backfill the resource with jobs from VOs (Virtual Organizations, think projects/groups) you choose. It is similar to BOINC in the sense you can pick what science to support.
The idea is that the sharing will go both ways. You will give spare cycles to other users on OSG, and in return, you will be able to use spare cycles on other resources.
Spend some time on the OSG website reading the things under "Learn About Us". Also check out the research highlights to see what kind of science is being done on OSG resources:
http://www.opensciencegrid.org/About/Research_Using_OSG/Research_Highlights/Research_Highlights_Archive [opensciencegrid.org]
(Shameless plug - I'm part of this team) The good news is that OSG have an ongoing effort to help you join your resource to OSG, and help your users get going on the grid:
https://twiki.grid.iu.edu/twiki/bin/view/Engagement/WebHome [iu.edu]
Re:How about you don't? (Score:5, Interesting)
xkcd (Score:5, Interesting)
xkcd Network [xkcd.com]
Make applications (Score:5, Interesting)
Instead of randomly generating pictures or something as someone suggested above, why not make it generate say a 64k program, keep iterating through this until you get an executable that will actually RUN (Without crashing) and see what happens.
Seriously, there is good research money to be had. (Score:5, Interesting)
Re:Coolest? (Score:5, Interesting)
Started me rethinking my Doctorate plans in CS. If this pool of PhD's couldn't figure out that you were going to need to maintain the systems and have a number of people to work on it, then what the hell, book learnin didn't get them very far.
I'm sure though, that their thought was that once they got it in that surely the Administration would budget for maintenance.
This was an SGI system about 6 years ago.
Gori, what university is this anyhow? (Score:3, Interesting)
Re:Super photogenesis (Score:5, Interesting)
Interesting ramifications:
- Theoretically, there should be enough similar pictures to create movie of a ball-kicking fight between Lennon and Bush.
- You would have just also recreated the entire world's supply of past and future porn, albeit thumbnailed down to 255*255, and limited to 65k color.
- You would have produced at least one picture of a second shooter on the grassy knoll, even if there wasn't one to begin with.
Other options. (Score:5, Interesting)
A variant on the rendering theme: Instead of simulating rays, simulate light wavelets and the modern theory of light. You should have enough compute power there to render reasonable scenes using such a technique.
Fluid dynamics: This is a popular one, and NASA offer source codes for free for subsonic, supersonic and even hypersonic flows. In fact, they offer quite a number of subsonic ones. They're also not the only source. There's several open-source CFD packages, ranging from river simulations to aircraft simulations.
Supernovae simulation: There are packages (freeish, rather restricted in access) that allow you to simulate thermonuclear and supernovae explosions within stars. The restrictions are for rather obvious reasons, even though the odds of anyone nasty obtaining a star is, well, unlikely.
COLOSSUS: There are still a couple of ENIGMA ciphers that have never been broken, which can be obtained along with the algorithm Colossus used in World War II to crack such codes. You could complete the set and maybe discover some lost secret (yeah, right).
BLAST: Other posters have suggested renting out the computer time, but that just transfers the problem of what to run, rather than solving it. BLAST, or one of the MPI-based variants thereof, is an exceedingly popular tool for examining nucleotide sequences, but as the databases grow ever-larger, the demand for ever-more information also increases, creating a need for significant compute power to produce the volume of results desired.
Re:better yet (Score:5, Interesting)
Re:Reduce Your Carbon Footprint (Score:3, Interesting)
That actually does sound interesting!
Even with one node on all the time... you could pretty reasonably pull that off with the rest of the nodes. You could even have it dynamically scale the "awake" part of the cluster to match the number of jobs in the queue or the utilization of the awake parts. Just imagining going from 200W to 30-40kW and back automatically, on demand... yeah, that'd be handy.
You could also tie into environmental factors ... scaling back the cluster if the HVAC is underperforming and the room's getting warm. Or tying in power and cooling load balancing, powering up appropriate systems to keep the room from getting hot and cold spots.
How much CPU do ya got? (Score:4, Interesting)
If an infinite number of monkeys......
Write a program that creates random code and tries to run it, then starts again. Eventually, we'll have the Singularity and not have to worry about finding cool things to do.
Re:Coolest? (Score:3, Interesting)
This is pretty much the way that grant money works in many disciplines. Don't budget for maintenance or "upgrades," or else the original bid is going to come in far too high to ever be approved in the first place.
Physicists often blame the cancellation of the SSC [wikipedia.org] on the fact that it was realistically priced when it was canceled (and $2bn had already been thrown down the toilet).
Re:How about you don't? (Score:3, Interesting)
"Come on, who's better at pissing people off than a smug, whiny hippie like me?"
A smug, whiny, know-it-all smartalec like me.
Re:How about you don't? (Score:3, Interesting)
Simulate the oil futures market and figure out with precision the effect the "Enron loophole" has had on the price of oil. Publish results.
Grid clients (Score:2, Interesting)
I've worked with supercomputer class systems in the past, common question - "what can I do now?"
Two suggestions:
1. Set aside the first couple "downtime" sessions to run your crazy "OMG it actually worked" stuff out of the way. The stuff you'll brag about for years to come.
2. Every time thereafter, set it up with a grid computing client. Seriously. It may not take advantage of the interconnect fabric, but you will be doing your fellow humans a great service.
Check out http://www.worldcommunitygrid.org/ for some ideas (from a big company whose sponsored code will not cause infinite loops on the nodes....).
SETI is IMNHO a complete waste of time.. if or when the aliens want to be found, it will probably be in Hollywood ID4 style and not some little green guy screetching "Can you hear me now?" on all frequencies and modulations.
Re:Coolest? (Score:3, Interesting)
Sadly, I have also seen waste like this. I know of a cluster that can only be 75% of the nodes can be turned on, because the building to house it was delayed a year and it's in a temporary space, but the money was allocated and had to be spent before the space was ready. If the money wasn't spent, the money would be taken away. I've been a cluster admin for a cluster that barely had 10% usage and I almost cried. Interestingly, this ties in with the "parallel programming is hard" problem. Some researchers don't bother to parallelize or optimize their applications because after the paper is written, the code isn't run any more. If you only need to run a few times to confirm your hypothesis, then why bother optimizing if the time between runs isn't terrible.
Java? (Score:2, Interesting)
Couple of (half-baked) ideas... (Score:4, Interesting)
Reading Slashdot dulls creativity sometimes... I suggest you deprive yourself of sleep, overdose on caffeine, read something by Neal Stephenson (Snow Crash or Diamond Age), and meditate on your question with a 2-3 geeky buddies of yours around 2am.
1) Find a picture/piece of code/ISO image/etc. you'd like to compress. Treat it as a very big number. Now find a prime number that comes very close to it. Compress it by treating it as the Nth prime + the remainder. Repeat for the remainder See if the compression is any better than bzip. (I think i saw something like this done for the DeCSS code once...)
2) Find something to optimize via brute-force. (My favorite is neural nets predicting time-series). Run a distributed simulated annealing algorithm. (Run an instance of the algorithm on every core, check every N cycles to see which core is more "optimal", share the parameters with the rest of the instances).
3) Create an interactive multi-user raytraced environment. Get someone to bring in an ImmersaDesk or a CAVE. Stun all visitors.
4) Model a giant neural network. Teach it to do something cool in real-time with your favorite training algorithm...
5) Get dictionaries from 50-60 languages. Write something to correlate similarity of words meaning the same thing. Make pretty graph clustering them by similarity. Post pretty graph on Slashdot.
6) ???
7) Profit!
Re:How about you don't? (Score:2, Interesting)
Re:How about you don't? (Score:5, Interesting)
Actually, we are designing a sophisticated monitoring and control system that will power down all the nodes that have nothing to do. Sustainability is *the* reason why we build models in the first place. Most of them are used to estimate CO2 emissions of different future industrial systems.. so yeas, you are right, this is an important issue...
How about you do? (Score:2, Interesting)
Try to find the least power consuming configuration that works under full load.
Then, starting from that point try to find the best way to automatically adjust between idle and full load (think disabling/enabling cores).
After you're done compare the power consumption with that after your modifications, translate that into management speak ($/year saved) and go get your fat bonus.
Oh, once you get it you owe me a beer
Re:Vista? (Score:3, Interesting)
You may laugh, but Mac OS X Panther (2003) ran just fine on a 500Mhz processor with 256MB RAM and a Rage 128, although its install size was more than 500MB and the installer was more complicated than dd.
I know 2003 was quite a while ago, but it just goes to show that modern OSs don't have to require GBs of RAM and malti-core multi-GHz processors.
Re:high performance java? (Score:3, Interesting)
If you're just an armchair critic who has never had to write serious multi-threaded code, it's very easy to say "do it in C". I'm coding in C now for an embedded platform, but if it was a desktop PC, you can bet I would choose to write in Java. Trade-off of a longer dev cycle is just not worth it for the performance increase. If you can even find anyone who can write good multi-threaded code in C.
Re:How about you don't? (Score:2, Interesting)
It's a matter of perspective, i don't think anyone would complain if i told you not to throw your garbage wide on the street in your neighborhood because everyone would recognize it as a useless, degrading act that would damage (in this case just annoy but follow my thinking please..) the whole community. Your neighbors probably wouldn't be that happy about it and i'm hoping you wouldn't tag their complaining as a mean of superimposing their will on you.
Energy is pretty much the same, it's a common property like air, water, ground and every other crucial resource for the sustaining of life (the fact that we pay water, energy and ground doesn't mean they aren't key requirements for life and thus needed by everyone). The problem doesn't lie in the usage itself but more on the fact that several of the said resources don't come back once you used them all. This means that every time you use even a little bit, you're subtracting that bit from the collective and thus it would be smart to use it meaningfully, since it's something quite precious (being unique, one-way).
The argument about avoiding to post on slashdot to save energy (even if it was preposterous) has its meaning but before questioning ourselves about something we _do_ with our energy-wasting computers, better is to start not let them waste energy when it's not needed, even just for a matter of money saving. And that applies to pretty much every piece of technology that does use energy. Saving *any* kind of energy is just a proper community behaviour and a pretty good way to save money.
My way in all this is to try as much as i can to not waste energy (in any way or form) and to give meaning to what i do when i use it. The habits are something very hard to change but often it's just a matter of will.. the average power of US owned cars is quite higher than European-owned ones and right now there isn't a practical reason for it. Changing the habits to buy less power-hungry cars would be just a matter of will since that much power usually isn't a requirement (if you're a ranger with mountain-patrolling duties you're excused).
Most of IT-related electricty-produced pollution comes from datacenters, they have way more density than before and very,very efficient systems still aren't mainstream in that market frame. Home and Office computers nonetheless counts in the millions (at least) and promoting low power ones when you don't need more and adopting good behaviour when you own one (the famed standby issue but even activating cpu throttling) are all good measures that don't take much effort and _will_ find their place in the ever-evolving world in due time, not to mention in your pocket at the end of the year.
I don't think this kind of thoughts are hippie's, i wear proper cloths, i don't do anything more psychoactive than coffee, i'm not familiar with molotov bottles, i don't like jimi hendrix that much and i am a very boring person... i'm pretty sure i can't qualify as a hippie. Sorry for the length but i wanted to be sure not to be considered a flamebait.
RegardsRe:better yet (Score:3, Interesting)
Try to solve go.
Re:Super photogenesis (Score:2, Interesting)
Somewhat smaller than 1x2 pixels. If we assume three 8-bit channels, then each pixel has 2^24 possible combinations of color. While this would take only a little while to generate on your home computer (a few seconds or minutes), a 1x2 image has 2^48 possible combinations. This is roughly 10^15 (million billion). If we assume some sort of TGA encoding, this takes 16 words for each image, thus requiring about ~2^53 words to store. That requires petabyte storage. The latency is what kills you in writing these to disk, but if you allow the machine to write a billion images to disk per second it only takes about 50 days-or-so to write the entire set. A 1x3 image-set would be intractible, as they would require ~Avagadro's number of images to be produced and written, e.g., about tens of millions of petabytes, and on order of millions of years to write to disk. A 2x2 image-set is well beyond our current comprehension of storable information, and probably beyond the energy output of the sun, and 4x4 would require more atoms than there are in the universe, i.e., generates 2^(16*24)~~10^120 images.