Building the World's 4th Fastest Supercomputer 75
ngkabra writes "In November 2007, a previously unheard of supercomputer called EKA, built by CRL, India came out of nowhere to become the 4th fastest supercomputer in the world. It is also the only supercomputer in the top 10 that hasn't taken any government funding — which means it has no strings attached against commercial exploitation. That is one of the reasons why Yahoo! chose EKA for the cloud computing research that they announced at the Hadoop Summit earlier this week. Yesterday, I attended a presentation by the team that built EKA, and they touched upon a lot of the technical details of EKA, and the challenges faced in designing and building it, which makes for interesting reading."
Re:This is old news (Score:2, Interesting)
Thought experiment time (Score:4, Interesting)
Let's start with the switches. You can parallelize network traffic - fragment the packets, stick full headers on each fragment, switch the fragments in parallel, reassemble, have the queue re-order inbound traffic by sequence number. (That last step isn't too hard, you have some fraction of the sequence number and the full fragment number map to a unique address in a permanently allocated ring buffer. Copy the payload to that address and the packets are in sequence order, not delivery order.) So, instead of having individual switches that are fast, bank the switches up and have the combined virtual switch run very fast. You can do that on exportable commodity hardware.
Or, you could sneak through Homeland Security (who have much more interest in nipple rings than dangerous weapons) a bunch of 24-channel 5Gb/channel InfiniBand switches. You wouldn't need many of those to get a decent Quake LAN party, and not many more before you could run weapons design software. It is unclear how many are required to run Vista, once service pack 1 is installed.
Interconnects. Obviously, InfiniBand is hellishly fast. So is SCI. 10 Gb ethernet, ideally with iWarp extensions, would be much slower but still perfectly good for a commodity cluster. If you scrap the idea of having machine-to-machine communication and do memory-to-memory communication, you could actually use PCI-e 2.0 as one gigantic interconnect. Ideally, you'd have the memory appear as two separate devices - slave and master - so that direct memory-to-memory RDMA could be initiated. A lot of very similar work has already been done by US supercomputer giants, and given how many have either been bought, gone bankrupt, or otherwise vanished, it's reasonable to suppose large quantities of such RAM could have "migrated" overseas.
CPUs - well, there are some respectable 16-bit pile-of-pc clusters. One was reviewed on Slashdot some time back. Even a cluster of Cell processors, if large enough and well-enough programmed, could be very effective. A hostile nation wouldn't need high-end 4x4 multi-threaded multi-core SMP systems, although again given how juvenile airport security is, I can't imagine it would be hard for someone to export, say, a couple of hundred motherboards at that spec.
Ok, what about OS? Who needs one? Anyone with a copy of OSKit or something similar can work at almost bare metal levels as if they had a full OS. If they did want a full OS, then NetBSD or MOSIX would be quite sufficient. Or they could take an OS project like Exopc and add high-performance networking to it.
Software? If you've a decent copy of BLAS, LAPACK and some solvers, tightly optimized for the platform, you're set. Those core maths functions are critical. Since the functions and API are fixed, it would not be impossible for someone wanting raw power to have put them into an FPGA, SoG or ASIC. Collective operations are also nasty, but they too can be done entirely in hardware, giving you orders of magnitude speedup over conventional software solutions. Synchronizing is the third killer, but there are meta schedulers to handle that and you could again place those on dedicated hardware.
In short, although I couldn't afford to build a top 500 machine, it is only the affording of it that is a problem, and foreign countries are quite well aware of that. Especially after China built its first (publicly-announced) Government-funded Beowulf. Supercomputing is easy, it's the price tag that isn't.
Odd storage requirements... (Score:3, Interesting)
# 80TB storage. Simple SATA disks. 5.2Gbps throughput.
So using the 80TB * 1024GB/TB / 1800blades gives us about 45GB/blade. If they're using "simple SATA disks" this would imply internal disks and 1800 internal disks would have an aggregate throughput much higher than 5.2Gb/s (5.2Gb / 1800 = 2.95Mb/s per disk). Now typically you'd boot the nodes from the network (so you can change the identity of the node easily by booting it from a different image) from some sort of FC array accessed via an IB to FC gateway. However, 5.2Gbps is an odd number to get to since FC comes in 2 and 4Gb formats (1Gb fibrechannel is outdated and 2Gb is on the way out).
While I always see all the CPU details in these articles, I really wish they'd publish more about the storage requirements and methods rather than just staying (we have xTB of disk...). How do they back the finished datasets all up? Tape? MAID? VTL?
Re:India will be respected (Score:4, Interesting)
Re:Like an old episode of Red Dwarf. (Score:3, Interesting)
True story: a few months ago, my program was running slowly, and I needed a lot more CPUs. Yelled to the guy a few desks away, and he gave me the names of a few idle compute farms. Job was 50% complete after 10 minutes, I thanked him, and laughingly said that I like to measure my compute power in megawatts rather than MIPS.
Now, a couple of metawatts of compute don't sit below your desk: a platoon of sys-admins, building support staff, guards, external liasons, etc, are managing those facilities. When I grabbed them, I didn't worry about minor hardware failures: there are people with full-time jobs who hot-swap failed hardware.
Re:Quite a baseball match last night, yes? (Score:3, Interesting)