Building the World's 4th Fastest Supercomputer

ngkabra writes "In November 2007, a previously unheard of supercomputer called EKA, built by CRL, India came out of nowhere to become the 4th fastest supercomputer in the world. It is also the only supercomputer in the top 10 that hasn't taken any government funding — which means it has no strings attached against commercial exploitation. That is one of the reasons why Yahoo! chose EKA for the cloud computing research that they announced at the Hadoop Summit earlier this week. Yesterday, I attended a presentation by the team that built EKA, and they touched upon a lot of the technical details of EKA, and the challenges faced in designing and building it, which makes for interesting reading."

Interesting maybe

Well, interesting enough for me to read to the end of the article. The article, however, does seem to lack substance and details. It seemed to spend a lot of time apologising for inadequacies and seemed to aimed at a less technical audience. That is fine,

Re:

You would think that the target audience would be more tech savvy people though. It is an article about a supercomputer which is something I don't think Joe Schmoe really cares about too much. I really want to know more about this interconnect architectu

Re:

You are absolutely correct. And you have alerted me to the question: What audience was the article targeted at? Reading it (TFA) again, I am not sure it is targeted to any audience, but was written by a person without a clue. I can't for the life of me ima

Re:Interesting maybe

I concur. If you asked Joe Schmoe if he knew what a super computer was he'd probably tell you Windows Vista, which would be close, except he'd leave out the words "anything that can run".

India will be respected

India is now on track to earning the respect it deserves. Good for them! But I wonder what countries like Russia have hidden from the public. I am sure they have respectable hardware. Those very deadly [nuclear] weapons and missile systems must have had their design done on some pretty cool hardware.

Re:

Lazy Americans. Who needs a supercomputer when you have pens, paper, and an endless supply of cheap labor?

Re:India will be respected

Actually people are not very good building nodes of supercomputers. They think slow, make way too many errors, and have horrible i/o speed, not to mention the best have about 7 abstract bytes of working memory (can hold about 7 concepts at a time). It would take thousands of years and millions of people to make calculations which this supercomputer can manage in minutes.

Thought experiment time

What does it take to make a supercomputer? Well, very fast switches, very low-latency, high-bandwidth interconnects, and decent CPUs. (You don't need mega-fast CPUs, you just need lots of them.)

Let's start with the switches. You can parallelize network traffic - fragment the packets, stick full headers on each fragment, switch the fragments in parallel, reassemble, have the queue re-order inbound traffic by sequence number. (That last step isn't too hard, you have some fraction of the sequence number and the full fragment number map to a unique address in a permanently allocated ring buffer. Copy the payload to that address and the packets are in sequence order, not delivery order.) So, instead of having individual switches that are fast, bank the switches up and have the combined virtual switch run very fast. You can do that on exportable commodity hardware.

Or, you could sneak through Homeland Security (who have much more interest in nipple rings than dangerous weapons) a bunch of 24-channel 5Gb/channel InfiniBand switches. You wouldn't need many of those to get a decent Quake LAN party, and not many more before you could run weapons design software. It is unclear how many are required to run Vista, once service pack 1 is installed.

Interconnects. Obviously, InfiniBand is hellishly fast. So is SCI. 10 Gb ethernet, ideally with iWarp extensions, would be much slower but still perfectly good for a commodity cluster. If you scrap the idea of having machine-to-machine communication and do memory-to-memory communication, you could actually use PCI-e 2.0 as one gigantic interconnect. Ideally, you'd have the memory appear as two separate devices - slave and master - so that direct memory-to-memory RDMA could be initiated. A lot of very similar work has already been done by US supercomputer giants, and given how many have either been bought, gone bankrupt, or otherwise vanished, it's reasonable to suppose large quantities of such RAM could have "migrated" overseas.

CPUs - well, there are some respectable 16-bit pile-of-pc clusters. One was reviewed on Slashdot some time back. Even a cluster of Cell processors, if large enough and well-enough programmed, could be very effective. A hostile nation wouldn't need high-end 4x4 multi-threaded multi-core SMP systems, although again given how juvenile airport security is, I can't imagine it would be hard for someone to export, say, a couple of hundred motherboards at that spec.

Ok, what about OS? Who needs one? Anyone with a copy of OSKit or something similar can work at almost bare metal levels as if they had a full OS. If they did want a full OS, then NetBSD or MOSIX would be quite sufficient. Or they could take an OS project like Exopc and add high-performance networking to it.

Software? If you've a decent copy of BLAS, LAPACK and some solvers, tightly optimized for the platform, you're set. Those core maths functions are critical. Since the functions and API are fixed, it would not be impossible for someone wanting raw power to have put them into an FPGA, SoG or ASIC. Collective operations are also nasty, but they too can be done entirely in hardware, giving you orders of magnitude speedup over conventional software solutions. Synchronizing is the third killer, but there are meta schedulers to handle that and you could again place those on dedicated hardware.

In short, although I couldn't afford to build a top 500 machine, it is only the affording of it that is a problem, and foreign countries are quite well aware of that. Especially after China built its first (publicly-announced) Government-funded Beowulf. Supercomputing is easy, it's the price tag that isn't.

Re:

Danger, danger, left-field high-velocity incoming! Last I heard, Kerrighd is French, MOSIX is Israeli, most chips were Taiwanese or Chinese, Linux is Finnish, BSP is from England (as is the Transputer and Occam). Only a really sick bastard assumes that non

Re:

I believe Russia had some very credible Vax clones during that era, reverse-engineered and built in bulk. Also a rather eclectic array of 2nd gen stuff, a couple of PC (including Apple ][) clones and the odd mainframe. See http://en.wikipedia.org/wiki/Hi [wikipedia.org]

Re:

If by "pretty cool hardware" you mean "pen and paper" then yes.

Quite a baseball match last night, yes?

Will tech support for this Indian computer be outsourced to Dell reps in Texas?

Re:

where's the imagine a beowulf cluster of these post?

I think there would only be need for about six of these in the entire world.

  rd

Re:

Maybe funny but look at the investments of some large Indian companies (no names.) Some are heavily investing or at least interested to invest for support centers in US. Computing is getting more and more global but, I have to say, Indians are fast learnin

Finally .......

A Vista spec machine in non Governmental hands!

Re:Finally .......

Please don't exaggerate, any of the top 100 could run Vista, no problem, with Aero switched off.

This is old news

I have known about EKA for years now. Being an Indian, I try to be aware of all the developments in IT in and around India, so I knew about this project when it was at it's earliest stages.

What most people don't know is that this project is funded by the Tata Group, and is directly influenced by TCS (Tata Consultancy Services), which I am strongly related to. The initial goal of this project was to surpass IBM's Blue Gene in the field of supercomputing, but then, things went wrong (as is usually the case with Indian projects), and it landed up in the 4th position, which I still feel, is quite a big feat for a projcet funded privately.

Re:

Another thing that is impressive about Eka is that it was built for just $30 million. This is, according to the Eka architects, significantly lower than the costs of the other computers that are in this league.

Re:

What most people don't know is that this project is funded by the Tata Group

Nice. We like big tatas.

We like nano [slashdot.org] tatas too.

Blue Gene has government funding?

It is also the only supercomputer in the top 10 that hasn't taken any government funding

So this implies that IBM has taken government funding for their fastest server? That's news to me...

Re:Blue Gene has government funding?

Blue Gene has government funding?

Err, yes? Blue Gene/L is situated *inside* the Lawrence Livermore national labs. I'm pretty sure the department of energy funds the entire project. IBM is the largest, but not the only technology partner in it.

Odd storage requirements...

# 1800 blades,
# 80TB storage. Simple SATA disks. 5.2Gbps throughput.

So using the 80TB * 1024GB/TB / 1800blades gives us about 45GB/blade. If they're using "simple SATA disks" this would imply internal disks and 1800 internal disks would have an aggregate throughput much higher than 5.2Gb/s (5.2Gb / 1800 = 2.95Mb/s per disk). Now typically you'd boot the nodes from the network (so you can change the identity of the node easily by booting it from a different image) from some sort of FC array accessed via an IB to FC gateway. However, 5.2Gbps is an odd number to get to since FC comes in 2 and 4Gb formats (1Gb fibrechannel is outdated and 2Gb is on the way out).

While I always see all the CPU details in these articles, I really wish they'd publish more about the storage requirements and methods rather than just staying (we have xTB of disk...). How do they back the finished datasets all up? Tape? MAID? VTL?

4th Fastest

The posting title reminds me of the Simpsons gag of Homer imagining what it would be like to be a moderately rich man. "Tee hee! I could rent anything I want!"

Fourth fastest according to who?

My understanding is that the list of Fastest Supercomputers [top500.org] is for supercomputers that are publicly funded - which is why we don't see any mention of google or microsoft clusters. But yet the summary claims that this one has not taken any government fundin

Re:

It's really not that much of a problem. Supercomputers have very different hardware vs support cost trade-offs compared to consumer devices.

True story: a few months ago, my program was running slowly, and I needed a lot more CPUs. Yelled to the guy a fe