Scaling To a Million Cores and Beyond

mattaw writes "In my blog post I describe a system designed to test a route to the potential future of computing. What do we do when we have computers with 1 million cores? What about a billion? How about 100 billion? None of our current programming models or computer architecture models apply to machines of this complexity (and with their corresponding component failure rate and other scaling issues). The current model of coherent memory/identical time/everything can route to everywhere; it just can't scale to machines of this size. So the scientists at the University of Manchester (including Steve Furber, one of the ARM founders) and the University of Southampton turned to the brain for a new model. Our brains just don't work like any computers we currently make. Our brains have a lot more than 1 million processing elements (more like the 100 billion), all of which don't have any precise idea of time (vague ordering of events maybe) nor a shared memory; and not everything routes to everything else. But anyone who argues the brain isn't a pretty spiffy processing system ends up looking pretty silly. In effect, modern computing bears as much relation to biological computing as the ordered world of sudoku does to the statistical chaos of quantum mechanics.

1 billion cores

[should_be_linear]

thats about 30 forks(), and there you go.

Bluudy Blogs

[jd]

• Hardware design tools [man.ac.uk]
• Transactional Memory [man.ac.uk]
• TERAFLUX: Exploiting Dataflow Parallelism in Teradevice Computing [teraflux.eu]
• SpiNNaker - A Universal Spiking Neural Network Architecture [man.ac.uk]
• The Balsa Asynchronous Synthesis System [man.ac.uk]

I've left out links to some projects, by request, but everything can be found on their homepage anyway. Anyways, it is this combination that is important, NOT one component alone.

Problems with this blog.

[Anonymous Coward]

The problem posed by the author is somewhat of a straw man argument: "The trouble is once you go to more than a few thousand cores the shared memory - shared time concept falls to bits."

Multiple processors in a single multicore aren't required even today to be in lockstep in time (it is actually very difficult to do this). Yes, locally within each core and privates caches they do maintain a synchronous clock, but cores can run in their own clock domains. So I don't buy the argument about scaling with "shared time".

Secondly, the author states that the "future" of computing should automatically be massively parallel. Clearly they are forgetting about Amdahl's Law (http://en.wikipedia.org/wiki/Amdahl's_law). If your application is 99.9% parallelizable, the MOST speedup I can expect to achieve is 1000X, forget about millions. High sequential performance (ala out-of-order execution, etc.) will not be going away anytime in the near future simply because they are best equipped to deal with serial regions of an application.

Finally, I was under the impression that they were talking about fitting "millions" of cores onto a single die, until I read the to the end of the post that they are connecting multiple boards via multi-gigabit links. Each chip on a board has about 20 or so cores with privates caches or local store. They talk to other cores on other boards through off-chip links...... SO isn't this just a plain old message passing computer?! What's the novelty here? Am I missing something?

Distributed Computing

[thomasinx]

The problems with "coherent memory/identical time/everything can route to everywhere" isnt only seen when you get up to a million cores. I've done plenty of work with MPI and pthreads, and depending on how it's organized, a significant portion of these methods start showing inefficiencies when you get into just a few hundred cores.

Since there are already plenty of clusters containing thousands upon thousands of individual processors (which dont use coherent memory..etc), the step to scale up to a million would likely follow the same logical development. There should already be one or two decent CS papers on the topic, since it's basically a problem that's been around since beowulf clusters were popularized (or even before then)

Re:Reminds me of Hillis

[headhot]

The CM-1 has 65,536 one bit processors. The CM-5 was in Jurassic Park, and some phone companies.

Re:1 billion cores

[TheSpoom]

I'm pretty sure the poster meant to do something like this:

fork();
fork();
fork(); // etc.

which would make the number of processes increase exponentially every time the forked processes forked again. Not 1, 2, 3, but 1, 2, 4, 8, 16... and 2^30 gets you above 1 billion.

Re:multi core design

[jd]

You've got to be careful when talking about threads. There are four basic models: SISD, SIMD, MISD and MIMD. Of those, only SISD is serial, but if you've two independent SISD tasks, you can run them in parallel. Most modern supercomputers are built on the premise that SIMD is good enough. Not sure where MISD is used, MIMD fell out of favour when vector processors became too expensive but may be revived on more modest CPUs with modern interconnects like Infiniband.

Silly Brain

[Anonymous Coward]

anyone who argues the brain isn't a pretty spiffy processing system ends up looking pretty silly

Yet the brain fails miserably at incredibly easy calculations. Most applications require failure-tolerant, repeatable, and highly reliable results, none of which the brain is really good at. The brain might not be the best model for massive parallel processing. Sure we can learn from it but perhaps we can also do better. Even in AI research it is questionable whether a connectionist modeling of the brain is the really the right approach. As anyone with halfway demanding job knows, the brain is a fairly imperfect result of non-purposeful and undirected evolutionary processes and it is easy to experience its limits in your daily work. (If not, you should perhaps get an intellectually more challenging job.) Even for the purpose of building AI we might be able to come up with something better or something else entirely than mimicking the brain.