Is Parallelism the New New Thing? 174
astwon sends us to a blog post by parallel computing pioneer Bill McColl speculating that, with the cooling of Web 2.0, parallelism may be a hot new area for entrepreneurs and investors. (Take with requisite salt grains as he is the founder of a Silicon Valley company in this area.) McColl suggests a few other upcoming "new things," such as Saas as an appliance and massive memory systems. Worth a read.
TRIPS (Score:4, Interesting)
Re:1% of programmers (Score:3, Interesting)
The entire hierarchy system in the IT fields has to deal with the painfully obvious fact that less than 1% of programmers know what they're doing: that is, in an ideal scenario, everyone would know what they're doing, and you'd have a FEW hardcore computer scientists to handle the nutso theoritical scenarios (most parallel programming for example can be done with only basic CS knowledge... like, let say, trimming all of the strings in an array...everyday stuff). But thats not how it works: In the current world, you usually have a few douzan coding monkeys with some understanding of the theory (but no clue how to use it), and a few software architects who call the shots (Note i'm talking about software architects...those that work on software design, not system architects).
In an ideal world where programmers know how to do the -basics- of their job, software architects would be an obsolete job, yet they're really not.
Now, thats a side note, but my point is that the typical IT shop hierarchy is MADE with the "all programmers suck" in mind...so it really has to be a problem, and thus the 1% number isn't far fetched.
As for anecdotal evidence... I'm a
The closest to parallel I've seen was people using a queue dispatcher (MSMQ) and have multiple triggers watching the same queue, thus resulting in some kind of parallel execution... thats it though.
So, 1% from my point of view actually seems pretty -high-, as sad as it is.
Re:TRIPS (Score:3, Interesting)
If each subsequent step is dependent on the previous step then this by nature makes the program impossible to parallelize. So what we're most likely going to find out is that we'll hit the 'wall' of parallelization in programs in a relatively short period and be back to the familiar place: increasing the clock speed.
Re:More so now, but depends ... (Score:3, Interesting)
Being able to simply say "the order in which these tasks are made doesn't matter" lets you run a lot of tasks in parallel right there.
Then you have a lot of tasks which do not depend on each other at ALL, but will still wait on each other... Let say a typical MVC web app ala Rail or Strut... you have a lot of ressource handling going on in the view that doesn't even rely on the Model/database whatsoever...and then you have the operation of rendering the model data in the view... so querying the data, and handling the ressources, could be done at the same time, and once both are done, do the actual render. ASP.NET has a decent mechanism for that, but no one uses it.
These are all things that are being worked on (and some already exist but most people ignore them), and can dramatically boost performance of virtually ANY application...but there needs to be a certain awareness... And some of these features are just freagin late (Microsoft is working on PLINQ...ways to make it easy to process data structure in parallel... for example: int[] blah = listOfInts.ForAll( item => item * 2 );... and it will do it in parallel. Seems so obvious, but why hasn't this been around for years in
Re:1% of programmers (Score:3, Interesting)
How do you see that as different from what Java J2EE does? Most J2EE servers these days use pools of threads to handle requests. These threads are then utilized based on poll/select APIs so that one thread can handle many requests depending on the availability of data. Database connections are similarly pooled and reused, though any optimization on the blocking level would need to be done by the JDBC driver.
Re:About time (Score:1, Interesting)
Windows NT supports multiple processors since 1993.
>> When attempting the same thing on a closed source, properietary operating system, things were much more difficult.
Multithreading in the past had better support on closed OS like Windows NT and BeOS than you had in the first releases of Linux kernel (where you were stuck with a user-mode threading scheme).
Rule #1 : Windows/Microsoft are shitty enough on their own. No need to invent stuff just to make them appear worse than they are.
Re:TRIPS (Score:4, Interesting)
I guess my point is that I think we'll actually create the basic, expandable model fairly quickly. Would you agree that today's supercomputing, which utilizes parallelization on a scale far beyond desktop computing, has successfully harnessed parallelization? I hope you would. If so, then the next step is miniaturization of what supercomputing is already doing. That step is just now taking place. It's not something that will happen overnight, but I do think that after we've fully integrated parallelization into everyday computing we'll be back to the same old game again: that of looking for ever better ways to increase FLOPS through transistor/switch speed.
My basic thoughts on this are that it is, in theory, easier to model the perfect parallelization of a program, and the optimum number of cores for a specific type of computer, than it is to model the fastest possible clock speed of a CPU. Because of this we'll probably see diminishing returns in advancement of parallelization at an accelerated rate compared to CPU design and clock speed.
Then we are all doomed (Score:3, Interesting)
Re:Most companies need parallel developers (Score:3, Interesting)
Calling the latter architecture parallel computing is misnomer, it is really "simultaneous" computing i.e. things can happen at the same time, but there is a big difference between the same thread executing n times in parallel, and different threads doing different things simultaneously.
For example, a "Trivial" program which reads in a list of numbers from a file, computes something (say the sum of the magnitudes squared), and prints the result out to the screen might be implemented as follows:
while not eof
read n numbers
compute something from them
print result
a multithreaded version might look something like this
Thread 1 (Disk IO): read n number from disk, write them to a queue/shared memory, repeat
Thread 2 (Outputting): wait for outputs to become available, print them, repeat
Thread 3 (Compute): wait for inputs to arrive in a queue, process them, write output to another queue, repeat
A parallel version would just have more than 1 Compute thread, and they would subdivide the work between them (for example 2 threads dividing the input array into stripes, one handling even indices, the other odd...or a bunch of threads computing different slices of the array). Note that the threads would still have to combine their results at the end of the computation, and that is not always simple to do in parallel.
Some problems or algorithms simply cannot execute in parallel. Also issues of memory access patterns, caching, branch divergence (if they threads take different code paths will this affect performance) come into play. It requires a whole new set of issues to worry about, but they are not too difficult. As the professor who teaches the course http://courses.ece.uiuc.edu/ece498/al1/ [uiuc.edu] says, learning parallel programming is not hard, he could teach it to you in 2 hours. But doing parallel programming well and efficiently is difficult. You can write a trivial parallel program which uses just 1 processor and has just 1 thread, which is identical to the sequential version, and it will work logically, although the performance will be severely limited. You can then extend it to use n threads, and it will experience a speedup. But to take full advantage of the hardware on your board, you will need to know a few tricks, and understand the hardware and your program's behavior intimately.
Another issue is that programmers were spoiled by processor upgrades coming along and speeding up their programs "for free" by virtue of their higher clock speed. Now with clock speeds reaching physical limits, the only evolution in new processors will be in the number of cores they have. So the only way to coax more out of a program will be to make it more parallel, and that might be trivial or it might be difficult. We're going to have to think laterally to get more performance out of software.
Re:About time (Score:3, Interesting)
It was a highly parallel machine. At every level. It could execute 10 instuctions at once. There was a 10 way path to main memory so we could do 10 store or fetches at once. Some machine had multiple CPUs and all machine had at least 10 PPUs, there were smaller CPU where most of the operating system actualy ran keepping the main CPu free for user level tasks.
To this day I've not seen this level parallelsm. Ok the old Pentium had a long pipeline and Sun will sell you an 8-core SPARC but still nothing compared to the old CDC machines
The wiki article below has some details wrong but the ideas are describbed well. The machine was built at a time when electronics tachnology had hit a wall and couldn't be had faster. They had already gone to extreams like giing up on water cooling to use freon cooling where the refridgerent was plumbed directly into the guts of the CPUs card guides. The only option to go faster was parallelium. I think we wil see this again as we hit a wall with IC technology
http://en.wikipedia.org/wiki/CDC_6600 [wikipedia.org]