Slashdot Log In
Collaborative Map-Reduce In the Browser
Posted by
kdawson
on Tue Mar 03, 2009 03:50 PM
from the suercomputer-on-the-very-cheap dept.
from the suercomputer-on-the-very-cheap dept.
igrigorik writes "The generality and simplicity of Google's Map-Reduce is what makes it such a powerful tool. However, what if instead of using proprietary protocols we could crowd-source the CPU power of millions of users online every day? Javascript is the most widely deployed language — every browser can run it — and we could use it to push the job to the client. Then, all we would need is a browser and an HTTP server to power our self-assembling supercomputer (proof of concept + code). Imagine if all it took to join a compute job was to open a URL."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Random Thoughts (Score:5, Interesting)
Two comments:
1. He places the map/emit/reduce functions in the page itself. This is unnecessary. Since Javascript can easily be passed around in text form, the packet that initializes the job can pass a map/emit/reduce function to run. e.g.:
In fact, the entire architecture would work more smoothly using AJAX with either JSON or XML rather than passing the data around as HTML content. As a bonus, new types of jobs can be injected into the compute cluster at any time.
2. Both Gears and HTML5 have background threads for this sort of thing. Since abusing the primary thread tends to lock the browser, it's much better to make use of one of these facilities whenever possible. Especially since multithreading appears to be well supported by the next batch of browser releases [owensperformance.com].
(As an aside, I realize this is just a proof of concept. I'm merely adding my 2 cents worth on a realistic implementation. ;-))
A bunch of problems (Score:2, Insightful)
I think this approach to MapReduce is a pretty creative angle to take on it. However, there are a number of distributed systems-type problems with doing it this way, that would need to be solved to actually make this realistically possible:
1) The dataset size is currently limited by the web server's disk size.
Possible solution: push the data to S3 or some other large store.
2) There is a single bottleneck/point-of-failure in the web server. In theory 10,000 clients could try to emit their map keys all at once to the web server. IIRC, Google's mapreduce elects nodes in the cluster to act as receivers for map keys during the map/sort phase.
Possible solution: Again, if you were using S3, you could assign them temporary tokens to push their data to S3 -- but that would be a large number of S3 PUT requests (one per key).
3) Fault-tolerance -- what happens when a node in the browser compute cluster fails for any of N reasons? How does the web server re-assign that map task? You'd especially want to ensure that computation finishes on a job in an unknown environment such as 1,000,000 random machines on the internet.
Possible solution: If you haven't heard from a node in N seconds, you could reassign their map task to someone else. This is a similar idea to the MapReduce paper's description of sending multiple machines on a single map task, and racing them to the finish.
4) Security -- there is no way to deterministically know whether the data emit()ed from a user's browser session is real or not. How do you trust the output of 1,000,000 users' Javascript browser executions (I think the answer is, you don't).
Re:A bunch of problems (Score:4, Insightful)
Further down in the Slashdot comments, a poster also pointed out that Javascript is a poor platform for computationally intensive work. Which I agree with on a general level. The Javascript number system is designed for genericity, not performance.
In the end this is just a cute idea that has any number of practical problems. Many of them reflect the fact that distributed computing is hard, but many of them also reflect the fact that the suggested platform is less than ideal for this function. Especially if you're going to be pushing workloads that take more time and resources to transmit back and forth than to simply compute them.
Doesn't stop me from humoring him, though. We all have to dream. ;-)
And besides, this may just inspire the next fellow down the line to use the technology for a more practical purpose.
Parent
Re: (Score:2, Informative)
What does Java have to do with anything?
Maybe you should try reading the post you're responding to?
Re: (Score:3, Interesting)
I found this somewhat startling:
http://code.google.com/p/doctype/wiki/ArticleHereComesTheSun [google.com]
If you create a javascript object named 'sun' (or several other names), netscape and family (including firefox) load java into memory.
Re: (Score:2)
Translation: Who wants a patent suit from Digg.com?
Re: (Score:2)
And then Firefox (or Epiphany) pops up a dialog after JS maxes out the CPU for 30 seconds asking if you want to permit the execution to continue.
So you have to limit yourself to 20 seconds per page load, and have the overhead of using Javascript as opposed to a better language like Java. Actually, a Java applet seems like what you would really want to do if you were re
The future of banner ads? (Score:3, Interesting)
Re: (Score:2)
Who wants to open up a page that suddenly uses all of their cycles and makes their computer useless for anything else while this is running.
Can you actually do that on a modern OS and a modern CPU? I regularly have my CPUs on 101% load and it stays snappy as ever. Only heavy I/O on the system drive makes it unresponsive. This applies for both Gentoo and Vista. (The default install of Linux Mint, however, sucks horribly.)
Botnet (Score:4, Insightful)
Imagine how much *spam* you could send using this approach.
No, wait...
Re:Botnet (Score:5, Insightful)
With ever-increasing JavaScript performance, there's a lot of cpu power available for cracking passwords and captcha's... Just include the code in an ad and you're done. No tricky installs needed, just the idletime of the user's web browser.
Parent
Rather have a cold PC (Score:3, Insightful)
-B
Re: (Score:2)
I really don't think laptops were designed to run at 100% all the time anyway, so yeah, I'd avoid any distributed computing projects on your computer.
I run it on my two desktops at home though, and there's barely any difference in my electric bill. Idle vs load for me is about 40W difference -- I could save more by turning off a fairly dim bedside lamp.
Join compute cloud (Score:5, Insightful)
BOINC (Score:5, Insightful)
If you were really interested enough to donate your CPU cycles, is it really that much harder to install BOINC, and get a job running?
Plus then you can run native code instead of having to run in [shudder]Javascript[/shudder].
Re: (Score:2)
BOINC is quite possibly the single worst bit of software I've ever seen. It's kind of like the team did a detailed study of the best practices for software usability and then did the exact opposite.
Re: (Score:2)
If you're talking about the UI, then I'll agree it needs a bit of work, but then it is still a "nerd project" at this point. With any nerd project, the interface is at the bottom of the TODO list.
If you're talking about the code, care to explain? I've never looked at it.
Re: (Score:2)
I don't get what's the big problem people have with "[shudder]JavaScript[/shudder]".
It's a Turing-complete language, which means it can be used to do anything from simple form validation to ray tracing and neural net simulations.
With AJAX to handle file interactions, I don't understand the problem that people have with it. What is it that you think JavaScript can't do that 'x' language can?
I wish people would get over this childish bias and accept that JavaScript is a /real/ language, and not
Re: (Score:3, Insightful)
A big thing is the same thing people have against VB: there may not be anything technically wrong with it, but bad programmers are drawn to it because it's easy, so you hardly ever see a good VB program. There's especially nothing wrong with VB now, when writing a program in VB.NET gets you the same result as if you'd written it in C#: you still get CIL code when it's compiled.
However, Javascript gets used for way too much, and historically it's been a huge browser security issue. Even if you use it respons
Noscript (Score:5, Informative)
Progress is running less JavaScript, not more.
Re:Noscript (Score:5, Funny)
Parent
Re:Noscript (Score:5, Insightful)
Actually it was the '90s, but whatever. The thing is, non-DHTML web pages are actually pretty good for most things... what made those early '90s web pages so awful was no CSS, slow connections, and the fact that people really didn't know how to design for this new medium.
Probably 99% of the web still shouldn't need Javascript or flash, though pages usually do need to be dynamic on the server side.
Parent
Re: (Score:2)
But the argument against javascript is one that is countered by your own comment: "the fact that people really didn't know how to design for this new medium".
Javascript is a tool just like another on the Internet. It can be used for good or evil depending on who writes the program. And as you mentioned, retreating from javascript means going back to a purely
Re: (Score:2)
Actually it was the '90s, but whatever. The thing is, non-DHTML web pages are actually pretty good for most things... what made those early '90s web pages so awful was no CSS, slow connections, and the fact that people really didn't know how to design for this new medium.
Sure it's fine when you've got a 2GHz processor and a smack of RAM to compile and run an interpretive language -- with the sole purpose of relatively simple data manipulation, validation, and perhaps some light processing to kick a chunk of data back. But when you are talking about serious data crunching, you want code running natively, not in a locked down little box, like SETI@Home, and optimized for that architecture and platform.
People think because you can put it on the web, you should. That is, at bes
Re: (Score:2)
Which is to say that Javascript is the future of all computing progress.
"First they ignore you, then they ridicule you, then they fight you, then you win." --Mahatma Gandhi
Mwuhahaha! MWHahaha! MUHAHAHAAAAAAaaaaaa--
*cough* *cough* *wheeze* *sputter* *wheeze*
*clears throat* 'scuze me!
MWUHAHAHAAHAHAAAA!!!! :-P
Re: (Score:2, Informative)
"First they march you through hundereds of miles of jungle without food or water, then they shoot you, then they disembowel you, then you lose." --Mahatma Gandhi, had the Japs won WW2.
Re: (Score:2)
Yeah, yeah, and Usenet was the ultimate discussion group and everything's been downhill from there, right? And 25x80 column monitors were plenty (who needs proportional fonts?) and color is way overrated, and...
Why is it that we always need the previous generation twho remembers "what it was like before all this newfangled nonsense" to die off before we can make progress?
Just because you're looking for the web to look like a static newspaper doesn't mean the rest of the world wants the same thing.
Re: (Score:2)
Just because you're looking for the web to look like a static newspaper doesn't mean the rest of the world wants the same thing.
There are situations where JavaScript is good, but it simply breaks things like the ability to bookmark your page and then restore it as it was from the bookmark. Then you have the sites which really abuse it: for example, you can't book a flight with Ryanair if you have JS disabled (or a browser which doesn't support it: they don't seem to have come across the concept of degrading gracefully).
Would this work for music? (Score:4, Funny)
You could also use this to index the MP3 files on everybody's hard drives, then share the music just by visiting a URL!! ... oh wait...
Why? Why? WHYWHYWHYWHY??? (Score:5, Insightful)
Javascript really isn't suited for this kind of thing, even with worker threads, for two reasons I can think of. First, web clients are transient... they'd have to report back often in case the user clicks away.
But more importantly, Javascript just isn't a good language for massive computation. It only supports one kind of number (double), has no vectorization or multicore capabilities, has no unboxed arrays, and even for basically scalar code is some 40x slower than C, let alone optimized ASM compute kernels. (This is for crypto on Google Chrome. Other browsers are considerably slower on this benchmark. YMMV.)
Re:Why? Why? WHYWHYWHYWHY??? (Score:5, Insightful)
Parent
Re: (Score:3, Interesting)
It would need to be 10000x at the very minimum.
If a user downloads, say, folding@home, it's running all day, every day, on all cores of the machine, whenever the computer is on and idle, which is most of the time. The user doesn't have to remember to run it, doesn't have to devote screen real estate, attention and so on, and the program is less annoying because of its low priority and relatively low memory footprint (less boxing).
Additionally, the 40x I cited was in the fastest available browser (Chrome),
Re:Why? Why? WHYWHYWHYWHY??? (Score:4, Insightful)
Javascript really isn't suited for this kind of thing, even with worker threads, for two reasons I can think of. First, web clients are transient... they'd have to report back often in case the user clicks away.
I don't see why web clients being transient is a problem. The whole point of the MapReduce algorithm is that each worker (the web clients in this case) don't need to know anything about what the other worker is doing, what the system as a whole is doing, nor what it had done with any past job.
Parent
Link (Score:5, Informative)
http://en.wikipedia.org/wiki/MapReduce [wikipedia.org]
Re:Link (Score:5, Insightful)
Parent
Re:Link (Score:4, Funny)
Parent
Re:Link (Score:4, Funny)
Parent
Re: (Score:3, Funny)
Here, let me google that for you. [lmgtfy.com]
MapReduce fanboyism (Score:5, Insightful)
Oh, please, make the MapReduce fanboyism stop.
Yes, it's a neat technique. It's also very old and obvious. Google's implementation is also good, but this stuff is just not rocket surgery. It's just a simple pattern of how to massively parallelize some types of computational tasks.
But somehow, just because some dudes at Google wrote a paper about it, it's become the second coming of Alan Turing or something among some silly folks. Hell, a couple of weeks ago somebody was saying on the comments here that MapReduce was a good alternative to relational databases. Now that is silly.
Re: (Score:3, Funny)
I stopped at... (Score:3, Insightful)
"Javascript...â" every browser can run it..."
There is a huge difference between being able to run javascript apps and run javascript apps well - not to forget that a lot of the javascript I see out there really only works on PC's with IE or Firefox, Opera and Safari, especially on OS X seem to have trouble with some sites that aren't coded for compatibility, but instead pushed out quickly with little regard for anything other than IE on Windows.
Bandwidth and Exercise (Score:4, Insightful)
A common mistake in multi-server builds is that bandwidth is free.
Bandwidth Costs Money and Time. Both are reduced by having the network closer to the processing. This is one of the reasons google bought all that "dark fiber" left around after the .com bust.
Another flaw is that computation of data is difficult to provide "good results" in blocks unless they're doing relativity matrices (Think PageRank).
Something to think about:
If I'm sending names to your pc, what can I derive from that list without having the entire list?
Re: (Score:3, Interesting)
Something to think about: If I'm sending names to your pc, what can I derive from that list without having the entire list?
Frequency of each name? Frequency of characters in names? Bayesian probability of one character following another in names? Number of names of a particular length?
Each worker would compute the stats for their chunk of work (the "Map" part of MapReduce), and then send the results back to the server to be aggregated (the "Reduce" part of MapReduce).
Some of these may seem interesting, but then again, what interesting data can you derive at all from a list of names, even if you had the whole list?
Pay Me (Score:5, Interesting)
If there were a couple-few or more orgs competing to use my extra cycles, outbidding each other with money in my account buying my cycles, I might trust them to control those extra cycles. If they sold time on their distributed supercomputer, they'd have money to pay me.
As a variation, I wouldn't be surprised to see Google distribute its own computing load onto the browsers creating that load.
Though both models raise the question of how to protect that distributed computing from being attacked by someone hostile, poisoning the results to damage the central controller's value from it (or its core business).
Scripts taking too long (Score:4, Funny)
Is this why my browser keeps telling me scripts on the slashdot main page are taking too long and do I want to stop them for the last few months?
Re: (Score:3, Informative)
I sometimes have that with a 1.7GHz box. And even when I don't, reloading the front page of /. makes Firefox sluggish or non-responsive for 5 to 20 seconds.
Re: (Score:3, Informative)
Just change your prefs- under Index/General uncheck "Beta Index" and check "simple design" and "low bandwidth." With those prefs Slashdot loads almost instantly on my somewhat aged machine (P4 2.4) and is still usable on a 700MHz P3.
Self-defeating idea (Score:2, Insightful)
imagine a Beowulf cluster of trojans and drive-bys (Score:2)
hijack-weasels who need to be shot have pretty much ruined the idea of distributed donated computing resources, thanks.
Re: (Score:3, Insightful)