Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Software The Internet

Swarm — a New Approach To Distributed Computation 80

Posted by timothy from the or-old-approach-to dept.
An anonymous reader writes "Ian Clarke, creator of Freenet, has been working on a new open source project called Swarm. The concept is to allow a computer program to be distributed across multiple computers in a manner almost completely transparent to the programmer. The system observes the program executing and figures out how the workload should be distributed for maximum efficiency. Swarm is implemented in Scala. Its at an early-prototype stage, and Ian has created a good 36 minute video explaining the concept and the current implementation."
This discussion has been archived. No new comments can be posted.

Swarm — a New Approach To Distributed Computation More

Swarm — a New Approach To Distributed Computation

Comments Filter:

  • Earlier (Score:5, Interesting)

    by WetCat ( 558132 ) on Sunday October 11, 2009 @01:21PM (#29712243)

    .. was Mosix http://www.mosix.org/ [mosix.org]
    It allowed mosix-running linux computers to distribute their loads over a connected other mosix-running linux computers.
    Processes migrate to other nodes transparently. No programming changes were needed.

    • Re:Earlier (Score:4, Informative)

      by K. S. Kyosuke ( 729550 ) on Sunday October 11, 2009 @01:30PM (#29712293)
      And this one works at the application level, across various OSes. No computer repurposing and reinstalling is needed.

    • Re: (Score:2)

      by Sanity ( 1431 )
      Very interesting, hadn't seen that before!

      One key component of Swarm is that a supervisor process uses a clustering algorithm to determine how data should be distributed such that it minimizes the number of times a continuation must jump between different computers. Does Mosix have any equivalent?

      Why has Mosix not achieved wider usage, for example, allowing web applications to scale up using multiple servers?

    • Another Earlier - ERLANG! (Score:5, Informative)

      by mcrbids ( 148650 ) on Sunday October 11, 2009 @07:40PM (#29714491) Journal

      Erlang apparently gets it right. Scales smoothly from single core to multi-core to multi-server in a near linear fashion. Astonishingly reliable, having achieved nine nines of uptime - much less than a second of downtime - in a year. Purposely designed to mitigate shared memory problems. Built for hot-switchover - you can upgrade Erlang problems without closing them first!

      In just about every conceivable way, Erlang is the right choice for high-end multi-core multi-system clustered application development. I have a large-stack, clustered application written in PHP. While it works well, there are limits to what we can do within a single process - a problem that's likely to become worse over time as needs continue to scale up. If I were to do it all over again, I'd take a good, hard, look at Erlang.

      • Maybe it's time for you to start looking into Erlang (and alternatives) NOW. Not when your demands get so high your current application breaks down. I suspect this is really complex stuff, very hard to get it right lest to understand what it is doing really in the first place.

      • Re: (Score:2)

        by tuomoks ( 246421 )

        Correct! I have written distributed systems half of my life (longer than the age of maybe most readers here?), relying mainly on 'C', TAL, even Pascal and assembler because of the company requirements but (just for fun) tried it in Erlang - amazing for so old language!

        It has about everything you can think and all that in language! Multiple platforms, own transaction / memory/ whatever databases, can (I tried that!) be used with all main languages, easy syntax, small programs, failsafe, etc, etc.

        Used by (hug

      • Re: (Score:2)

        by Mr.Ned ( 79679 )

        - Erlang didn't get less than a second of downtime in a year, an application written in Erlang got less than a second of downtime in a year. I bet people clever enough to write such an application in Erlang could have written it in another language. Would it have been more difficult? Probably. But just because you use Erlang doesn't mean that your application is going to magically never going to have downtime - you're still going to have to work hard at it.

        - Erlang is not necessarily the right choice fo

    • Well, there are good news and bad news...

      The good news is that Mosix is still alive and well.

      The bad news is that although Mosix is excellent for High Performance Computing (HPC), it is totally useless for data-mining and web applications, which is just what Swarm is all about!

  • Name... Neat idea though (Score:3, Interesting)

    by Anubis350 ( 772791 ) on Sunday October 11, 2009 @01:23PM (#29712259)
    At first I thought they were talking about Swarm [swarm.org], a "attempt to gather up many different kinds of models that go under the heading of "agent-based modeling" and create a common language and programming approach." [freefaculty.org] that I've worked with before. I'm surprised they went with the name of an established toolkit in another aspect of programming. Still, looks like a cool tool, another layer of abstraction to make distributed computing easier might make it more attractive to those that don't use it much at the moment.
    • Gah, /. ate my formatting apparently....

    • Re: (Score:3, Insightful)

      by hazem ( 472289 )

      I'm just getting into Agent Based Modeling myself and I had exactly the same thought... why would they use the name of an established tool; especially when there are similarities in the concepts. This seems like a recipe for confusion.

      A good first check when starting an open project is to check propesedprojectname.org and see if there's anything active there. Or even just Google it - if another project shows up near the top with the same name, it's probably a good idea to pick another name.

      I'm sure there

    • Re: (Score:2, Informative)

      by Anonymous Coward

      From the FAQ [google.com]:

      Did you know that there are other projects called "Swarm"?

      Yes we did. We do not believe that 100% uniqueness is a prerequisite for a project name. Remember that the word "swarm" has been in use for over a thousand years, it wasn't invented by any software project!

      Our opinion (born of painful past experience) is that it is better to have a good non-unique name than a bad unique name. Of course, if someone can suggest a good unique name, we'll give it serious consideration.

  • Obligatory (Score:4, Funny)

    by arcsimm ( 1084173 ) on Sunday October 11, 2009 @01:43PM (#29712385)
    Imagine a Beowulf cluster of... err. Oh.

  • looks intriguing (Score:5, Insightful)

    by Trepidity ( 597 ) <delirium-slashdotNO@SPAMhackish.org> on Sunday October 11, 2009 @01:45PM (#29712397)

    The thing that's always killed this idea (along with automatic parallelization even on the same machine) is that the overhead of figuring out what's worth distributing, and the additional overhead from mistakes (accidentally distribute trivial computations), often swamps the gains from the multiple processors banging away on it simultaneously. Determining statically what's worth distributing is very hard, since solving it properly is undecidable (basically equivalent to the halting problem), and even solving it in a significant enough subset of cases to be useful has proved difficult. It looks like this project is monitoring dynamically to determine what to distribute, which seems likely to be more fruitful, although historically that approach has suffered from the overhead of the monitoring (like always running your code with debugging instrumentation turned on).

    I certainly hope he has a breakthrough vs. past approaches, or it could just be that advances in a lot of areas of technology have given him a better substrate on which to build things that naturally mitigates lots of the problems these things used to have (automatic parallelization research started probably ahead of its time, back in the 1970s, so that most academic stuff was killed off by the 1990s after no really knock-down results emerged). It's not entirely clear to me what the killer advance is, though. The particular variety of portable continuations? A good way of easily monitoring computations? Something that makes the data-dependency analysis particularly easy?

    • Re: (Score:3, Interesting)

      by djupedal ( 584558 )
      > The thing that's always killed this idea (along with automatic parallelization even on the same machine) is that the overhead of figuring out what's worth distributing

      That kind of thinking is so 90's. Brute force data mining, as an example means harvest it all and let target groups sort out what they want. It is a waste of time to 'decide'. That's like stopping to inspect every shovel full of ore as it comes out of the ground. All or nothing has been the default for some time now, and this is just a

      • Re: (Score:2)

        by vidarh ( 309115 )
        That makes sense when you have enough resources to throw at the problem. But the entire point of this technology is to semi-transparently add multi-server scalability. In that context what you wrote above makes no sense whatsoever - without a scaling mechanism you won't *have* enough resources to throw at the problem. Now, that scaling mechanism, for some problems, is as easy as "divide problem by number of servers and spawn appropriate number of workers to process each sub part", in which case this technol

    • Re: (Score:3, Insightful)

      by FlyingBishop ( 1293238 )

      Depending on how many cores you have access to, distributing trivial computations may not matter. If we ever start seeing 32 core desktop machines, for example, you start to get to the point where forking could create a realtime speedup even though in absolute terms you've wasted 5 times as many cycles.

      • Re: (Score:2)

        by Trepidity ( 597 )

        It definitely lowers the bar at how good you have to be, but I'm not sure it makes it irrelevant. Just the overhead of putting computations into some sort of container (thunks of some sort) and getting them back out can get absurd if the computations turn out to be, say, smaller than 100 instructions.

    • Re: (Score:2)

      by six11 ( 579 )

      The appealing thing about this is the problem stems from the way we program. It is really difficult to escape the single-core mindset with the languages we have today. I don't necessarily want to invoke the Wharf hypothesis here, but to some degree our expressive power is limited by the language we are using. Imperative languages that I know lend themselves to single-processor execution. I wonder if new languages specifically designed for multi-core programming would be able to avoid some of the problems yo

    • Re:looks intriguing (Score:5, Interesting)

      by david.given ( 6740 ) <dg@cowlark.com> on Sunday October 11, 2009 @06:42PM (#29714223) Homepage Journal

      A friend of mine did a system like this about ten years ago --- hi, Iain! --- called Flit. It had a number of the same features, although using a custom language; it had some rather interesting concepts, such as asynchronous function calls that would return immediately, spawning a new thread, but return a future: a value whose value was not known yet. Accessing the value would cause the thread to be waited upon.

      Unfortunately the killer problem that sunk Flit was that of distributed garbage collection. Collecting data over multiple machines is really, really hard, and he never found a usable approach to make it work. I was very disappointed to see that Swarm's garbage collection is still on the to-do list --- he doesn't appear to have started to think about it yet.

      I hope he can make Swarm work --- it's something that we could all definitely use. But there are fundamental theoretical problems that have to be solved first...

    • It's not always true that monitoring is a cost: JIT (Just In Time compilation) monitors execution, and has yielded significant speed-ups in Java.

      I conjecture that the key to distributed computing will turn out to be wasting resources (inefficiency) in some way that serves the overall goal.

      • Re: (Score:2)

        by vidarh ( 309115 )
        I see what you're trying to say, but claiming monitoring doesn't have a cost is blatantly false - gathering the monitoring data takes a non-zero amount of resources. You're right that monitoring can often be a *net benefit* though.

  • Sounds good (Score:2, Interesting)

    by cwire4 ( 831206 )

    It sounds like a good idea, but I don't think the project is far enough along in this video to warrant a posting. Maybe he was using too much of a trivial example to be appreciated in the video, but his explicitly offloading the task to another computer doesn't appear to be very far beyond standard client server models. If it were already automatically transporting processing between different nodes, it'd be much cooler, but that is not a trivial problem to solve. Deciding what should and what shouldn't

  • We have just witnessed the birth of a new meme... (Score:4, Funny)

    by Linker3000 ( 626634 ) on Sunday October 11, 2009 @02:00PM (#29712477) Journal

    In Ian Clarke's Swarm, World "Hellos" you!

  • I doesn't do much yet (Score:4, Informative)

    by svick ( 1158077 ) on Sunday October 11, 2009 @02:31PM (#29712617)
    If I understand what he says correctly, it is something like this: Distributing computation is hard, really hard. It's so hard that nobody ever did it properly. But Swarm will change this! How? Well, we don't know yet, there are so many interresting problems we have to solve first. And you can help!

    • Re:I doesn't do much yet (Score:4, Insightful)

      by Anonymous Coward on Sunday October 11, 2009 @03:41PM (#29713015)

      Mod parent up. This is exactly what Ian did with Freenet.

      He cobbled together an overly-simplistic prototype to address a set of very difficult unsolved problems in anonymous communication and then farmed out the actual real-world legwork on those problems to interested open source developers while Ian himself effectively abandoned Freenet for other (paying) gigs. To this day he is credited, somewhat ironically, as "the creator of Freenet," and a decade later the Freenet project still hasn't solved the problems it set out to solve, even after changing the fundamental network architecture several times.

      Great career strategy though. Get credit for the shiny things and pass the shame of failure off on others. He's CEO material all the way.

      • Re: (Score:3, Interesting)

        by Hobbex ( 41473 )

        I don't think this characterization is fair, and I think you would have a hard time finding somebody who actually worked on Freenet to agree with you. Ian's orginal technical ideas for Freenet - as well as his vision - are very much still a big part of the architecture, and he could never be said to have abandoned it. In fact, time has vindicated many of his ideas to a far greater extent than I expected when we started working with them. You are right that the project has not yet solved the problems it set

        • Re: (Score:2)

          by Concern ( 819622 ) *

          I agree. That said, I actually sat through the entire video, and sadly, it was a disappointment. I ignore everything that he did that came before. He has great intelligence, a wonderful idea, some vague but intriguing suggestions on how to accomplish it, and very little working code.

          At least he is up front about this - if by up front, you count a litany of near-impossible problems you do not even have an idea about how to solve in the last 8 minutes of your 35 minute video. :) I do hope for the death of the

          • Re: (Score:2)

            by Neoncow ( 802085 )

            I do hope for the death of the fad of using videos where a perfectly good blog post will do.

            I thought that's what slashdot comments were for?

    • Re: (Score:1, Insightful)

      by Anonymous Coward
      "Writing free operating systems is hard, really hard. Its so hard that nobody ever did it properly. But Linux will change this! How? Well, I've produced some code that works, but there is a lot left to do, and you can help!" - Linus around 1991 (paraphrased)

      Clarke never said he doesn't know how to solve the remaining problems (of which, he freely admits in the video, there are many). Would you prefer that no open source project was released to the world until it was 100% finished? Good luck with tha

      • Re: (Score:1)

        by svick ( 1158077 )
        I just felt from the summary that he already had something to show. And in the quite long video, he shows that he already has written some code, but it doesn't do anything interresting (yet). I expected more. Not perfect, 100% finished project. But at least something.
  • http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html [pg.gda.pl] (Dynamic Allocation and Migration Parallel Virtual Machine ?
  • So instead of R-ing TFA, I have to WTFV? Sigh.

  • Alpha code... (Score:4, Funny)

    by Linker3000 ( 626634 ) on Sunday October 11, 2009 @04:00PM (#29713171) Journal

    Computer 1: MOV AL...what? No more? MOV AL what? I need a value! WTF am I supposed to do with that!?

    Computer 2: 09? Nine? Who gave me nine on its own. That doesn't make any sense! Jeez! Hey, anyone out there missing some data?

    Computer 3: Not me, I'm pushing the registers onto the stack

    Computer 4: Nope, I've got an INT

    Computer 5: Oh, hey, it could be me - does NOP have a value. No? Sorry, my bad!

    Computer 1: Nine - yeah, nine - Well, I could stick that in AL if no-one else wants it!?

    Computer 3: Oh, heck, give it to 1. I've just got a POP instruction so I am going to obliterate it anyway.....

  • I do respect Ian, but cant we do this with the existing language infrastructure and just extend it?

    • with the existing language infrastructure and just extend it?

      That's exactly what Scala does. It "extends" Java in a sense and still runs on the JVM. Swarm is essentially a creative new way of using a mature platform, assuming it works at some point.

  • Isn't that what the new vSphere or some up-and-coming release from VMware supposed to do?

    -m

  • I think he should fix the monstrosity that is Freenet before he jumps onto other things.

  • I read that as "distributed copulation" for some reason. I need more sleep.

  • "Ian Clark of the Freenet fame".. Actually, practically no claim about Freenet came true. The authors advertised "anonymity" etc. etc. at the same time as university professors published studies of statistics about the snooped connections: to any node present on the network for some time it is elementary to collect IPs.
    It was painful to see so many users completely duped by the untrue claims, which their authors knew pretty well were untrue (and of which fact one-word admissions can be found buried somewher

  • Is there a potential to use this on a GPU? The current problem with GPU programming seems to be solved with swarm.

  • I used SWARM an year ago and I was impressed by the possibilities it offers. It was also pretty stable. I'm sure it would have achieved a very reliable level of stability by now.
  • way way back, IBM did some stuff with Java and agents... http://wapedia.mobi/en/Aglets [wapedia.mobi]
  • I've not RTFA'd yet, but on first blush, this sounds suspiciously like the Amoeba project/work/files. The net result of Amoeba was that you'd end up with a large virtual machine, comprised of many individual machines scattered across different sites. How is this different?

Slashdot Top Deals

Talent does what it can. Genius does what it must. You do what you get paid to do.

Close