Slashdot Log In
Swarmcast GPLed
Posted by
CmdrTaco
on Wed May 23, 2001 05:33 PM
from the worthwhile-concepts dept.
from the worthwhile-concepts dept.
miguel writes "OpenCola has just released SwarmCast which is a very interesting mechanism for distributing software. At Ximian we are looking into integrating this into Red Carpet to accelerate software downloads by using their sharing software. The demo of their product is pretty amazing." Very clever: essentially it creates a peered network so larger files can be shuttled around faster. Each client can serve a small piece of data to other clients so that a massive centralized data center isn't necessary. Now the cola on the other hand...
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Only Netscape/IE? (Score:2)
Re:Could this be used as a "legitimate use of p2p" (Score:1)
What, what what? (Score:1)
I am terribly afraid of this trend of making EVERYTHING too easy to do for the people who do not understand HOW or WHAT they are doing in fact.
So now, as if it wasn't enough that red-carpet requires root access for it to work BEFORE really needing root previledges, you will have software distributed in pieces in that shareing mechanism?
Make usage easy on people, but please, do not exagerate on giving power to those who cannot handle it. Next thing we'll know, viruses for linux will start spreading because of all the guys who have no idea they should avoid using root as much as possible.
I love gnome as a desktop, but primates, please stop and consider... do not open precedents please.
First of all, cut root usage to the bare minimum in red-carpet (it currently isn't).
Apart from that, thanks for the neatly tied gnome packages.
Hugs, Cyke
This is p2p, but not a napster clone... (Score:1)
I think the hackers at OpenCola want to create a product FOR large media streamers. The fact of the matter is that when something really big comes out for download, like a trailer to a new movie, a large portion of the internet endusers will want to download the same file(s). Instead of each joe or jane sitting at his/her computer, waiting for the content to download at 4kbps, the hosting company can place up 3 or 4 decent servers (peferably in diffrent areas of the nation) to be the first 3 or 4 nodes of this download.
The best part about this settup is that the servers will recive their prapotion of the download, and that's it. another really cool feature that could be implemented (And it possibly could be, I didn't read the code or faq) like automatic selection of servers with good ping/transfer-rates/hops, while dropping poor connections to make sure I'm not reciving packets from accross the contenent.
Questions I have: Can a packet be restarted if it's droped? Can a faster server take over a slower server's packet? Is it possible to have streaming media? Can there be a diffrent client software, that's not necessarialy Java? (not saying I don't like the java client, but not everybody will.)
All in all, I like the consept. Large companies would spend big bucks to have their heavaly downloaded files swarmcasted, even the content providers and hosting companies will like this... if it works, and it catches on.
Good job FreeSoda-- er, OpenCola. Now, Don't give up! Take ideas and suggestions from your peers (us) and keep polishing the code untill it shines!
Pathway
Re:This is p2p, but not a napster clone... (Score:1)
None the less, I'm off to send off my post.
Pathway
Re:your gay justin (Score:1)
Ooops. Sm:)e.
Re:This is p2p, but not a napster clone... (Score:1)
Why not have this as a plug-in? Keep the Java client (more compatibilty, the better... plus competing products), but have a Plug-in that integrates with the browser? This will increase the popularity of the software, as people are used to using plugins, but not used to installing seprate java clients.
What kind of requirements does the server need? Is it a simple http or ftp server? Or is there a diffrent format for the download?
Can a modem/dsl/cable handle several streams at once? How many, in your experiance?
Is there a log of where I got each segment somewhere? Is this log on my local machine, or is it logged on the web, or on the provider?
If I was a large company wanting to host a large file, how much would/could it cost? (Obviously, it could be free, but I want to know how much somebody would charge to provide such a service...)
Are there plans at OpenCola to make a Napster style distributed network using this technology? Does OpenCola know of any other software that plans to?
Okay, that enough new questions for now...
Pathway
Mojo Nation vs. Swarmcast (vs. Freenet, vs...) (Score:2)
It isn't true that Mojo Nation [mojonation.net] is "not focussed on performance". I'm one of the Mojo Nation hackers, and we care about performance. It is true, however, that Mojo Nation is pretty complex, providing both data transfer (using a "swarm" like technique), and data storage, and a queriable search engine. The end result is something like a distributed, non-deletable World Wide Web. (Sort of like Freenet plus persistent data, or the earlier concept of Ross Anderson's "Eternity Service".)
Performance isn't that great on Mojo Nation right now, but it is good enough, in my experience, for daily use.
I'm pretty excited about the Swarmcast open source release, both because I think Swarmcast is a cool app in itself, and because I can now start taking ideas and code from Swarmcast to put into Mojo Nation, and vice versa.
In the long run, both Mojo Nation and Swarmcast will improve because of this sharing, as will other related open source projects like Freenet [sourceforge.net] and Free Haven [freehaven.net].
Regards,
Zooko
P.S. I've been talking with Justin Chapweske, the Swarmcast, lead, on irc.openprojects.net, and he's already pointed out a potential bug that we need to avoid in future versions of the Evil Geniuses Transport Protocol...
Re:Mojo Nation vs. Swarmcast (vs. Freenet, vs...) (Score:2)
Hm -- it is my impression that data is more persistent in MN due to the IDA splitting and heterogenous block-buying policies (i.e. some block servers rarely buy new blocks at all), plus MN's random-block-dropping policy.
In any case it is a goal of Mojo Nation that the data be persistent, whereas if I understand correctly, Freenet doesn't make persistence a goal. I hope I haven't incorrectly labelled Freenet's goals here.
Technically, data falls out of Freenet and Mojo Nation in the same way -- due to no servers wanting a copy -- but Mojo Nation contains an essential element which hopefully makes it possible maintain persistent blocks even when you are the only one who cares enough to maintain them: exchangeable network karma in the form of Mojo Tokens.
As the cypherpunks who investigated Ross Anderson's Eternity Service in the 1990's concluded, you can't have persistent data unless you have some way to motivate other people to hold on to copies that they personally are not interested in. Hopefully Mojo Tokens provide that motivation -- by holding on to blocks that they personally have no interest in (and very likely that they can't decrypt anyway), they earn network karma which they can use to store and retrieve data which they are interested in.
If I understand correctly, based on some comments by Ian Clarke, Freenet is happy with blocks that only a few people care about being dropped in favor of blocks that more people care about. That is a good design in many ways, but I hope that exchangeable Mojo Tokens will make it so that blocks are retained as long as anyone on the network wants those blocks to be retained, instead of as long as the servers who currently hold them want them to be retained.
Slashdot is a terrible medium for a real conversation, so I'll post this to the p2p hackers list...
Regards,
Zooko
Re:First Impressions (Score:2)
True. If you have an adequate dedicated server with plenty of bandwidth, it will be hard to top with swarmcast.
The benifit comes in terms of cost. Those beefy servers cost a shitload of money in terms of colocation and bandwidth.
A small, unfunded content shop may want to stream animations but cannot afford the cost of a central server to do it.
If they use swarmcast, however, the streaming is accomplished by utilizing bandwidth of everyone participating. No expensive server is needed, and performance is at least decent.
Real world uses for this technology are still lacking, so we shall see how swarmcast gets adopted IRL.
Re:Possible Security Issues (Score:2)
I'm not sure how you would deal with corrupted files, however. Any thoughts?
--Ben
Re:Possible Security Issues (Score:2)
-Justin Chapweske, Lead Swarmcast Developer
Re:What's with this gateway thing? (Score:2)
Thanks!
-Justin Chapweske, Lead Swarmcast Developer
Re:Ok, but... (Score:3)
-Justin Chapweske, Lead Swarmcast Developer
Re:What's with this gateway thing? (Score:5)
Re:How is this new? (Score:2)
With mojonation, I've never seen the performance that a solution like this should provide. Mojonation's not focused on performance, and Swarmcast is.
Mojonation's a lot more complex, too. With Mojonation, there's a searchable, virtual repository of files. Every host on the network is a peer.
With Swarmcast, only hosts downloading a given file are peers, and files are simply linked at web sites. (Personally, I think the proxy server approach is niftier, though.)
More info (Score:2)
Re:Mojo Nation vs. Swarmcast (vs. Freenet, vs...) (Score:2)
Zooko, I didn't mean that you don't care about performance. I know that you folks are trying to make things faster. I read it in the changelogs. But Swarmcast is focused (not "focussed", btw) on performance. It's the only purpose of Swarmcast.
They've aimed at something a lot smaller than what you folks are doing, but they've succeeded. You guys have more ambition-- it's not done yet, but when it is, it will be glorious.
Gott in Himmel! (Score:3)
And no, it's not like a bucket brigade. It's more like building a house with more than one bricklayer.
Re: slow hosts-- if I understand correctly, all swarmcast hosts maximize their available bandwidth, automatically balancing the download.
Each node downloads packets one at a time. If a host is slow, it won't grab as many packets, and someone else will.
If machines A and B are downloading, and A is 5x as fast as B, then in the same time, A will download 5 packets and B will download 1. If there are 12 packets in the file, A will download 10, and B will download 2.
Re:Could this be used as a "legitimate use of p2p" (Score:2)
A series of high-speed mirrors requires a lot of (expensive) fat pipes. Getting away from that requirement is the whole point behind swarm distribution.
Re:How would this work? (Score:2)
Re:Possible Security Issues (Score:4)
First of all, what type of security is going to be implemented to prevent hacks. It seems that it would be pretty easy to shore up a single server, or even several in a single datacenter, but it would be a daunting task to protect thousands of machines spread thoughout the world against hackers.
So long as the swarmcast client software doesn't have any holes, how is this an issue? If it does have holes, yes, this is a big deal, but if you've got a machine on the net with other security holes, that's an entirely different problem.
Secondly, what type of redundancy is going to be built into this system. Again, if a file is going to be served from a single centralised machine, it should ideally be fairly reliable, with multiple connections, and RAID to ensure continuous uptime. However, if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.
This is a pretty trivial problem to solve. There's still got to be a central server somewhere that tells you where to look for the peer bits. I don't know the details of their setup, but it certainly seems trivial to implement MD5-checking per chunk, so you can tell if there are bad chunks in your download. You just have to trust the main server's md5 signatures, and never end up trusting the peer servers.
This applies to the first point, too. If a serving machine is hacked, it's not a problem beyond that box because the real security comes from the MD5 key, or equivalent. It may be possible for a hacked server to serve bad code, but, given reasonable client design, it should be impossible for this bad data to actually be used by another machine. (Or, at least, should be impossible without big flashing lights and sirens screaming "You fool, don't do that!" :)
Re:Ok, but... (Score:2)
Ok, what is this, some kind of new "all your base" meme-storm?
Re:Possible Security Issues (Score:4)
Secondly, what type of redundancy is going to be built into this system. ...if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.
One of the main benefits of distributed anything is that there's an incredible amount of redundancy because every node is willing to contribute a little towards achieving your goal.
Are you trolling?
First of all, what type of security is going to be implemented to prevent hacks.
Against what? To prevent against file modification, checksums can be distributed from a single point since they're much smaller. All other security problems will be the same as any other P2P system.
--
Re:Could this be used as a "legitimate use of p2p" (Score:2)
The courts would laugh at this. It is distributing content that is legal to be distributed. That is the issue with napster, not peer to peer file sharing. In fact I used to get all my mp3 in a client/server setup.
Real World Usage (Score:2)
Maybe its time these developers list what exactly their software is being used for, and who is using it, to promote it, as opposed to waiting for groups like RIAA, MPAA to cry foul over them. Doing so would provide a nice argument, such as the ones EFF was looking for earlier.
As for the brief mention on security I browsed through, personally I don't see it as a big deal provided you know how to set perms, and or can configured some form of SSL behind it or something similar, perhaps make some rules on your firewall or IDS to ensure nothing gets broken along the way.
I can't wait to see how groups will react to cDc's Peekabooty, thats sure to be a kick in the ass for groups like RIAA, MPAA.
Freenet solves security concerns (Score:4)
Re:What's with this gateway thing? (Score:2)
--
What's with this gateway thing? (Score:3)
Still a cool idea, but if OpenCola wants everybody to put eggs in their basket, maybe somebody should release a GPLed server too...
--
Re:First Impressions (Score:3)
nlh
License (Score:5)
(a) Reverse assemble, reverse compile, or reverse engineer the Gateway (or any component or portion thereof), or otherwise attempt to discover any underlying Proprietary Information (defined below) of the Gateway;
Isn't reverse engineering explicitly granted by law?
(b) Sublicense, rent, sell, lease or otherwise transfer the Gateway (or any portion thereof) to any third party;
(c) Remove or alter any marks or designations indicating the ownership of copyrights, trademarks or other intellectual property rights of any party contained in the Gateway;
In fact I'd go so far as to say these conditions wouldn't feel out of place in a Microsoft EULA.
Do it in JXTA! (Score:2)
Valid use of P2P (Score:2)
Could this be used as a "legitimate use of p2p"? (Score:5)
Maybe this might be a very good argument in court... content distribution at high speeds.
Re:Ok, but... (Score:2)
Re:The Cola... (Score:2)
Re:Mojo Nation vs. Swarmcast (vs. Freenet, vs...) (Score:2)
How Swarmcast works (Score:2)
Swarmcast breaks a file into packets, which are encoded using Forward Error Correction (FEC), a mathematical technique that Swarmcast uses to make it easy to reconstruct the file. The encoded packets are then distributed randomly to the computers that have requested the file. These computers become nodes in the mesh (a temporary network) that Swarmcast creates for this download.
Each node receives only a portion of the original packets. But each node is also aware of some of the other nodes receiving packets. Even as a node receives a packet, it also rebroadcasts it to other nodes, so packets are rapidly swapped back and forth.
As each packet is received, the receiving node checks to see if it is useful in rebuilding the file; if it is, Swarmcast decodes that packet. As soon as a receiving node receives sufficient useful packets, it reconstructs the file. Thanks to FEC, a Swarmcast download is a bit like playing poker when every card in the deck is a wild card: as soon as you've received five cards, you can build a winning hand.
So there seems to be some amount of redundancy built in, provided enough servers are running at a time. When I tried it out, there was just one provider, but at 500Kbps.
So, what does "GPL'd" mean to you? (Score:2)
Using GPL'd code may put you into severe financial jeopardy. Make sure to check with your local Microsoft representative to make sure you have paid the appropriate "GPL tax". Remember: this applies even if you don't own any Microsoft software.
There are stringent limitations on the ways you can redistribute GPL'd code. If you redistribute any GPL'd tool in whole or in part, you must put a link to Midori's home page in the About box of your program.
The GPL is "viral", and can cause your program to become GPL'd too. However, instead of including the code directly in your new program, use a C trick called "linking". To do this, all of the GPL'd code you wish to use must be in a "library". If it isn't, take all the parts you need and compile them into a library, then distribute that with your program.
If you follow these tips, you can make great use of other people's hard work-- without running the risk of having to share any of your own. Good luck!
Re:Possible Security Issues (Score:2)
Well, that's an easy one. One centralized server which tells client what file to get, AND also tells md5 sum of the file & possibly also fragments (512k each, maybe).
Has Swarmcast done anything to prevent this from happening.
Now, to get an answer to that, you should read the article..
--
Saving content providors money? (Score:3)
This saves the content providor bandwidth at my expense. Bandwidth is, of course, not free and somebody has to pay for it. Ximian take note, if you don't want to pay for the bandwidth involved in being in the software distribution business, get out of that business. For Joe Average that installs Red Carpet and doesn't understand that it now includes this new feature, is going to be mighty pissed when he gets his bandwidth bill only to discover that he's been serving 10 gigabytes to strangers due to this feature.
great (Score:2)
;-)
And tie up more band width.
Although I acknowledge that this is the opposite of what is intended by the system.
maybe they ought to have a few distros on the system so we can help them test it out.
Check out the Vinny the Vampire [eplugz.com] comic strip
How would this work? (Score:2)
No matter how good you think humanity really is, I sure don't see it- I'm a greedy bastard and I want my bandwith.
Possible Security Issues (Score:3)
First of all, what type of security is going to be implemented to prevent hacks. It seems that it would be pretty easy to shore up a single server, or even several in a single datacenter, but it would be a daunting task to protect thousands of machines spread thoughout the world against hackers.
Secondly, what type of redundancy is going to be built into this system. Again, if a file is going to be served from a single centralised machine, it should ideally be fairly reliable, with multiple connections, and RAID to ensure continuous uptime. However, if you're serving tiny pieces of a file from thousands of boxen, it seems to me that if even one of those fragments doesn't make it all the way downstream, the whole file would be worthless. Has Swarmcast done anything to prevent this from happening.
Re:Ok, but... (Score:2)
This is the same problem currently being experienced with Mojo Nation [mojonation.net], a P2P file sharing system which implements a similar method of delivering blocks of data from various locations then re-assempling the requested file.
Perhaps OpenCola's solution to this problem is explained someware on their website but I looked and didn't see it.
--CTH
--
Ok, but... (Score:2)
-- Chris
Hello, thats the wrong license (Score:4)
You've successfully quoted the Sun Java Web Start license. Now please click the word 'license' in Swarmcast's about box and you will see that there are no such reservations or claims.
You are the weakest link. Goodbye!
First Impressions (Score:5)
Thing is, I don't see why anyone would use this. In reality, the transfer rates aren't as good as a single fast dedicated server (I can easily get 75-80KB/sec on this line), so there's really no gain on the client end from using this. On the server end, yeah you're using a lot less bandwidth. This might be useful for open source projects or other products that are downloaded by knowledgable people, but you're average computer luser isn't going to want to download 6MB of Java Runtime/Swarmcast Client just so they can save your company money by getting slower-than-normal downloads. (And I shudder to think what a distributed p2p network comprised mostly of 56k modems would be like, at least at the moment it seems most of the users online are broadband.)
Anyways, pessimism aside, here's what I'd like to see:
OK, I'm done for the moment. If they play this right I can see it helping out smaller outfits with knowledgable users, maybe eventually even going mainstream if they can convince people it's worth their while to install. It's definitely sparse on information in it's current form, but hey, it's a beta. It's a good idea, and I wish them luck.
-Jade E.
This is really good (Score:2)
Finally, a justifiable killer app for p2p (Score:2)
This is a great idea! It'll let high traffic sites like kernel.org distribute the work out to others. I mean it's nice that they have a 100 Mbit pipe thanks to donated bandwidth, but now they can more optimally use it by allowing users to share portions of files between each other.
Don't get me wrong, there are plenty of problems with this system also. Security, lowering the bar for newbies and other problems have been brought up in other posts. I won't rehash them here.
However, this will go a long way towards helping the open source community more easily share files -- without requiring that you beg some private money making companies to donate bandwidth. And that in my mind outweighs some of the negatives.
I especially like that they forbade sharing any files for which you don't have a copyright. Yes, I'm sure some people were thinking of using this for mp3's, warez, etc, but that would have killed the general acceptance of what is a great way to scale file downloading.
Just my $ 0.02 worth.