Evolving Swarms with Swarmstreaming 246
Orasis writes "Applications like Bittorrent have broadly validated swarming technology in the real-world. Now, the inventor of swarming has released a new technology called swarmstreaming that allows smooth progressive playback of content, skipping ahead, and random access without downloading the entire file. It's an HTTP proxy, so browsers, podcasting, and RSS apps should be able to use it transparently. "
In other news... (Score:5, Funny)
Re:In other news... (Score:5, Funny)
in a statement issued by the SIAA they call humans who use swarming technology of violating copyright and tarnishing their image as insects by using swarming for illegal activites......
Re:In other news... (Score:2)
Re:In other news... (Score:2, Interesting)
Re:In other news... (Score:2)
No. They were placebo bees.
Re:In other news... (Score:2)
I mean, ow!
Re:In other news... (Score:2)
BZZZ!
Ow!
OK, OK, Sor-Ree!
Well, the linked site isn't using it... (Score:5, Funny)
Google Cache [216.239.57.104]
The text (formatted better) (Score:4, Informative)
Swarmstreaming: Swarming Downloads Evolved
I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly
The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.
Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix
Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.
The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.
Re:The text (formatted better) (Score:2)
This sounds very useful, hopefully we will see these features adopted widely in the future
-dk
Re:The text (formatted better) (Score:3, Funny)
A nice link from April 29th on his site... just figured I'd share... picture intensive... but its the "Rumsfeld Fighting Technique"
No more slashdotting? (Score:2, Funny)
Re:No more slashdotting? (Score:2, Interesting)
No, http is perfect, that's why nobody ever ever bothers to change it.
But I'm n
Re:No more slashdotting? (Score:2)
can't home serve due to lack of NAT traversal
Isn't that a transport layer issue?
can't serving large files is suicide because of lack of swarming
Isn't that a Phat Pipe issue?
no way to differentially update content or inform the user of updates
Heard of HTTP code 304 - Not Modified? Servers and browsers already know how to use it, and they do it transparently to the user. And maybe you should consider putting a "last modified" timestamp on your web page, to help stop clueless/hopeful people
Inventor? (Score:4, Funny)
Uh, so the killer bees are inventing technology now, and nobody is alarmed? I, for one, welcome our new technology-wielding killer bee overlords.
Uhoh (Score:3, Funny)
It's been done before. (Score:2)
But all those features mentioned in the
Re:It's been done before. (Score:3, Insightful)
Either you missed the word "swarming" here, or I've missed what exactly the QuickTime Streaming Server does.
How is this new? (Score:2, Informative)
Quicktime has had all that for several years. Apple called it "Instant On". I think both Real and Microsoft already use something similar.
Re:How is this new? (Score:2)
Re:How is this new? (Score:2)
Re:How is this new? (Score:2, Informative)
Re:How is this new? (Score:5, Informative)
Progressive downloading is where you download something like http://www.whatever.com/movie.mov in a web browser and it starts to play as soon as part of it is downloaded. You can then skip to wherever you want once you have downloaded that part (because at that point all you are doing is scrubbing through a movie file stored on your local machine.)
Streaming is where you load something like rtsp://stream.whatever.com/something.mov into a video player and it streams it to you. At no time during that process is anything stored on your local mahcine aside from what you are currently viewing and whatever the client has buffered ahead of that. Instant On instructs the server to skip the stream to another section.
Re:How is this new? (Score:2)
Re:How is this new? (Score:2)
So... (Score:4, Funny)
Re:So... (Score:2, Funny)
Re:So... (Score:2)
How does this work? (Score:3, Insightful)
Re:How does this work? (Score:2)
(Most movie releases these days are Xvid-encoded to fit on a single CD. Assume 93 minutes of movie and 700 megabyte file size, and it works out to just about exactly 1Mbps.)
Re:How does this work? (Score:2)
Do you often achieve the maximum download rate, and then sustain it for an hour and a half? If you do, you have a much better connection than most average ADSL users.
Also, if we're talking about swarming, it'll be even tougher to sustain the max download rate while upload clients are dropping in and out of the swarm.
Re:How does this work? (Score:2)
As for swarming making it tougher, have you ever actually used BitTorrent? I have never seen any sort of evidence of that. Although of course that would depend entirely on the quality and number of seeders in your particular torrent, so I guess once again there are people less lucky than myself.
But what I'm trying
Re:How does this work? (Score:2)
I pretty much have something going through torrent at all times. I see wildly varying download rates during a download of a same item. It can go anywhere from 100KB/s to 30KB/s and back up in one hour. And, this is with very active torrents with 100+ clients.
The inactive torrents are actually a lot more stable, because there are only a few seeds and they are not being leetched to deat
Bandwidth (Score:2)
I would fill it with 300GB drives.
Re:How does this work? (Score:2)
Re:How does this work? (Score:5, Interesting)
The technology to eliminate lag already exists and has been implemented. I have used it myself.
What's more, usually you cannot download one second of movie in one second of time, unless you have a crazy tricked out connection.
What nonsense. Have you ever downloaded a trailer from here [apple.com]? If the trailer starts to play immediately when you start downloading (i.e. the gray progress bar proceeds faster than the location marker), then you are downloading 1 second of movie in a time faster than 1 second. I can assure you that millions of people have a connection fast enough to do this.
This means that if you skip to a part you haven't seen yet, you will have to wait even longer for buffering.
Again, not necessarily. Buffering is when the streaming software requires that you download x amount of content ahead of the time you actually view it to account for inconsistencies in the stream or packet loss. If those can be eliminated, and connections made fast enough, there is no empirical reason why buffering must continue to be utilized.
Re:How does this work? (Score:3, Insightful)
And that's before you even deal with people needing to set up port forwarding.
Re:How does this work? (Score:2)
Can you provide any details? I don't understand how you can see something without a lag if you can't download it in real time?
What nonsense. Have you ever downloaded a trailer from here?
Those trailers are reduced to very small resolutions, so that you can watch them right away. They are hardly good enough to replace regular TV or way off from DVD or HD content.
Re:How does this work? (Score:2)
You can. That was the point I was trying to make. There are plenty of connections out there fast enough to download video in real time, provided the data rate is reasonable.
They are hardly good enough to replace regular TV or way off from DVD or HD content.
Well that's obvious, but I wasn't referring to DVD or HD content. Streams at those data rates are basically nonexistent on the net today because very
Re:How does this work? (Score:3, Informative)
You'd still probably want your nice expensive HDTV for stuff
Re:How does this work? (Score:2)
This can be getting the bits from 100 different sources, achieving the same effect, for potentially larger (height/width, such as HD content) video clips.
Now, how does this "already exist"?
Re:How does this work? (Score:2)
The capability to eliminate lag exists. I didn't say it can currently be applied to all forms of serving video or all data rates. Swarming might help there.
The grandparent post seemed to be taking the position that buffering would be a perpetual necessity, i.e. that as long as there was streaming, there would be lag. This is not necessarily the case. It is true that currently there is no easy way to avoid it for very large video sizes and data rates. However that is
Re:How does this work? (Score:2)
Is there a divx player for firefox as a plugin? (please say yes)
Re:How does this work? (Score:3, Informative)
Re:How does this work? (Score:2)
1.5 Mbps DSL here. Downloading compressed movies from Starz online movie service I regular have download times that are 2/3rds to 1/2 the playing time of the movie. Granted, these are heavily compressed, but they are better than VHS quality, and I would not call my connection "tricked out"
-josh
buffering (Score:2)
I dunno... (Score:2)
I mean, if you're downloading a feature film or TV show, do you really want to watch the middle before the beginning?
Re:I dunno... (Score:3, Insightful)
Re:I dunno... (Score:2)
Congratulations on completely ignoring my point.
Re:I dunno... (Score:3, Interesting)
But BT doesn't even allow you to watch first (Score:3, Informative)
Re:I dunno... (Score:3, Insightful)
I am a regular listener of the RTHK radio archive (a Hong Kong government funded radio station). The subscription is free. The audio clip contains the news, government ad (like don't throw rubblish blahblah) and the radio show itself... Not one bothers to cut the junk out. In a typical 2-hr session, 20-25 mins are news and other junk... It is just odd and a waste of time sit still and listen to "news" several months or even years ago...
How to get it (Score:5, Funny)
Le Duh! (Score:2)
Were we just beta testers? (Score:5, Funny)
Re:Were we just beta testers? (Score:2)
i wonder how long (Score:2, Insightful)
We have a new record! (Score:2, Funny)
Deeper Link (Score:2, Informative)
Translation: You got to pay to play (Score:2)
Especially if it's patented.
Nobody will want to touch it.
Re:Translation: You got to pay to play (Score:2)
The site says "I have no animosity towards Bittorrent"... no animosity, huh?
How generous not to begrudge people developing for free!
Server slowing down.. article text: (Score:4, Informative)
I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly
The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.
Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix
Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.
The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.
He links to http://onionnetworks.com/technology/swarming/#swa
Re:Server slowing down.. article text: (Score:2)
you mean bees can now make honey and milk?
Crichton (Score:5, Funny)
Lets not forget the price of entry. (Score:5, Informative)
* Object code for the entire suite of SwarmStream(TM) APIs, including WebRAID(TM), DirectCache(TM), Throttling, and THEX.
* Visualization tools to perform live inspections and demonstrations of what SwarmStream is doing during your application run time.
* One full license for WAN Transport(TM) Server (normally $2950), an HTTP server specifically designed provide advanced SwarmStream features such as self-healing downloads and automatic mirror discovery.
* One full day of developer training
* 20 hours of ongoing support
* One year of free upgrades for all of the above software.
* Unlimited right to use and implement SwarmStream technology for testing, prototyping, demonstrations, or creation of reference designs or applications. Production deployment requires an additional Deployment License.
* One-time fee: $25,000
Swarmcast is Free (Score:2, Informative)
Re:Lets not forget the price of entry. (Score:2)
Re:Lets not forget the price of entry. (Score:2, Informative)
It already has. Dijjer [dijjer.org] is open source, and was developed by the creator of Freenet [freenetproject.org]. It is still alpha, but is developing rapidly.
These guys must be pretty pissed that someone got slashdotted weeks before they did with some software that is entirely free, and does at least as much as what they claim their non-free software does.
Re:Lets not forget the price of entry. A BARGAIN! (Score:2, Interesting)
Consider internet radio or TeeVee
Streaming the same packets to each IP wanting them gets to be a real mess real fast. The beauty of this system is that so long as a recipients have adequate upload bandwidth to accomodate the stream bandwidth plus some delta (bigger delta will mean lower latency as parallism will increase with fewer stes away from the source) than the 'broadcaster' only needs enough bandwidth to get the stream out to a few people in order to each millions and mi
Orasis? (Score:4, Informative)
http://www.advogato.org/person/orasis/ [advogato.org]
-c
Re:Orasis? (Score:2)
Re:Orasis? (Score:5, Informative)
Shameless self-promotion of closed-source software, $25k USD for a dev-kit, and Taco fell for it. uugh.
more linkage (Score:2, Informative)
On a sidenote, I seriously doubt that he is the very first one to have thought of swarming. Swarming has been around since before 1999 (when he claims he invented it). He *may* be the first one to have applied it to p2p/networking however.
Re:more linkage (Score:2)
I rememeber the CIO of the bank I worked for wanting to spread data around on everyone's HD instead of getting a new RAID for the server. People were getting 4GB HD's, and only using 500MB or so, this was probably back in '97.
It wasn't a bad idea, there just wasn't a
Re:more linkage (Score:2)
Swarming Downloads (Score:2)
Swarming + streaming (Score:5, Insightful)
The only thing swarmstreaming changes is that it tries to download data in order, so you can use it more quickly, like any other conventional stream-oriented protocol (which is basically anything that uses TCP, along with various streaming media protocols). Now, the innovation is putting together streaming media with the power of swarming--imagine being able to feed a live TV feed from a single stream from the "seed". This is basically what multicast promised, but due to infrastructure problems, has yet to deliver.
Now, the devil is in the details. You're going to have problems with a distributed application that tries to deliver the same data to all nodes in the network at once, since you don't get all those nice properties of randomized distribution of different pieces. Some lossiness would definitely be desirable, meaning you don't really want to use it like a Web proxy. Furthermore, it's physically impossible to deliver data around the planet without many tens or hundreds of milliseconds of latency, so it's not good for interactive applications.
It might be a big win for TV-on-the-Web, though. Imagine if just anyone with a couple hundred kbps could serve a worldwide audience... all those Internet radio stations that are begging for donations to pay bandwidth costs could slash their total bandwidth needs, while upgrading service as well.
I'm not sure if this particular product is going to do the trick (swarmstreaming isn't a new idea, after all, and lots of people have been working on it), but anything that gets people thinking about it should help in the long run.
Re:Swarming + streaming (Score:3, Informative)
Re:Swarming + streaming (Score:2)
A kick ass technology would be to put a swarming like technology into a transparent caching web proxy server. Typically this would be installed by the
Re:Swarming + streaming (Score:2)
iceswarm? (Score:2)
Re:iceswarm? (Score:3, Interesting)
Re:Swarming + streaming (Score:2)
I would contest this on three grounds
1) If you use an operating system and filesystem which support sparse files then the missing blocks will
related presentation on javalobby (Score:2, Informative)
http://www.javalobby.com/eps/swarmstream/
Anyone notice the time this story was posted (Score:4, Funny)
similar (Score:2)
Re:similar (Score:2)
the concept is appealing BUT... (Score:3, Insightful)
Second, there is a lot of boasting , marketspeak and references to patents, business and whatnot. We're far, far away from GPL territory here. At least bittorrent is opensource.
Suprnova? (Score:3, Insightful)
Imagine if instead of having to wait a few hours downloading torrents off of Suprnova, you could simply browse through their catalogue (which I swear is bigger than Blockbuster's, and has music and tv shows), click something you wanted to watch, and BAM, its on.
Welcome to the future of Internet TV. I just hope the law doesn't fuck it up.
Re:Suprnova? (Score:3, Insightful)
Re:Suprnova? (Score:3, Insightful)
Would be a KILLER app. Being able to download and save, schedule, find 'if you like this you might also like...' shows. Works with tv and movies.
Man, that would be a powerful use of a couple megs of harddrive space.
Swarmstream audio/video presentation available (Score:2)
Justin's presentation [javalobby.com] (Flash). I just watched it earlier today.
Prevention of Downloading Bogus Files! (Score:2, Interesting)
This looks like it could be the next big thing in preventing the download of large bogus files.
Currently, in p2p programs (ala Kazaa, etc), you'd have to download the entire 600 MB file "Lord of the Rings.avi" (or "Busty Nurses. avi".. depending on your cinematic preferences), only to realize that someone has posted a bogus video in it's place.
Swarming the file (ie: "Lord of the Rings.avi"), would allow you to preview various portions of the file to ensure it's integrity... (personal integrity aside)
Re:Forget about browsers and RSS (Score:5, Funny)
You must be doing it wrong... you are masturbating at the same time right?
Re:Not pause! (Score:5, Funny)
Re:Forget about browsers and RSS (Score:5, Funny)
Is it just me, or isn't that the default implication for any new video streaming technology?
Oh wait... You must be new here.
Re:not free (Score:2, Interesting)
Imagine for example that you are a company distributing a maintenence release of a 40m application.
You seed this on a web server on your US east server, and you have the "swarm" running on US west, EU, Asia Pacific, etc.
Users connect to the proxy, but the proxy can use bandwidth from all of those sites. Assuming most users upgrade during the day, you're probably paying for a lot of bandwidth you're not using, that you could use to distribute the conte
Re:Something like this (Score:2)
That's the problem with flat-rate unlimited bandwidth for the consumer. If they can use as much as they want, they will use as much as you let them. Like how when you widen road, the traffic situation doesn't improve because more and bigger cars start using it.
Filesharing of video is a good example. Used to be a 30 minute episode would end up enc
Re:Something like this (Score:4, Insightful)
Re:Something like this (Score:3, Interesting)