BitTorrent For Enterprise File Distribution? 291
HotTuna writes "I'm responsible for a closed, private network of retail stores connected to our corporate office (and to each other) with IPsec over DSL, and no access to the public internet. We have about 4GB of disaster recovery files that need to be replicated at each site, and updated monthly. The challenge is that all the enterprise file replication tools out there seem to be client/server and not peer-to-peer. This crushes our bandwidth at the corporate office and leaves hundreds of 7Mb DSL connections (at the stores) virtually idle. I am dreaming of a tool which can 'seed' different parts of a file to different peers, and then have those peers exchange those parts, rapidly replicating the file across the entire network. Sounds like BitTorrent you say? Sure, except I would need to 'push' the files out, and not rely on users to click a torrent file at each site. I could imagine a homebrew tracker, with uTorrent and an RSS feed at each site, but that sounds a little too patchwork to fly by the CIO. What do you think? Is BitTorrent an appropriate protocol for file distribution in the business sector? If not, why not? If so, how would you implement it?"
Re:Different torrent client ? (Score:5, Interesting)
rtorrent [rakshasa.no] watching a directory for .torrent would be the way to go. And then use unison [upenn.edu] to keep the .torrent directory in-sync.
How I would do it... (Score:5, Interesting)
...is quite straight forward in fact.
This has many advantages:
The beauty of this system is that it relies heavily on existing technology (BitTorrent, RSS, GnuPG, etc), so you can just throw together a bunch of libraries in your favourite programming language (I would use Python for myself), and you are done. Saves you time, money and a lot of work!
Furthermore you do not need to have a VPN set up to every destination as your files are already encrypted and properly signed.
Another advantage is: As this is a custom-built system for your use-case it should be easy to integrate it into your already existing one.
it's called dsync (Score:5, Interesting)
and you can find documentation for it here:
http://www.cs.cmu.edu/~dga/papers/dsync-usenix2008-abstract.html [cmu.edu]
It is rsync on steroids that uses a BitTorrent-like P2P protocol that is even more efficient because it exploits file similarity.
You may have to contact the author of the paper to get the latest version of dsync, but I am sure they would be more than happy to help you with that.
Re:Cisco already makes a product to do this - WAAS (Score:5, Interesting)
Bittorrent will transfer the differences too, if you make a new file overwrite an old one, it will replace any chunks which are different.
Re:Sure, why not? (Score:3, Interesting)
The main problem is that it introduces an extra vulnerability. With it the capability of very efficiently spreading malware and viruses around. Depending upon how locked down things are, it might not be a problem, but still it's definitely something to worry about.
And yes, I am assuming that somebody's going to get their machine infected or that somebody's going to break into the VPN traffic. Not necessarily likely, but still has to be considered.
Re:No, you fool! (Score:2, Interesting)
Oooooh... I can see the whole issue of throttling suddenly becoming very amusing as the corporate behemoths start slugging it out.
Re:load balancing? (Score:5, Interesting)
Just don't call it BitTorrent (Score:2, Interesting)
...or P2P when you first mention it to the CIO.
I would venture most CIOs' exposure to such things has been limited to what the popular media is pushing: BitTorrent == PIRACY.
I'd recommend sticking to vague terms like "Distributed file transfer".
This is a solved problem (Score:4, Interesting)
It's spelled 'NNTP'. Look at how Usenet newsgroups, especially for binaries, have worked for decades for a robust distribution model. The commands to assemble the messages can be scripted as well.
Similarly, the bottorrent files you describe can also be pushed or pulled from a centralized target list and activated via SSH as needed.
Re:Cisco already makes a product to do this - WAAS (Score:4, Interesting)
Use bittorrent to distribute git blobs. They are immutable & append only; perfect for something like bittorrent. All you'd really need is a good means of syndication via Atom, & end users capable of understanding SCM.
Comment removed (Score:4, Interesting)
Re:Sure, why not? (Score:1, Interesting)
It's not the pirates that prefer FTP, it's the Scene.
They're not the same. If those very FTP sites were the only places you could get pirated material online neither Hollywood nor the music industry would care about online piracy. Common people wouldn't know about the sites, just like normal users that copied game floppies from their friends in the 90s didn't know what a BBS was.
The "pirates" today are the "common people" (at least for my own generation), ie p2p users. The Sceners hates it when their releases "leak" to torrent-sites just as much as the original authors of the pirated content does.