BitTorrent Calls UDP Report "Utter Nonsense"

Ian Lamont writes "BitTorrent has responded to a report in the Register that suggested uTorrent's switch to UDP could cause an Internet meltdown. Marketing manager Simon Morris described the Register report as 'utter nonsense,' and said that the switch to uTP — a UDP-based implementation of the BitTorrent protocol — was intended to reduce network congestion. The original Register report was discussed enthusiastically on Slashdot this morning."

Best of intentions

BT may have the best of intentions here in developing this experimental protocol, but this quote leads me to believe that their understanding of the problem is terribly naive:

It so happens that the congestion control mechanism inside TCP is quite crude and problematic. It only detects congestion on the internet once "packet loss" has occurred - i.e. once the user has lost data and (probably) noticed there is a problem.

Packet loss is a normal and deliberate mechanism by which TCP detects the maximum thoughput of a path. Periodically it increases the number of packets in flight until the limit is reached, then it backs off. You have to test again from time to time, in order to increase throughput if more capacity becomes available. This in no way incurs "loss of data" or a noticeable problem. Packets lost due to congestion window growth are handled by the fast retransmit algorithm, which means that there is no timeout or drop in throughput (that would be pretty stupid if the whole purpose of growing the congestion window is to _maximize_ throughput).

I wonder if Simon Morris was merely oversimplifying for the benefit of the layman, but I still find that statement disturbing. As I sugggested in the other thread, it really sounds like they're going to reinvent TCP (poorly). That's not to say you couldn't design a better protocol specifically for point-to-multipoint transfer, but I question if they're on the right track here.

Re:Best of intentions

I agree with you Seanadams, but I just finished an interview with Simon Morris [networkper...edaily.com] and it's not that he's saying that the way TCP handles packet loss is a particular problem, he just thinks he can do better.

BitTorrent essentially already has it's own methods to deal with dropped packets of information - it gets the information from elsewhere. Moving to UDP eliminates the triple handshake, and it eliminates throttling down packet sizes in response to a dropped packet.

The only problem is that it also eliminates the Layer 4 [transport] traffic congestion safeguards, which is why BitTorrent is looking to establish new and better ones at layer 7 [application].

Re:Best of intentions

Because application-specific knowledge allows easier and often better optimizations than application-generic protocols, which have to be good enough for all applications at the expense of top end performance for specific applications. Isn't it obvious?

Re:Best of intentions

On top of this TCP hasn't seen a major update since the 80's. Most of it was implemented to deal with a very different internet than the one we have today. If you can side step TCP's shortcomings by doing the congestion control more efficiently at the application level then why not give it a shot?

Re:Best of intentions

Maybe he has spent years working with the technology he is trying to one up?

This relates directly to a reply I just finished on another thread regarding whether a degree is required for success.

This Morris character may be right, he may be wrong, but you citing the education level of his rivals blows your point out of the water IMO.

Re:Best of intentions

I know, how dare anyone try to improve things without being old and having a PHD.

Fucking punk ass kids. Next thing we know we will have guys with only master degrees trying to write operating systems.

Re:Best of intentions

I wonder if you know what you're talking about. TCP is a great general-use protocol, as is UDP. But for specific cases, like this one, developers will tend to roll their own as an extension to UDP to fit their needs.
 
Take any modern networked game. It will not use straight TCP, as that's stupid, nor will it simply use UDP because that doesn't work, either. TCP is fine if you're writing apps that require data to get to the other end, without regards to time taken, but if you need a happy middle-ground you need to do it yourself.

Re:Best of intentions

does Simon Morris think that he, single-handedly, can do better than the folks at the IETF, most of whom have PhDs in computer science?

You really have a very strange over-estimation of the value of a PhD. It doesn't mean you're a super-genius, you're smarter than everyone (or even anyone) else, or that you're always right. It simply means you've been willing to go through some schooling. That's it. Hopefully you've learned something from that education.

There's plenty of examples of non-PhD's making major contributions. The WWW was largely invented by someone with a degree in physics (undergrad I believe), with no degree in computer science. Linus Torvalds only attained a mere masters degree in Computer Science, but yet his OS seems to have become a bit more successful than quite a few other OS's written by people with more education.

Re:Best of intentions

Please mod parent up. This place is so damn full of armchair wannabe network experts who've clearly no understanding of how TCP congestion avoidance works it's bordering on the physically painful.

Re:Best of intentions

TCP does two things at the cost of some overhead: it ensures packets arrive in order and it ensures they arrive. While doing that, it also has to ensure that in the case of network congestion, packets are resent within reasonable time while being fair and allowing each TCP connection an equal share of the speed and bandwidth. The problems TCP solves are at the root of the success of bittorent; bittorrent is extremely good at spreading traffic in such a way that links that have those problems are avoided. It can do this since the order in which the packets arrive does not matter and in fact it does not matter whether they arrive from a certain host at all; it can simply request a lost packet from another host minutes or hours later. In the case of TCP/IP there is no provision to handle such a case; it keeps trying to get the packets at their destination in the right order as quickly as possible.

So none of the problems that TCP solves affect bittorrent and all the overhead that TCP causes, however small, serves no purpose in this case. Instead of many small TCP ACKs, resends, negotiation, and what else TCP does, bittorrent will do just fine with one status update every now and then, which it can conveniently combine with the packets that are sent the other way anyway. Therefore, IMHO, using UDP for bittorrent is fine; it will help spread bittorrent traffic even better over the fastest links while using less bytes of network traffic and it might even end up making it easier on the Internet since now bittorrent traffic no longer has to fight with other TCP connections for a fair share of the bandwidth; it can be tuned to get just a little bit less.

Of course they can fuck it up completely and fill the poor pipes of the Internet with loads of packets that never arrive, but they don't have to; it is not inherent to this solution and therefore such rumours should be classified as... FUD.

Disclaimer: I did not RTFA and now practically nothing about TCP/IP :-)

Re:Best of intentions

There are a lot of good reasons why BitTorrent should use a UDP file transfer protocol, but I probably wouldn't put TCP's congestion control mechanism on the top of the list.

If you're going to argue UDP, you might as well bring out the major benefits:

* Going to a NAK-based or hybrid NAK/ACK-based protocol which can significantly improve performance over high latency or poor connections

* Multicast - assuming anyone implements IPv6 or multicast over the internet :)

* NAT to NAT transfers (you can do it with TCP, but it is just harder and you generally have to build a user-space TCP stack anyway).

* Faster start time since you no longer have to do a three-phase startup and all the annoying things Microsoft does to prevent people from starting too many per second

There are plenty of UDP-based protocols with TCP-friendly congestion control mechanisms out there and plenty of research into the subject.

The biggest problems I see happening here revolve around different BitTorrent clients all reimplementing uTP and doing a poor job at it. I'd like to see a spec for uTP and a public domain implementation to help minimize the problems that could pop up.

Re:Best of intentions

Detecting maximum throughput via packet dropping is really bad in high-latency links and in applications that need low latency.

I disagree. Please be more specific. First, what exactly do you believe is the problem? Secondly, how else would you do it (on an IP network)?

It is also apparently easy to implement TCP in such a way that overall transfer speed takes a nosedive when latency gets high, as evinced by Microsoft having done just that.

So you're saying it's possible to implement it with a bug? I've recently found a heinous bug in a recent Redhat kernel which would result in _deadlocked_ TCP connections. It happens to the best of us.

Re:Best of intentions

No it isn't. The size of the window can be adjusted to compensate for this. With a larger window, you get more throughput over high-latency links at the expense of having to wait longer for retransmits when they are needed. Modern TCP stacks dynamically adjust the window size based on the average RTT for any given link.

The Register

Nonsense? From The Register? Ya don't say.

In other news...

My ISP, Comcast, is already on top of this new bittorrent over UDP idea and has summarily blocked all UDP traffic.

So, I wonder if I'll be experNO CARRIER

unknown host slashdot.org

The problem is...

we have an opportunity to detect end-to-end congestion and implement a protocol that can detect problems very quickly and throttle back accordingly so that BitTorrent doesn't slow down the internet connection
----
The major problem I see is that UDP doesn't play as nicely as TCP. Not by a longshot.

As soon as TCP notices a single packet loss, the Jacobson Algorithm kicks in and it's throttled to maybe 50-60%, and raises the limit slowly. I highly doubt that uTorrent's reworked version of UDP will play this nicely.

As soon as TCP's throttling kicks in, space will be cleared in the tubes. uTorrent will be able to send more data through UDP without noticing any loss, so it'll quickly move to fill this space. Then, TCP gets hit with more data loss - and goes slower. It seems like a vicious cycle.

Re:The problem is...

The major problem I see is that UDP doesn't play as nicely as TCP. Not by a longshot.

That statement makes no sense whatsoever. It's the same as saying "IP doesn't play as nicely as TCP". They are not comparable.

You have to be talking about whatever transfer protocol might be implemented ON TOP OF UDP, because UDP is merely a datagram protocol. It's nothing more than an IP packet plus port numbers.

In other words, I could write a simple program which blindly fires UDP packets at the rate of 1GB/s. This would kill my internet connection. I could also write a program which transmits one UDP packet per hour. This would have no effect. See what I mean? It's entirely a function of the application.

UDP and DHT are a nightmare

everyone thought dht was great too, but I found every time I used it it caused massive headaches. I would jump on a popular torrent and for days afterward I would be having poor performance, checking logs etc would show several dozen connection attempts per second on the utorrent port, even 2-3 days after I was done with the torrent because the DHT tracker was still advertising my IP address. I'd have to release renew to bring my performance back up. This was with a fairly standard Linksys router. Any situation where the other party might not just get the message that I'm not there anymore is bound to lead to headaches on popular torrents.

Remove TCP-like requirements, and it's a WIN.

TCP guarantees in-order, mildly error-corrected, delivery of transmitted *DATA*. Not packets. It is a streaming protocol where the data being transmitted is of unknown and indeterminate length, or open-ended length. Since BT already doesn't care that files arrive in pieces at the beginning, middle or end (well, it does a little, but not enough to matter) then you can relax and basically eliminate one of the TCP guarantees right there: in-order delivery of data.

You can eliminate sliding windows, ACK-based retransmits, fast-retransmission, and pretty much every other mechanism that TCP uses to guarantee in-order delivery of data. You can simplify it and the application itself beautifully, and provided it correctly throttles back based on detected packet loss (it MUST be exponential back-off,) you end up with a net win for those reasons. The application can set up its own optimised data structures that don't necessarily have (but likely will end up having anyway) the overhead of an OS-backed TCP stack.

I mean who cares if you miss pieces, until the end when you can re-request them?

Heck, there are already P2P apps that use a UDP-based transfer mechanism and they are WAY less impactful on systems that a typical BitTorrent stream is. They way the hell slower, too, but that's not the point.

I do think there is a point that bears repeating: BT *MUST* have exponential back-off. If it doesn't there is logic already built-in to core routers, ISPs, and firewalls that WILL drop the connection more severely when the endpoints don't respond properly to an initial packet-drop attempt at slowing them down.

There are some really nice academic papers about it, and there are lots of algorithms and choices that companies have. They all assume TCP-like back-off on the endpoints, and they ALL uniformly punish greedy floods.

What BitTorrent REALLY needs

The probablem with BitTorrent is not that it uses a large amount of bandwidth, but that it's using the wrong bandwidth. In every country other than the United States, international bandwidth is substantially more expensive than local bandwidth, and often in short supply. Local bandwidth is cheap, or even free. Even in the US, inter-ISP bandwidth has the same cost issues, but is plentiful.

What I've never understood is what's the excuse for not implementing a peer selection algorithm that prioritises nearby users. Even a naive algorithm is going to be vastly better than a purely random selecton. Simply selecting peers based on, say, the length of the common prefix of the IP address will often produce excellent results. Why in God's name should I transfer at 0.1 kbps from some guy in Peru, when a peer down the road could be uploading to me at 500 kbps?

The truth is that the BitTorrent folks are not playing ball with ISPs. In reality, I think most major ISP could care less about copyright violation, or excessive bandwidth - it makes people pay for more expensive monthly plans - but they DO care about international bandwidth costs.

If they just took 10 minutes to revamp the peer selection algorithm, they would reduce the impact in ISPs enormously, and then they woudldn't be villified and throttled.

Reducing network congestion

If they want to improve network congestion why not start by implementing a better peer selection algorithm. IIRC currently peers are selected at random. A network topology aware peer selection algorithm might improve network congestion a great deal. Currently I see peers which are on another continent being 'preferred' (to due the randomness) to peers on my own ISP's network, with which I have a 50+ mbit connection.

Where where?

"Hear", not "here".

Re:Where where?

Their their now, let the poor unedjumicated bastard post in piece.

I mean, those in glass houses shouldn't through stones. ;)

Re:Where where?

You misspelled bastige.

If you're glass house is made of bullet proof glass then it's OK to throw stones.

Re:Where where?

I'm not sure, bouncing stones could be pretty painful.