Bufferbloat — the Submarine That's Sinking the Net

gottabeme writes "Jim Gettys, one of the original X Window System developers and editor of the HTTP/1.1 spec, has posted a series of articles on his blog detailing his research on the relatively unknown problem of bufferbloat. Bufferbloat is affecting the entire Internet, slowly worsening as RAM prices drop and buffers enlarge, and is causing latency and jitter to spike, especially for home broadband users. Unchecked, this problem may continue to deteriorate the usability of interactive applications like VOIP and gaming, and being so widespread, will take years of engineering and education efforts to resolve. Being like 'frogs in heating water,' few people are even aware of the problem. Can bufferbloat be fixed before the Internet and 3G networks become nearly unusable for interactive apps?"

cringley explains

[szo]

http://www.cringely.com/2011/01/2011-predictions-one-word-bufferbloat-or-is-that-two-words/ [cringely.com]

Re:pegged connection == latency, who'd of thunk it

[TheThiefMaster]

As an extreme example, say you request a 1GB file from a download site. That site has a monster internet connection, and manages to transmit the entire file in 1 second. The file makes it to the ISP at that speed, who then buffers the packets for slow transmission over your ADSL link, which will take 1 hour. During that time you try to browse the web, and your PC tries to do a dns lookup. The request goes out ok, but the response gets added to the buffer on the ISP side of your internet connection, so you won't get it until your original transfer completes. How's 1 hour for latency?

The situation is only not that bad because:
A: Most download sites serve so many people at once and/or rate limit so they won't saturate most peoples' connections
B: Most buffers in network hardware are still quite small

Re:I think buffers are a good thing

[Coriolis]

He's not arguing against application-level caching. He's saying that too much caching at the IP layer is confusing TCP's algorithm for deciding how fast the link between two points is. This in turn causes massive variability in how fast the data can be downloaded; or in your terms, how fast the video can be buffered (and, in fact, how much buffer the video player needs).

Re:QoS

[Shakrai]

Given that most traffic on a domestic connection is incoming, that doesn't help much.

It's not that hard to shape downstream traffic. Take a Linux router with two ethernet cards. eth0 is the LAN and eth1 is the internet. You shape eth0 with a maximum throughput of 75%-80% of your line speed. All of the downstream traffic has to go out on that interface so that's your opportunity to shape it. I do this at work and successfully share a 3.0mbit/s connection with 60+ employees. We use latency sensitive services like VoIP and RDP alongside streaming video and other large downloads without any major hassles. It stinks to lose some of your bandwidth because of this (you have to shape it to a number less than 100% of your line speed, otherwise buffering occurs at your ISP and your QoS scheme is defeated) but I'll take responsiveness over throughout any day of the week.

Re:First link in the first article

[GooberToo]

Demands for definition are a bit pompous...

A bit?

Even more pompous is making a post about it when everyone can clearly see, "bufferbloat" is shorter than constantly saying something tedious like, "excessive packet buffering in the entirety of a network path."

Perhaps this will help the uninitiated. The article describes a wide problem of excessive packet buffering in the entirety of a network path, which has been dubbed, "bufferbloat."

And this is a known problem, and fairly intuitive

[ebrandsberg]

let me summarize the problem that is being observed: On a given interface, if you have more buffer memory than is needed as packet buffer on the transmit side, it can induce latency. As an example, consider a 1Mb/s link. If you want to have a peak of .1s latency added by buffering at high load, then you want 1Mb*.1=12,500 bytes of buffer. If you have 1MB of buffer, then you have 8 seconds of buffer, therefore triggering the "buffer bloat" issue. Part of the problem is that buffer size would be set based on the top speed a piece of hardware could drive, i.e. if you want a 1Gb/s interface to be able to buffer .1s, then you use it at 100Mb/s, then it has 1s worth of buffer. In most home deployments where you have a router that may have a 1Gbps upstream, maybe 4 100Mb/s physical connections, and a 54Mbps wireless router, you probably have a shared buffer for all the interfaces. The result of this is that when using the 54Mb/s wireless, you can easily have the buffer over saturated, while the buffer size may be just right for the 100Mb/s interfaces.

What is the solution to this? Realistically, the alternative is to drop packets that have resided in the buffer longer than a configured amount of time, which causes it's own performance issues. Net result: TCP would slowdown for a period of time, but would speed up again resulting in a sawtooth behavior. This would result in periodic issues with other protocols as well, i.e. VOIP would have dropped packets every time TCP ramps up again, etc.

Solution: Don't download porn when you are trying to do VOIP calls.

Jim Gettys did the world a great service with this

[iwbcman]

I discovered this series of blog posts about 2 months ago, when he accidentally published one of his blog posts prematurely. I started reading it and followed the links and saw that this was a like a sleuth tale-if I had started reading this with his very first blog on the topic I would have had no idea where he was going with this. Now as to why this contribution by Jim Gettys does the world a great service:

• Gettys is not pointing fingers at someone. The problem he is describing is truly vast, and involves lots of different people in different industries(router manufactures, ISP's, kernel driver authors, carrier grade network manufactures, etc.) with, presumably, a myriad of different intentions. The problem has been building over a long time-this didn't start yesterday, and won't be solved in a short time span, without a concerted effort on the part of everyone involved in all of these divergent industries, who often have quite divergent interests.
• This approach that Gettys takes allows him to describe a problem which confronts everyone. By taking the high road and not pointing fingers he is able address an issue in such a way that a lot of the people who did contribute to this problem can recognize what they have done and own it, without being labelled, accused or feeling attacked. This should be a lesson to anyone who really wants to redress an issue that effects everyone.
• Gettys develops this theme over many, many blog posts. It makes for some of the best internet reading I have experienced in years. Things only gradually become clearer-not merely what the problem is, but also all of the issues involved in it. I can read away in the internet for months at a time and not learn as much as I did by reading this series of posts.
• Gettys knows what he is talking about. He developed this theme by talking with lots of experts -engineers at the ISP, people who played a pivotal role in the creation of the www and network specialists. He himself is not a network specialist, but he went out and met with people to discuss his findings and took clues and information from these exchanges to inform him and his quest to find out what was going on.
• The series is short on answers. It may prove frustrating to many that he offers so little in the way of solutions to this problem. But this this due to the fact that the problem cannot be resolved by you, the end user. To solve this problem means rearchitecting countless millions of devices and altering hundreds of thousands of lines of code in multiple OS's.
• Failure to redress this problem means that every effort to decrease latency by upping available bandwidth or upgrading network infrastructure will fail to deliver. If packets are not dropped fast, due to excessive buffering, the negotiation process fails, which invariably means congestion, which means latency-only something that addresses this issue has any chance of actually effecting change. Saying that this problem is just an issue already solved by QOS show that you don't understand the problem.
• One of the first thoughts I had reading this was: if the techs on wallstreet read this article they will inevitably exploit this issue to win precious milliseconds on the stock exchange-ring a bell?
• Any ISP could exploit this issue to offer a relative market advantage. Sadly when resolving an issue is in everybody's interest, market players will exploit the issue for their own relative gain. Getting everyone to actually tackle this is going to a gargantuan task.

Hats of to Jim Gettys. Thanks for your service.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Buffer bloat or inadequate bandwith

[OeLeWaPpErKe]

You forgot the tiny little fact that unless one pulls his connection to the limit with a lot of tcp connections, there isn't any problem.

Turn off your bittorrent client while you're playing starcraft online, and the problem disappears.

The post fails to explain what happens in the case of insufficient buffers - and dropped packets : it can take up to 2 minutes for tcp to recover from a single dropped packet (granted - on slow links or long distance connections). Would you really feel that interactive response has improved if things work fast 95% of the time, and then your web browser* - for no apparent reason at all - takes 2 minutes to load pages** ?

(yes, I'm an ISP's network engineer. Big buffers or small buffers ? Trust me, you want big)

* the fun thing about webbrowsers is that they open lots of tcp connections, then barely send any packets at all (ie. connection generally closes after 4-5 packets tops - sometimes after 2 packets). If you lose the first packet in a connection, which is quite likely when browsing, the SRTT algorithm has no choice but to wait 2 minutes before retry - guaranteeing the user will have to interfere (ie. "F5"). This results in the massive deterioration of web browsing experience with trivially small packet loss. Unless you've never wondered why internet is near-unusable with 0.1% packet loss on your link, and nothing at all gets through at 1% packet loss. You'd think 99% correctly transmitted packets would translate in 99% of bandwidth available, no ? (in case you have this problem : a simple trick to put everything through a single tcp connection. ssh -D 1025 server_at_work; set up firefox to use socks5 proxy at port localhost:1025. You'll have 40-50% of your link bandwidth available on recent windows or linux)
** Just try, go to a big company that's upgraded to cheap gigabit switches, with tiny buffers. Ask them if, perchance, they've been experiencing sudden "timeouts" all of a sudden. Ask again if they like this.

The way to fix this - not that I'm expecting political interference from large groups of idiots - all large groups are large groups of idiots, because most people are idiots in most subjects - to go in a sensible direction all of a sudden, so "let's get the mob to 'fix' this" doesn't work regardless of good intentions - is not to go with small buffers but to have intelligent queuing algorithms in all devices. Of course, bittorrent will always cause this behavior, because one of two things will happen when bittorrent opens it's 5000 connections
1) either routers slow down bittorrent traffic in favor of http, much better performance, but results in underwear kids who haven't seen sunlight in a year shouting "NET NEUTRALITY !"
2) or they "treat all traffic the same" - and with tcp the one with the most connections "wins" the most bandwidth - meaning if you open 500 connections, your web browser is only going to get 1/500th of the link bandwidth - resulting in abysmal performance

This is what network engineers mean when they're saying bittorrent is destroying network performance. As to what lawyers and politicians mean, yes that's something else, and frankly, I don't care.