Forgot your password?
typodupeerror
The Internet Networking Technology

Bufferbloat: Dark Buffers In the Internet 124

Posted by timothy
from the perfect-for-dark-fiber dept.
Expanding on earlier work from Jim Gettys of Bell Labs with a new article in the ACM Queue, CowboyRobot writes that Gettys "makes the case that the Internet is in danger of collapse due to 'bufferbloat,' 'the existence of excessively large and frequently full buffers inside the network.' Part of the blame is due to overbuffering; in an effort to protect ourselves we make things worse. But the problem runs deeper than that. Gettys' solution is AQM (active queue management) which is not deployed as widely as it should be. 'We are flying on an Internet airplane in which we are constantly swapping the wings, the engines, and the fuselage, with most of the cockpit instruments removed but only a few new instruments reinstalled. It crashed before; will it crash again?'"
This discussion has been archived. No new comments can be posted.

Bufferbloat: Dark Buffers In the Internet

Comments Filter:
  • the existence of excessively large and frequently full buffers

    Seems better than the existence of excessively large and seldom if ever full buffers.

    • by Anonymous Coward on Friday December 02, 2011 @10:38PM (#38246918)
      You never want a full buffer. At that point, it ceases to do its job.
    • by skids (119237) on Friday December 02, 2011 @10:39PM (#38246926) Homepage

      Seems so, but isn't. For TCP traffic, a shallow buffer that drops traffic will result in more goodput than a deep buffer. Which is the point.

      • by pla (258480) on Friday December 02, 2011 @11:00PM (#38247022) Journal
        Seems so, but isn't. For TCP traffic, a shallow buffer that drops traffic will result in more goodput than a deep buffer. Which is the point.

        Yes and no...

        If you don't (or only rarely) fill your buffer, a smaller buffer introduces less latency than a large one, while still allowing you to maximize throughput. If, however, you usually have your buffer full, you increase latency for literally no benefit, since you've already maximized throughput simply through resource demand.

        The former will occur when your average load falls below your actual bandwidth, and allows you to get the most out of your link. The latter occurs when you consistently exceed your bandwidth, in which situation you may as well not even have a buffer, because it only increases latency without increasing throughput. That describes TFA's real point.

        What he suggests amounts to actively choosing between those two conditions - If your average demand falls below your link speed, a larger buffer will help smooth the load over time. If, however, your average demand exceeds your link speed, throw away the buffer because it doesn't help.

        But as per the GP's point - If you have an always-full buffer, you literally gain nothing but latency.
        • by CyprusBlue113 (1294000) on Friday December 02, 2011 @11:08PM (#38247046)

          The problem with buffers is most all of the time they are configured by size in bits. They need to be sized based on bit flight time of the circuit, which is in delay ms times throughput in bits. The disconnect between those values is a problem in *either* direction, especially past the retransmit threshold on the above side.

          Buffers should be dynamicly sized based on flight time of data on the specifc link, and ideally kept updated. WRED is also highly suggested.

          What really exacerbates the issue is devices with buffers that must be the same size for all links on X (be it card, slot, or chassis).

          • by TheLink (130905) on Saturday December 03, 2011 @05:04AM (#38248430) Journal
            You don't necessarily have to size them in flight time of the circuit.

            What you can do is have huge buffers, but just drop packets that are older than say 50 milliseconds since the time they entered the device (if the link/hop is supposed to be fast and low latency).

            If the link is slow and/or high latency, you may wish to use higher values - 100 milliseconds. But not too high. I'm no networking expert but I don't really see the purpose of adding hundreds of milliseconds to a hop just to save a few packets that are likely to be dropped anyway, or should be dropped as an indirect signal that whoever is sending those packets should slow down.
        • by skids (119237) on Friday December 02, 2011 @11:16PM (#38247076) Homepage

          What he suggests amounts to actively choosing between those two conditions - If your average demand falls below your link speed, a larger buffer will help smooth the load over time.

          That's a pretty simplified way of putting it, but basically correct. Major equipment vendors have been slow to adopt more advanced queuing strategies (Stochastic Fair Queuing integrated with some of the more advanced flavors of early discard.) Fortunately we're budgeted for and piloting a shaper for purchase soon, and this time around have a chance to get something both well supported and cutting edge.

          Personally I pine for ATM's ABR CoS with it's fast end-to-end congestion notification, but as history has shown us, the inevitable fate of the tech world is for the inferior to be gradually, painfully, and kludgingly adapted to become the same thing as the technologies it displaced through lowballing. In this case, that inferior thing being IP/ethernet.

          • Re: (Score:1, Offtopic)

            ...the inevitable fate of the tech world is for the inferior to be gradually, painfully, and kludgingly adapted to become the same thing as the technologies it displaced through lowballing.

            In this case Gettys is acting as an apologist for the municipal and/or corporate resistance to augmenting American infrastructure to 21st century standards. Whenever you have an elite few pinching pennies at the expense of the many, then you know that the ZOG machine is behind it. Their propaganda will explain everything away using bureaucratic technobabble while the final solution* to the problem is painfully obvious to anybody.

            * the final solution being to railroad more fiber directly into the proces

            • No, because it only takes one congested link, and then the buffers start filling up along the entire path. In other words, my crappy d-link router can start filling up the buffers in the core Internet. OK, not just mine, but a few dozzen crappy routers, and it can be a regional problem. And we all know those "free" cable modems are of the highest quality...
              • by jg (16880)

                You can have an overall congested network. I've seen this on occasion.

                But it is very easy (and even more common) for you (or people in your house) to do it to you, than to have the overall ISP network congested. This is something a simple file copy can/does do to you, in practice.

                Some ISP's run AQM properly (e.g. RED) in the cores of their networks; some do not. On the ones that do not, you'll see problems at peak hours. Similarly on corporate networks.

            • I see what you did there.
            • by jg (16880)

              Heh. The worst problems (as far as I've seen) we have are in the edge.

              In our hosts and home routers (most of which are Linux boxes).

              The problem occurs anytime you are next to a bottleneck, and the "municipal and/or corporate resistance" have now built out the core of the internet to the point that the problem is most severe at the very edge, including your laptop/handheld device.

              *EVERYONE* has made the same set of mistakes. I have done so too at times....

          • by Animats (122034) on Saturday December 03, 2011 @03:56AM (#38248234) Homepage

            That's a pretty simplified way of putting it, but basically correct. Major equipment vendors have been slow to adopt more advanced queuing strategies (Stochastic Fair Queuing integrated with some of the more advanced flavors of early discard.)

            Right. The problem is not big buffers, per se. It's big dumb FIFO queues. There's nothing wrong with one big flow, like a file transfer, having a long latency, provided that other flows with less data in flight aren't stuck behind it. That's what "fair queuing" is all about. Each flow has its own queue, and the queues are serviced in a round-robin fashion. (With stochastic fair queuing, some hashing is done to eliminate some of the bookkeeping on flows, but the effect is roughly the same.)

            I figured this out in the early 1980s (see RFC 970 [ietf.org]) and by the late 1990s, it was an established technology. We shouldn't be having this problem at this late date.

            I wonder how much of the trouble comes from devices that are doing TCP-level processing in the middle of the network. Stateful firewalls and ISP ad-insertion engines [isp-planet.com] can introduce substantial latency.

            If you want to test for bad behavior, try running two flows, one that never has more than one packet outstanding, and one that just does a big file-transfer like operation like a download. If the latency of the low-traffic flow goes up to the same as that of the bulk flow, there's a big dumb buffer in the middle. If the packet loss rate of the low-traffic flow goes up, there's a small dumb buffer in the middle.

            • by sgt scrub (869860)

              big dumb buffer

              Nice. Stupid ToS settings, devices that always send the maximum data until they receive 100 ACK window size adjustments telling it to SLFD, and big dumb buffers. I like it!

            • I wonder how much of the trouble comes from devices that are doing TCP-level processing in the middle of the network. Stateful firewalls and ISP ad-insertion engines [isp-planet.com] can introduce substantial latency.

              I doubt that the processing itself is the cause of more than a few milliseconds of latency, but the machines doing it may have been configured with large buffers not because the processing needed them but because those configuring them thought erroneously that bigger buffers are always better

            • by jg (16880)

              To solve this, I think we need both AQM fully at the edge of the network, and some sort of "fair" queueing at the edge. The headache is that the classic AQM algorithms won't work in the edge case (and are flawed). "Fairness" is in the eyes of the beholder and a complex question. You may consider it "fair" your kids get half the bandwidth you do, for example. But having a situation where talking to a local CDN 10ms away gets tons more bandwidth than something across the globe may not be "fair" in your vie

          • ECN exists. It works. Just like S/MIME. And similarly, no one uses it.
      • by icebike (68054) on Friday December 02, 2011 @11:22PM (#38247084)

        Seems so, but isn't. For TCP traffic, a shallow buffer that drops traffic will result in more goodput than a deep buffer. Which is the point.

        Exactly.

        Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need.
        The deep buffer just make it worse for everyone.

        Oh, and And just as a Car Analogy is inappropriate to describe TCP traffic the Airplane Analogy is worse.

        • by skids (119237)

          Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need. The deep buffer just make it worse for everyone.

          There's something to be said for shaping at intermediate hops, even with ECN (and especially when ECN isn't implemented) but it has to be done in a manner that it doesn't add latency out of proportion to the unladen RTT.

        • by ObsessiveMathsFreak (773371) <obsessivemathsfreakNO@SPAMeircom.net> on Saturday December 03, 2011 @12:37AM (#38247396) Homepage Journal

          What we need is a ferry analogy.

          Packet transmission is like a ferry, crossing a river at fixed intervals. But ferry sets off when it is full rather than at set times.

          People wait at the shore and generally don't have to wait too long as the ferry is pretty fast and only needs a few people to fill up. For most people, walking onto the ferry involves very little waiting before the ferry actually departs and crosses the river.

          Buffer bloat is when big buffers act like ferrys with huge capacity. People enter a huge 2000 passenger capacity boat, and are let on by their hundreds with seemingly no delay. But the ferry will not depart until it is reasonably full. So the people who got on first may have to wait for hours before the ferry actually departs and crosses the river.

          It is clear that bigger ferries are no substitute for more ferries....or smaller rivers. Or possibly a bridge. In any case, you can get away without introducing cars or airplanes, so my job is done here.

          • "May the Maths Be with you!"

            May the understanding of reason be with you:

            http://bit.ly/dYaWUc [bit.ly]

          • It is clear that bigger ferries are no substitute for more ferries....or smaller rivers. Or possibly a bridge.

            Or intergalactic starships, and teleporters?

            What were we talking about, again?

          • by skids (119237) on Saturday December 03, 2011 @02:20AM (#38247888) Homepage

            That analogy doesn't quite do the trick. TCP windowing is a bit more sophisticated than that. You can think of it maybe as a commander sending couriers out to support a mobile squad through hostile territory. If too many of them never make it to the squad, or back, he sends them less frequently so they can sneak through more discretely. If the troops make it through then he sends them faster because the more ammo he can get through the better. But he also has to decide how many men to put on courier duty. If the couriers take too long the squad has obviously moved further away from the base, and if he waited for the next one to return, he wouldn't be sending enough ammo. If the couriers return quickly, he can make do with less couriers.

            Big buffers are like a flimsy rope bridge in the courier's path that takes a long time to cross. Couriers have to wait on one side because only one can cross at once, but the large groups waiting at the side of the cliff is more likely to get attacked. Until they do get attacked, however, the commander starts to think the squad has moved very far away, so he puts more couriers on duty. Since he thinks the squad is far away, he is not expecting them to return for a longer amount of time, it takes him longer to realize that they are starting to go missing entirely.

            One of the best solutions to this problem turns out to be for some of the couriers to randomly go AWOL, and for more of them to go AWOL the bigger the crowd at the rope bridge gets. This basic concept is called Random Early Discard, and there have been a lot of ways invented for deciding who goes AWOL and why. If some of the couriers go AWOL, the commander thinks they are being attacked, so he slows down and also takes some troops off courier duty.

          • So none are an appropriate analogy.

            HTH

          • by Troke (1612099)
            Maybe the passengers need swimmies? or life jackets so they can be tossed overboard! either way, +1 to explanation being clear, concise, informative, and funny.
          • by T Murphy (1054674)

            In any case, you can get away without introducing cars

            The only ferries I've ever been on were for cars, so you kinda did. Maybe you should've used a paraglider analogy, I've never seen a car use a paraglider.

          • by MagicM (85041)

            But what if I want to bring my goat?

        • by sgt scrub (869860)

          Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need.

          Unfortunately, early notification doesn't work with a ton of wireless devices. Their drivers have minimal abilities to be controlled and they always send data at the speed of their negotiation. .eg if they connect at 11g they always send data at that speed and always send acks with window size adjustments to speed traffic up to that speed until they receive multiple window size adjustments telling

      • by Idbar (1034346)
        I'm not sure about goodput, but for sure shallow buffers result in better latency.

        ECN helps to increase goodput, and AQM can help to keep high thoughput. The main concern of some (at least my research topic) is how to implement AQM to spread traffic spikes such that the link utilization increases while buffer occupancy reduces.
      • by Surt (22457)

        And the right balance between buffer size, drop percentage, and throughput should be measurable. But I bet those lazy bastards at cisco have never thought to measure performance, which is why no one uses their equipment.

  • Cringely again... (Score:5, Informative)

    by beetle496 (677137) on Friday December 02, 2011 @10:34PM (#38246890) Homepage

    Cingely has been writing about this all year. He cites Jim Gettys too. See: http://www.cringely.com/tag/bufferbloat/ [cringely.com]

    • Re:Cringely again... (Score:5, Interesting)

      by hairyfeet (841228) <bassbeast1968 AT gmail DOT com> on Friday December 02, 2011 @11:08PM (#38247044) Journal

      Uhhh...I just read the link but am a little confused. Even Cringley points out at the first of his article that originally TCP was written for a VASTLY different and weaker network than we have now, so instead of trying to make the networks go back to a mid 1980s design, wouldn't it be smarter just to update TCP to take advantage of new tech advances?

      Now i'm not really a heavy network guy so excuse me if I put it in more of my lingo, but lets compare it to something I've got more first hand experience with, hard drives. If you don't write the controller code to take advantage of the large cache frankly the cache becomes worthless but if you DO write the controller code to take into account the size of the buffer it makes a BIG difference, so much so that I've seen 5400RPM drives whip 7200RPM drives simply by having better cache management.

      So wouldn't the right way to go be to update TCP for the times? i mean we didn't slow computers down so we could keep PATA or PCI, we came up with new tech like SATA and PCIe to take advantage of the faster throughput. Shouldn't we do the same here as well?

      • by mellon (7048) on Friday December 02, 2011 @11:40PM (#38247160) Homepage

        Buffer and cache are not the same thing. Packets are written to a buffer once and read from it once. Caches are useless if, on average, blocks aren't read from them more than they are written to them. So treating them as analogous is highly misleading.

        The deal with throughput is that you can only win by storing packets if there is going to be room to send them without delay. If you buffer every packet that's sent, it does get delivered, but by the time it gets to its destination, it's too late. You can adjust the TCP algorithms to behave somewhat less badly in this situation, but what you can't do is get genuine flow control with big buffers, because the endpoints have no way to determine the throughput of the network.

        The only way the endpoints can determine the throughput of the network is if packets get dropped when there's congestion. When packets don't get dropped, what you see is that whenever there is more traffic to send over the link than the link can hold, it just winds up in a buffer. Latency rises. Eventually all the senders give up. Then the buffers start to drain, and packets get delivered. Then the acks start coming back. Now the endpoints think they are on a high latency link, so they crank back up again and fill the buffers again.

        So what you see is a network that works great as long as the total load presented to the network is less than the aggregate capacity of the network. As soon as the demand for bandwidth exceeds the supply, every single stream starts to stall. If you've stayed at a hotel recently, you've seen this: a dozen people try to watch video streams over a fairly wimpy connection, and then you can't do _anything_ over the connection, because the buffer fills up.

        If you didn't have that giant buffer, all the endpoints would be able to tell that the link was congested, and would slow down. If the total available bandwidth wasn't enough, the video streams would basically fail, but you could still get mail and surf the web. But with bufferbloat, not only can't you watch video streams, you also can't surf the web or get email or ssh to your server.

        You can see this by pinging a server somewhere out on the internet. When the link isn't congested, you'll see reasonable round trip times, typically 100ms. Then when it gets congested, you'll see packets dropped, and you'll see the RTT rise to as much as a minute. Then as all the senders notice that their packets aren't being delivered, they back off and suddenly the RTT starts to drop again, and you start to hope the network's been fixed. But it's fool's gold: as soon as the senders notice, they bomb the buffer again, and the RTT goes back up. Rinse and repeat until you give up.

        You probably don't see this very often on your home link, because you probably aren't saturating it. But it happens a lot at Wifi hotspots in particular, and also sometimes on 3G networks. It's quite disheartening, particularly when you're paying for the connection. You also see it on big ISPs like Comcast when you try to reach content providers that aren't willing to pay the ransom to Comcast to get on their uncongested link.

        • Re:Cringely again... (Score:5, Interesting)

          by m.dillon (147925) on Saturday December 03, 2011 @12:41AM (#38247416) Homepage

          Well, you definitely CAN tell when one or more buffers along the path begins to fill up, because latency increases. Packet loss is not necessary and, in fact, packet loss just makes the problem worse since many TCP connections implement SACK now and can keep the bandwidth saturated even in the face of packet loss.

          The ideal behavior is probably not to start dropping packets immediately... eventually, sure, but definitely not immediately. Ideally what you want to do is to attempt to shift the problem closer to the edges of the network where it is easier to fairly apportion bandwidth between customers.

          Send-side bandwidth limiting is very easy to implement since TCP already has a facility to collect latency information in the returned acks. I wrote a little beastie to do that in FreeBSD many years ago, and I turn it on in DragonFly releases by default.

          The purpose of the feature is not to completely remove packet buffering from the network, because doing so would put the sending server at a severe disadvantage verses other servers that do not implement similar algorithms (which is most of them).

          The purpose is to unload the buffers enough such that the algorithms in the edge routers aren't overloaded by the data and can do a better job apportioning bandwidth between streams.

          Our little network runs this coupled with fair queueing in both directions... that is, we not only control the outgoing bandwidth, we also pipe all the incoming bandwidth through a well connected colo and control that too, before it runs over the terminal broadband links. This allows us to run FAIRQ in both direction in addition to reserving bandwidth for TCP acks and breaking down other services. FAIRQ always works much better when links are only modestly overloaded and not completely overloaded. Frankly we don't have much of a choice, we HAVE to do this because our last-leg broadband links are 100% saturated in both directions 24x7. Anything short of that and even a single video stream screws up the latency for other connections beyond hope.

          This sort of solution works great near the edges.

          For the center of the network, frankly, I think about the best that can be done is modest buffering and RED and then trying to reduce the load on the buffers in the center with algorithms run on the edges (that can sense end-to-end latency). The modest buffering is needed for the edge algorithms to be able to operate without bits of the network having to resort to dropping packets. In otherwords, you want the steady state load for the network to not have to drop packets. Dropping packets should be reserved for the case where the load changes too quickly for the nominal algorithms to react. That's my opinion anyhow.

          -Matt

          • by mellon (7048)

            What you are missing in this analysis is that retransmissions are no more expensive whether they're done by the router or the end node, but when they are done by the end node, the end node has more information. Storing something in a buffer to send later when there's room in the pipe is exactly equivalent to retransmitting, except that there is no way for the router to inform the end node that this has happened.

      • by tepples (727027) <tepples@gmaiBLUEl.com minus berry> on Friday December 02, 2011 @11:51PM (#38247206) Homepage Journal
        A replacement for PATA or PCI has to interoperate only with other components in the same chassis, or possibly on the same desk in the case of eSATA and Thunderbolt. A replacement for TCP would have to interoperate with every other computer in the world. Imagine what a flag day [catb.org] that would be.
      • Each hop has its own buffer. Endpoints can fix their own buffers, but they can't do anything about buffering in the next hop. If something changes in the network to reduce the available bandwidth, the ideal behaviour is for packets to start getting dropped right away so that the originator gets notified of the drop and can slow itself down to compensate.

        If some device in the core network just buffers up seconds worth of packets instead of droping them it destroys the ability of the sender to adapt to the

      • Re:Cringely again... (Score:5, Interesting)

        by WaffleMonster (969671) on Saturday December 03, 2011 @03:51AM (#38248218)

        So wouldn't the right way to go be to update TCP for the times? i mean we didn't slow computers down so we could keep PATA or PCI, we came up with new tech like SATA and PCIe to take advantage of the faster throughput. Shouldn't we do the same here as well?

        We have SCTP which was intended to replace TCP except nobody seems to care.

        At the end of the day the concept of TCP is not rocket science - there is a limit and diminishing returns to what more can be done twoard making TCP a perfect reflection of the concept of TCP.

        Congestion management and ack/windowing have certainly evolved into high arts..but fundementally all TCP does is implement a loss free ordered data stream on top of an unordered lossy packet switched network.

        This means your core limitation is embedded in the definition of TCP itself...the problem of head-of-line blocking. By using TCP you are by definition limiting yourself to the constraints of TCP.

        Realtime voice/video and multi-player games use their own protocols because they are not willing to accept the constraints of TCP. It is not the implementation of TCP that is holding them back. It is the *concept* of TCP.

        In my opinion we need more IP protocols to better handle varied use cases more than we need a new TCP.

        • by jgrahn (181062)

          We have SCTP which was intended to replace TCP except nobody seems to care.

          Some telecom standards are built on SCTP (we use it at work). Not sure if it's all that great though -- a lot of its problems are probably hidden by the fact that few care about it, and that it's used in isolated, high-quality networks.

          At the end of the day the concept of TCP is not rocket science - there is a limit and diminishing returns to what more can be done twoard making TCP a perfect reflection of the concept of TCP. [---] In my opinion we need more IP protocols to better handle varied use cases more than we need a new TCP.

          And we need application-level protocols on top of TCP which suck less, and implementations which suck less. You need some elemental understanding of TCP to use it efficiently, and from what I've seen, most programmers don't have it.

          PS. As I recall it, last time Getty was i

          • by pehrs (690959)

            We have SCTP which was intended to replace TCP except nobody seems to care.

            Some telecom standards are built on SCTP (we use it at work).
            Not sure if it's all that great though -- a lot of its problems are probably hidden by the fact that
            few care about it, and that it's used in isolated, high-quality networks.

            We have used SCTP in production for some time. There are some serious and well known problems with using SCTP, but as far as I know the protocol itself is solid, far more so than TCP. The three major problems you run into with SCTP are:

            1. The LKSCTP implementation in the linux kernel has a few nasty bugs related to multihoming, that have been patched but the patches are slow to be pushed to all distributions.
            2. There are a lot of D-Link, NetGear and similar crap nat boxes on the internet that happily destro

            • Could it be used as a mid point to mid point protocol such that the big backbones convert to sctp, use it to transmit between each other, attempt to transmit to ISPs - convert if the attempt fails. Same could be done by ISPs as the transition over from TCP to SCTP, attempt to transmit with conversion as a failure mode.

              The premise is to start the transition with a backup to convert.

              A possible efficiency would be to whitelist successful attempt targets, ISPs that accept SCTP for instance. These ISPs could als

        • by sgt scrub (869860)

          I thought IBM designed SCTP to replace UDP. Regardless, I think your right. Using SCTP would benefit the web.

      • Re:Cringely again... (Score:5, Interesting)

        by evilviper (135110) on Saturday December 03, 2011 @06:13AM (#38248630) Journal

        Even Cringley points out at the first of his article that originally TCP was written for a VASTLY different and weaker network than we have now, so instead of trying to make the networks go back to a mid 1980s design, wouldn't it be smarter just to update TCP to take advantage of new tech advances?

        There's nothing about a "weaker" network that necessitates a protocol redesign. TCP has had problems with congestion handling from day one, that have necessitated a million and one hacks and workarounds, because it stupidly conflates packet loss with congestion... Some links will have packet loss without any congestion, and others (like these with huge buffers) will have congestion without (immediate) packet loss. It was a bad design decision.

        What's worse is that IP was designed correctly to begin with. The original design has ICMP control messages (eg. source-quench) to signal congestion, much like many other networking protocols. The real problem was that the specifics were vague, and there was no exact standard on how much to slow down, how it affects higher level protocols, etc., so it became a prisoner's dilemma, and highly unfair, and was deprecated.

        Of course, this problem could occur with TCP's congestion control just as easily if any particular implementations reduced the rate of exponential backoff, so there's nothing fundamentally wrong with the original congestion control design, just the lack of consistent implementation.

        Controlling congestion by dropping packets is like controlling freeway traffic by randomly pushing cars off the road with a bulldozer.

      • by Anonymous Coward

        wouldn't it be smarter just to update TCP to take advantage of new tech advances?

        But then you'll have a TCPv6 situation where no-one can communicate between the 2 different internets.

    • by Idbar (1034346)
      I'll give you a few more names: Sally Floyd, Van Jacobson, Leonard Kleinrock and, of course, Raj Jain have been writing about this since 1983.
    • Re:Cringely again... (Score:4, Informative)

      by Idbar (1034346) on Saturday December 03, 2011 @12:34AM (#38247376)
      Basic, and worth reading is Raj Jain's 1992 paper [wustl.edu].
  • by JoeMerchant (803320) on Friday December 02, 2011 @10:37PM (#38246910)

    To configure your active queue management, the first thing I need to know is: do you have a push system, or a pull system?

    Neither, sir, we have a suck system.

  • by Just Brew It! (636086) on Saturday December 03, 2011 @01:06AM (#38247498)
    As soon as I start trying to shove (or suck) more bits through the pipe than it can handle, round trip latency to "nearby" points of the Internet increases from ~25 ms to ~1 second. When I need to transfer a lot of data, I use rsync or wget if at all possible, and throttle the transfer to just below the rate the connection can handle; this results in ping times staying sane while only slowing down the transfer slightly. We shouldn't need to resort to doing stuff like this to make the network function properly!
    • I've had similar problems a few years back, but either TCP implementations are better or the buffer sizes in DSLAMs are better tuned these days because it doesn't seem to happen. Yes, latency goes up if I saturate the connection with a big download, but not so much that all other connections grind to a halt.

      Not like a few years back when a big download would cause latency to go go up to ~ 2 seconds and you could kiss goodbye to doing anything else with the connection at the same time, at least at tolerable

    • Forgot to mention, I also run the Folding@home distributed computing client. Uploads of completed work units were causing horrible lag spikes until I started routing them through a throttling proxy (implemented via apache + a Python script I cobbled together).
    • You are simply implementing ECN by hand.
  • by OrangeTide (124937) on Saturday December 03, 2011 @02:01AM (#38247796) Homepage Journal

    This analogy is like a bathtub, full of spiders, and on fire. It sounds dangerous, but it's self limiting.

  • by Twinbee (767046) on Saturday December 03, 2011 @02:11AM (#38247844) Homepage
    I thought this animation by Richard Scheffenegger was a good way to show what's happening: http://www.skytopia.com/project/articles/lag/nam00000.avi [skytopia.com] Here's a description of the video:

    The bad Bufferbloat setup is on the left (yellow dots), and the 'good' setup (i.e. how things used to be configured about 10-20 years ago when RAM was more expensive!) is on the right (cyan/blue dots).

    Both sides start off okay, but notice how the left side 'queues' (tall yellow dot columns) keep on growing over time, while the right side blue columns stop short because of the small buffer size. As they stop short, some data 'packets' must be dropped, and this gets reported back to the upload site that it's shoving data to the user too fast. As a result, the upload site temporarily slows the sending of data, and thus the system self-corrects.

    Meanwhile, on the left side, these packets of data never get dropped, so the giant bloated yellow buffers get filled more and more, but the computer at the upload site doesn't realise the carnage of these giant queues further down the line, and instead thinks "All is okay, let's keep sending data fast!".

    Finally, when a smaller piece of data needs to be sent to the user (see 2:30+ signified by red dots on the left and dark blue dots on the right), the left side shows the red dots (which could be say, a small email) wading through giant queues to reach their destination, really slowly. Furthermore these tiny bits of data often need special 'emergency' treatment as they hold up other larger data associated with it. On the good right side, the dark blue dots have no such giant queues.

  • by WaffleMonster (969671) on Saturday December 03, 2011 @02:49AM (#38248008)

    If you look at buffers allocated to fast multi-gigabit interfaces at the core of the network they are simply not large enough compared to forwarding rates involved to be able to induce the kinds of delays needed to cause Internet wide problems.

    You can argue they may not be ideal for real time voice, game or video communication when these links are oversubscribed but no doomsday is possible.

    Today buffer bloat effects are mostly observed at the edge even though they need not always be.

      Failure of a congestion control algorithm to control link saturation does not translate into congestive collapse of the larger network. It just results in *your* network connection turning to shit. When netalyzer runs it intentionally saturates your link at that time. In the real world only a few portions of the edge are ever saturated to the extent congestion control failure becomes an issue leading to more packets through core routers. The number of edge machines in this category would need to be significant to cause a rerun of previous issues.

    That condition can not be met due to self feedbacks. If everyone maxed their pipes at once the core would saturate self-limiting edge saturation due to gross over-provisioning of available edge bandwidth in relation to core bandwidth which would ensure congestion control algorithms function properly.

    I'm not arguing there is not a problem or more can't be done. I'm just arguing the doomsday congestive collapse scenario is bullshit.

    • by jg (16880)

      There are significant networks that do not look like the consumer edge Internet, one of which reportedly collapsed in a nasty way (not necessarily in the same way as 1986). Don't presume that the network that may collapse is the global internet (though time based self synchronizing phenomena are a worry there). One of the functions of AQM algorithms is to ensure that TCP flows don't synchronize their behavior. And those AQM algorithms are MIA on many networks today.

      Those of us who lived through the 1986 c

  • The bloated or big buffers causing more latencies than necessary only if it is designed with a single queue for all flows. If each flow gets a queue in the buffer and all queues are read and send out in round robin, the ping packet would not have to wait till the earlier started big file transfer which has completely filled the buffer would be through. The ping packet would practically overtake the large amount of queued bytes of the big file transfer instead of going behind it in a single queue.
  • by Anonymous Coward

    This government has literally spent billions through DHS to fehn internet safety. I may be simple, but that same overreaching government could give $500m to 4 private non-profits to actively deal with this and other "infrastructure" issues like IPv6, more rural broadband, urban wireless, and other issues.

    It has been done in the past through Red Cross for disaster relief, in floods, earthquakes, epidemics and the like. Society impact issues.

    NGO's have far more efficient resource allocation than the governm

    • by mtaht (603670)

      What Jim and the bufferbloat.net's group of volunteers have accomplished in a year - on nearly no money - boggles my mind.

      Today's commentary on slashdot is a hundred times more clueful than it was last year - and a few days back Byte Queue Limits went into linux's net-next tree, which fixes much of the bloat problems that exist at the ethernet driver layer.

      What has been discussed as 'Time in Queue' limits in the higher level schedulers is still awaiting a clean way to avoid layer violations. I've been too d

  • IMHO. If ISP's would build out their networks instead of relying on buffers there would not be an argument here. My attention would be on fixing dumb wireless devices and drivers that ignore every attempt of making them play nice.

  • This is a classic problem of economics. Publicly owned resources that are not owned by any one individual or company are very difficult for market factors to work on. A good example is fishing. The fish in a bay are not owned by any particular person, so their welfare is not in the economic interest of any particular person. It may be in a commercial fishermans long term interest to conserve the fish population and not over fish, but he's not the only fisherman. If he cuts back on his catch, other fisherma
  • Disclaimer I'm the author. I covered this in my June 2011 column: http://www.linuxpromagazine.com/Issues/2011/127/Security-Lessons-Bufferbloat/%28kategorie%29/0 [linuxpromagazine.com] direct link to the PDF http://www.linux-magazine.com/w3/issue/127/058-059_kurt.pdf [linux-magazine.com]. In a nutshell: my link latency at home is usually ~50ms to seifried.org, but with one single outbound file transfer to saturate my uplink ping times go to over 1000ms (1 second) reliably (which completely breaks VOIP/games/etc.).

  • by Edgester (105351) on Saturday December 03, 2011 @12:03PM (#38250154) Homepage Journal

    What can I do with my own laptop and wifi router to make my own situation better?

    • by Chirs (87576) on Saturday December 03, 2011 @04:58PM (#38252444)

      As an end-user there are only a few things you can do:

      1) Reduce the outgoing tcp queue size.
      2) Reduce the tx ring buffer size in the network device driver
      3) Set your router's upstream quality-of-service settings to throttle your upstream data transmission rate to just less than your upstream bandwidth.

      Alternately, if you only have one heavy user of upstream bandwidth you could do something like what is described at "http://wanners.net:8000/blog/2011/05/zapping-upload-bufferbloat-with-one-command/". Basically throttling the upstream bandwidth directly on the machine in question rather than on the router.

  • by Anonymous Coward

    The proposed solution of actve queue management is exactly the sort of discrimination the net neutrality folks want to forbid, no?

COBOL is for morons. -- E.W. Dijkstra

Working...