Bufferbloat: Dark Buffers In the Internet 124
Expanding on earlier work from Jim Gettys of Bell Labs with a new article in the ACM Queue, CowboyRobot writes that Gettys "makes the case that the Internet is in danger of collapse due to 'bufferbloat,' 'the existence of excessively large and frequently full buffers inside the network.' Part of the blame is due to overbuffering; in an effort to protect ourselves we make things worse. But the problem runs deeper than that. Gettys' solution is AQM (active queue management) which is not deployed as widely as it should be. 'We are flying on an Internet airplane in which we are constantly swapping the wings, the engines, and the fuselage, with most of the cockpit instruments removed but only a few new instruments reinstalled. It crashed before; will it crash again?'"
Is this a problem? (Score:2)
the existence of excessively large and frequently full buffers
Seems better than the existence of excessively large and seldom if ever full buffers.
Re:Is this a problem? (Score:4, Insightful)
Re:Is this a problem? (Score:5, Informative)
That is actually the exact problem. You do not want buffers larger than the flight time of your circuit. You absolutely want the buffers to fill and drop packets otherwise.
Re:Is this a problem? (Score:5, Funny)
That is actually the exact problem. You do not want buffers larger than the flight time of your circuit. You absolutely want the buffers to fill and drop packets otherwise.
You talkin' smack, fool? I will end you! I bloat like a buffer, sting like a TCP!
Re: (Score:2)
Yes, and since the BDP is not knowable usually (neither the bandwidth or delay are predictable these days in many/most circumstances), we have to be smart (need AQM).
Re:Is this a problem? (Score:5, Informative)
Seems so, but isn't. For TCP traffic, a shallow buffer that drops traffic will result in more goodput than a deep buffer. Which is the point.
Re:Is this a problem? (Score:5, Informative)
Yes and no...
If you don't (or only rarely) fill your buffer, a smaller buffer introduces less latency than a large one, while still allowing you to maximize throughput. If, however, you usually have your buffer full, you increase latency for literally no benefit, since you've already maximized throughput simply through resource demand.
The former will occur when your average load falls below your actual bandwidth, and allows you to get the most out of your link. The latter occurs when you consistently exceed your bandwidth, in which situation you may as well not even have a buffer, because it only increases latency without increasing throughput. That describes TFA's real point.
What he suggests amounts to actively choosing between those two conditions - If your average demand falls below your link speed, a larger buffer will help smooth the load over time. If, however, your average demand exceeds your link speed, throw away the buffer because it doesn't help.
But as per the GP's point - If you have an always-full buffer, you literally gain nothing but latency.
Re:Is this a problem? (Score:5, Informative)
The problem with buffers is most all of the time they are configured by size in bits. They need to be sized based on bit flight time of the circuit, which is in delay ms times throughput in bits. The disconnect between those values is a problem in *either* direction, especially past the retransmit threshold on the above side.
Buffers should be dynamicly sized based on flight time of data on the specifc link, and ideally kept updated. WRED is also highly suggested.
What really exacerbates the issue is devices with buffers that must be the same size for all links on X (be it card, slot, or chassis).
Re: (Score:3)
No, buffers are not for "rare peaks". They are fundamentally required due to the physics of data transmission from one device to another, especially when the link speeds are different from one hop to the next.
Re: (Score:1)
True, but the buffer requred for a single interface should only be large enough to cover the bit-rate difference between the slowest outbound intereface and the highest inbound interface.
more or less.
Re: (Score:2)
Now if we could just back off the military spending, even just a little, we could upgrade infrastructure in the US. But military spending feeds campaigns more than citizens do...... and here we are.
Huh? You are aware that the Internet is owned by companies like Level3, which doesn't spend any money on military campaigns, aren't you?
Re: (Score:2)
They don't pay taxes?
Re: (Score:2)
They don't pay taxes?
Do you have some evidence that if we weren't at war, that taxes would be lower? Or even if taxes were lower, that the extra money would be spent on infrastructure rather than stock dividends?
Re: (Score:2)
I thought that was the premise, not the argument.
Re:Is this a problem? (Score:4, Interesting)
Except that Cogent is the trouble instigator in every single instance. They are like the little trouble maker who runs around poking, kicking, pinching, insulting the larger kid at the playground, yet cries loudly and acts very hurt when he gets slapped in return.
Re: (Score:1)
Cutting military spending entirely would still leave us with a deficit in eah year's budget(Oh wait, a budget hasn't been passed in the past 4 years?) that is only going to grow exponentially from social programs, and that was before Obammy went on his spending spree and guaranteed a metric asston of future spending.
Re: (Score:2)
Re:Is this a problem? (Score:4, Interesting)
What you can do is have huge buffers, but just drop packets that are older than say 50 milliseconds since the time they entered the device (if the link/hop is supposed to be fast and low latency).
If the link is slow and/or high latency, you may wish to use higher values - 100 milliseconds. But not too high. I'm no networking expert but I don't really see the purpose of adding hundreds of milliseconds to a hop just to save a few packets that are likely to be dropped anyway, or should be dropped as an indirect signal that whoever is sending those packets should slow down.
Re:Is this a problem? (Score:4, Insightful)
What he suggests amounts to actively choosing between those two conditions - If your average demand falls below your link speed, a larger buffer will help smooth the load over time.
That's a pretty simplified way of putting it, but basically correct. Major equipment vendors have been slow to adopt more advanced queuing strategies (Stochastic Fair Queuing integrated with some of the more advanced flavors of early discard.) Fortunately we're budgeted for and piloting a shaper for purchase soon, and this time around have a chance to get something both well supported and cutting edge.
Personally I pine for ATM's ABR CoS with it's fast end-to-end congestion notification, but as history has shown us, the inevitable fate of the tech world is for the inferior to be gradually, painfully, and kludgingly adapted to become the same thing as the technologies it displaced through lowballing. In this case, that inferior thing being IP/ethernet.
Re: (Score:1, Offtopic)
...the inevitable fate of the tech world is for the inferior to be gradually, painfully, and kludgingly adapted to become the same thing as the technologies it displaced through lowballing.
In this case Gettys is acting as an apologist for the municipal and/or corporate resistance to augmenting American infrastructure to 21st century standards. Whenever you have an elite few pinching pennies at the expense of the many, then you know that the ZOG machine is behind it. Their propaganda will explain everything away using bureaucratic technobabble while the final solution* to the problem is painfully obvious to anybody.
* the final solution being to railroad more fiber directly into the proces
Re: (Score:2)
Re: (Score:3)
You can have an overall congested network. I've seen this on occasion.
But it is very easy (and even more common) for you (or people in your house) to do it to you, than to have the overall ISP network congested. This is something a simple file copy can/does do to you, in practice.
Some ISP's run AQM properly (e.g. RED) in the cores of their networks; some do not. On the ones that do not, you'll see problems at peak hours. Similarly on corporate networks.
Re: (Score:2)
Re: (Score:2)
Heh. The worst problems (as far as I've seen) we have are in the edge.
In our hosts and home routers (most of which are Linux boxes).
The problem occurs anytime you are next to a bottleneck, and the "municipal and/or corporate resistance" have now built out the core of the internet to the point that the problem is most severe at the very edge, including your laptop/handheld device.
*EVERYONE* has made the same set of mistakes. I have done so too at times....
Doing it wrong, again (Score:5, Interesting)
That's a pretty simplified way of putting it, but basically correct. Major equipment vendors have been slow to adopt more advanced queuing strategies (Stochastic Fair Queuing integrated with some of the more advanced flavors of early discard.)
Right. The problem is not big buffers, per se. It's big dumb FIFO queues. There's nothing wrong with one big flow, like a file transfer, having a long latency, provided that other flows with less data in flight aren't stuck behind it. That's what "fair queuing" is all about. Each flow has its own queue, and the queues are serviced in a round-robin fashion. (With stochastic fair queuing, some hashing is done to eliminate some of the bookkeeping on flows, but the effect is roughly the same.)
I figured this out in the early 1980s (see RFC 970 [ietf.org]) and by the late 1990s, it was an established technology. We shouldn't be having this problem at this late date.
I wonder how much of the trouble comes from devices that are doing TCP-level processing in the middle of the network. Stateful firewalls and ISP ad-insertion engines [isp-planet.com] can introduce substantial latency.
If you want to test for bad behavior, try running two flows, one that never has more than one packet outstanding, and one that just does a big file-transfer like operation like a download. If the latency of the low-traffic flow goes up to the same as that of the bulk flow, there's a big dumb buffer in the middle. If the packet loss rate of the low-traffic flow goes up, there's a small dumb buffer in the middle.
Re: (Score:2)
big dumb buffer
Nice. Stupid ToS settings, devices that always send the maximum data until they receive 100 ACK window size adjustments telling it to SLFD, and big dumb buffers. I like it!
Re: (Score:2)
I doubt that the processing itself is the cause of more than a few milliseconds of latency, but the machines doing it may have been configured with large buffers not because the processing needed them but because those configuring them thought erroneously that bigger buffers are always better
Re: (Score:3)
To solve this, I think we need both AQM fully at the edge of the network, and some sort of "fair" queueing at the edge. The headache is that the classic AQM algorithms won't work in the edge case (and are flawed). "Fairness" is in the eyes of the beholder and a complex question. You may consider it "fair" your kids get half the bandwidth you do, for example. But having a situation where talking to a local CDN 10ms away gets tons more bandwidth than something across the globe may not be "fair" in your vie
Re: (Score:1)
Re:Is this a problem? (Score:4, Informative)
You mean, like figure-2 or even better, figure-5, in TFA? Where the (most common) 2^n buffer sizes stand out so obviously in the data that you'd need to try not to notice the trend?
Of course, this situation doesn't actually require much "real" data to prove. If each 1500 byte packet takes 10ms to transmit, and you have a full 256KB buffer - Which will unavoidably happen any time you try to sustain a transmit faster than your link can handle - You will have 1.7 seconds of latency in a FIFO queue.
tldr
Don't worry, we could tell.
Re: (Score:3)
Umm ... your 1500 byte packet had best not take more than about 10 us to transmit. 10ms would be quite ridiculous.
Re: (Score:2)
1500 bytes per 10ms comes out to 1.2Mbit, very close to my actual (admittedly sucky) internet connection. 1500 bytes per 10us would mean you have a 1.2GIGAbit connection, faster than most LANs.
Re: (Score:2)
We're talking about 40+Gbit/sec internet backbones in this article, not end user connections.
Re: (Score:3)
The entire first half of TFA talks about his 1Mbit WiFi connection. I care about my paltry 1.2Mbit intenet connection.
Now in fairness, the same problem does indeed apply at the backbone level... But I didn't call you "ridiculous".
Re: (Score:2)
Sorry, I just read the 'bufferbloat' article, which seemed to be all about the transit, and not the endpoints.
Re: (Score:2)
Huh?
It talks primarily about what I observed on my *home* connection...
The Netalyzr data is all about edge connections.
There are problems in the core, but the worst problems I've seen are at the edge, and the edge is where most of the congestion is these days.
Re:Is this a problem? (Score:5, Informative)
Seems so, but isn't. For TCP traffic, a shallow buffer that drops traffic will result in more goodput than a deep buffer. Which is the point.
Exactly.
Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need.
The deep buffer just make it worse for everyone.
Oh, and And just as a Car Analogy is inappropriate to describe TCP traffic the Airplane Analogy is worse.
Re: (Score:2)
Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need. The deep buffer just make it worse for everyone.
There's something to be said for shaping at intermediate hops, even with ECN (and especially when ECN isn't implemented) but it has to be done in a manner that it doesn't add latency out of proportion to the unladen RTT.
Re:Is this a problem? (Score:5, Interesting)
What we need is a ferry analogy.
Packet transmission is like a ferry, crossing a river at fixed intervals. But ferry sets off when it is full rather than at set times.
People wait at the shore and generally don't have to wait too long as the ferry is pretty fast and only needs a few people to fill up. For most people, walking onto the ferry involves very little waiting before the ferry actually departs and crosses the river.
Buffer bloat is when big buffers act like ferrys with huge capacity. People enter a huge 2000 passenger capacity boat, and are let on by their hundreds with seemingly no delay. But the ferry will not depart until it is reasonably full. So the people who got on first may have to wait for hours before the ferry actually departs and crosses the river.
It is clear that bigger ferries are no substitute for more ferries....or smaller rivers. Or possibly a bridge. In any case, you can get away without introducing cars or airplanes, so my job is done here.
Re: (Score:1)
"May the Maths Be with you!"
May the understanding of reason be with you:
http://bit.ly/dYaWUc [bit.ly]
Re: (Score:2)
It is clear that bigger ferries are no substitute for more ferries....or smaller rivers. Or possibly a bridge.
Or intergalactic starships, and teleporters?
What were we talking about, again?
Re:Is this a problem? (Score:5, Informative)
That analogy doesn't quite do the trick. TCP windowing is a bit more sophisticated than that. You can think of it maybe as a commander sending couriers out to support a mobile squad through hostile territory. If too many of them never make it to the squad, or back, he sends them less frequently so they can sneak through more discretely. If the troops make it through then he sends them faster because the more ammo he can get through the better. But he also has to decide how many men to put on courier duty. If the couriers take too long the squad has obviously moved further away from the base, and if he waited for the next one to return, he wouldn't be sending enough ammo. If the couriers return quickly, he can make do with less couriers.
Big buffers are like a flimsy rope bridge in the courier's path that takes a long time to cross. Couriers have to wait on one side because only one can cross at once, but the large groups waiting at the side of the cliff is more likely to get attacked. Until they do get attacked, however, the commander starts to think the squad has moved very far away, so he puts more couriers on duty. Since he thinks the squad is far away, he is not expecting them to return for a longer amount of time, it takes him longer to realize that they are starting to go missing entirely.
One of the best solutions to this problem turns out to be for some of the couriers to randomly go AWOL, and for more of them to go AWOL the bigger the crowd at the rope bridge gets. This basic concept is called Random Early Discard, and there have been a lot of ways invented for deciding who goes AWOL and why. If some of the couriers go AWOL, the commander thinks they are being attacked, so he slows down and also takes some troops off courier duty.
All group transport mechanism run to a schedule (Score:2)
So none are an appropriate analogy.
HTH
Re: (Score:2)
They all have a schedule. Even it it's no more than "today".
Re: (Score:1)
Re: (Score:3)
In any case, you can get away without introducing cars
The only ferries I've ever been on were for cars, so you kinda did. Maybe you should've used a paraglider analogy, I've never seen a car use a paraglider.
Re: (Score:2)
But what if I want to bring my goat?
Re: (Score:3)
Early Congestion notification along with ONLY a minimal amount of client side buffering is really all you need.
Unfortunately, early notification doesn't work with a ton of wireless devices. Their drivers have minimal abilities to be controlled and they always send data at the speed of their negotiation. .eg if they connect at 11g they always send data at that speed and always send acks with window size adjustments to speed traffic up to that speed until they receive multiple window size adjustments telling
Re: (Score:2)
ECN helps to increase goodput, and AQM can help to keep high thoughput. The main concern of some (at least my research topic) is how to implement AQM to spread traffic spikes such that the link utilization increases while buffer occupancy reduces.
Re: (Score:3)
And the right balance between buffer size, drop percentage, and throughput should be measurable. But I bet those lazy bastards at cisco have never thought to measure performance, which is why no one uses their equipment.
Cringely again... (Score:5, Informative)
Cingely has been writing about this all year. He cites Jim Gettys too. See: http://www.cringely.com/tag/bufferbloat/ [cringely.com]
Comment removed (Score:5, Interesting)
Re:Cringely again... (Score:5, Insightful)
Buffer and cache are not the same thing. Packets are written to a buffer once and read from it once. Caches are useless if, on average, blocks aren't read from them more than they are written to them. So treating them as analogous is highly misleading.
The deal with throughput is that you can only win by storing packets if there is going to be room to send them without delay. If you buffer every packet that's sent, it does get delivered, but by the time it gets to its destination, it's too late. You can adjust the TCP algorithms to behave somewhat less badly in this situation, but what you can't do is get genuine flow control with big buffers, because the endpoints have no way to determine the throughput of the network.
The only way the endpoints can determine the throughput of the network is if packets get dropped when there's congestion. When packets don't get dropped, what you see is that whenever there is more traffic to send over the link than the link can hold, it just winds up in a buffer. Latency rises. Eventually all the senders give up. Then the buffers start to drain, and packets get delivered. Then the acks start coming back. Now the endpoints think they are on a high latency link, so they crank back up again and fill the buffers again.
So what you see is a network that works great as long as the total load presented to the network is less than the aggregate capacity of the network. As soon as the demand for bandwidth exceeds the supply, every single stream starts to stall. If you've stayed at a hotel recently, you've seen this: a dozen people try to watch video streams over a fairly wimpy connection, and then you can't do _anything_ over the connection, because the buffer fills up.
If you didn't have that giant buffer, all the endpoints would be able to tell that the link was congested, and would slow down. If the total available bandwidth wasn't enough, the video streams would basically fail, but you could still get mail and surf the web. But with bufferbloat, not only can't you watch video streams, you also can't surf the web or get email or ssh to your server.
You can see this by pinging a server somewhere out on the internet. When the link isn't congested, you'll see reasonable round trip times, typically 100ms. Then when it gets congested, you'll see packets dropped, and you'll see the RTT rise to as much as a minute. Then as all the senders notice that their packets aren't being delivered, they back off and suddenly the RTT starts to drop again, and you start to hope the network's been fixed. But it's fool's gold: as soon as the senders notice, they bomb the buffer again, and the RTT goes back up. Rinse and repeat until you give up.
You probably don't see this very often on your home link, because you probably aren't saturating it. But it happens a lot at Wifi hotspots in particular, and also sometimes on 3G networks. It's quite disheartening, particularly when you're paying for the connection. You also see it on big ISPs like Comcast when you try to reach content providers that aren't willing to pay the ransom to Comcast to get on their uncongested link.
Re:Cringely again... (Score:5, Interesting)
Well, you definitely CAN tell when one or more buffers along the path begins to fill up, because latency increases. Packet loss is not necessary and, in fact, packet loss just makes the problem worse since many TCP connections implement SACK now and can keep the bandwidth saturated even in the face of packet loss.
The ideal behavior is probably not to start dropping packets immediately... eventually, sure, but definitely not immediately. Ideally what you want to do is to attempt to shift the problem closer to the edges of the network where it is easier to fairly apportion bandwidth between customers.
Send-side bandwidth limiting is very easy to implement since TCP already has a facility to collect latency information in the returned acks. I wrote a little beastie to do that in FreeBSD many years ago, and I turn it on in DragonFly releases by default.
The purpose of the feature is not to completely remove packet buffering from the network, because doing so would put the sending server at a severe disadvantage verses other servers that do not implement similar algorithms (which is most of them).
The purpose is to unload the buffers enough such that the algorithms in the edge routers aren't overloaded by the data and can do a better job apportioning bandwidth between streams.
Our little network runs this coupled with fair queueing in both directions... that is, we not only control the outgoing bandwidth, we also pipe all the incoming bandwidth through a well connected colo and control that too, before it runs over the terminal broadband links. This allows us to run FAIRQ in both direction in addition to reserving bandwidth for TCP acks and breaking down other services. FAIRQ always works much better when links are only modestly overloaded and not completely overloaded. Frankly we don't have much of a choice, we HAVE to do this because our last-leg broadband links are 100% saturated in both directions 24x7. Anything short of that and even a single video stream screws up the latency for other connections beyond hope.
This sort of solution works great near the edges.
For the center of the network, frankly, I think about the best that can be done is modest buffering and RED and then trying to reduce the load on the buffers in the center with algorithms run on the edges (that can sense end-to-end latency). The modest buffering is needed for the edge algorithms to be able to operate without bits of the network having to resort to dropping packets. In otherwords, you want the steady state load for the network to not have to drop packets. Dropping packets should be reserved for the case where the load changes too quickly for the nominal algorithms to react. That's my opinion anyhow.
-Matt
Re: (Score:2)
What you are missing in this analysis is that retransmissions are no more expensive whether they're done by the router or the end node, but when they are done by the end node, the end node has more information. Storing something in a buffer to send later when there's room in the pipe is exactly equivalent to retransmitting, except that there is no way for the router to inform the end node that this has happened.
A lot more to interoperate with (Score:5, Insightful)
endpoints can't fix individual hop buffering (Score:3)
Each hop has its own buffer. Endpoints can fix their own buffers, but they can't do anything about buffering in the next hop. If something changes in the network to reduce the available bandwidth, the ideal behaviour is for packets to start getting dropped right away so that the originator gets notified of the drop and can slow itself down to compensate.
If some device in the core network just buffers up seconds worth of packets instead of droping them it destroys the ability of the sender to adapt to the
Re:Cringely again... (Score:5, Interesting)
So wouldn't the right way to go be to update TCP for the times? i mean we didn't slow computers down so we could keep PATA or PCI, we came up with new tech like SATA and PCIe to take advantage of the faster throughput. Shouldn't we do the same here as well?
We have SCTP which was intended to replace TCP except nobody seems to care.
At the end of the day the concept of TCP is not rocket science - there is a limit and diminishing returns to what more can be done twoard making TCP a perfect reflection of the concept of TCP.
Congestion management and ack/windowing have certainly evolved into high arts..but fundementally all TCP does is implement a loss free ordered data stream on top of an unordered lossy packet switched network.
This means your core limitation is embedded in the definition of TCP itself...the problem of head-of-line blocking. By using TCP you are by definition limiting yourself to the constraints of TCP.
Realtime voice/video and multi-player games use their own protocols because they are not willing to accept the constraints of TCP. It is not the implementation of TCP that is holding them back. It is the *concept* of TCP.
In my opinion we need more IP protocols to better handle varied use cases more than we need a new TCP.
Re: (Score:3)
We have SCTP which was intended to replace TCP except nobody seems to care.
Some telecom standards are built on SCTP (we use it at work). Not sure if it's all that great though -- a lot of its problems are probably hidden by the fact that few care about it, and that it's used in isolated, high-quality networks.
At the end of the day the concept of TCP is not rocket science - there is a limit and diminishing returns to what more can be done twoard making TCP a perfect reflection of the concept of TCP. [---] In my opinion we need more IP protocols to better handle varied use cases more than we need a new TCP.
And we need application-level protocols on top of TCP which suck less, and implementations which suck less. You need some elemental understanding of TCP to use it efficiently, and from what I've seen, most programmers don't have it.
PS. As I recall it, last time Getty was i
Re: (Score:2)
We have SCTP which was intended to replace TCP except nobody seems to care.
Some telecom standards are built on SCTP (we use it at work).
Not sure if it's all that great though -- a lot of its problems are probably hidden by the fact that
few care about it, and that it's used in isolated, high-quality networks.
We have used SCTP in production for some time. There are some serious and well known problems with using SCTP, but as far as I know the protocol itself is solid, far more so than TCP. The three major problems you run into with SCTP are:
1. The LKSCTP implementation in the linux kernel has a few nasty bugs related to multihoming, that have been patched but the patches are slow to be pushed to all distributions.
2. There are a lot of D-Link, NetGear and similar crap nat boxes on the internet that happily destro
Re: (Score:2)
Could it be used as a mid point to mid point protocol such that the big backbones convert to sctp, use it to transmit between each other, attempt to transmit to ISPs - convert if the attempt fails. Same could be done by ISPs as the transition over from TCP to SCTP, attempt to transmit with conversion as a failure mode.
The premise is to start the transition with a backup to convert.
A possible efficiency would be to whitelist successful attempt targets, ISPs that accept SCTP for instance. These ISPs could als
Re: (Score:2)
I thought IBM designed SCTP to replace UDP. Regardless, I think your right. Using SCTP would benefit the web.
Re:Cringely again... (Score:5, Interesting)
There's nothing about a "weaker" network that necessitates a protocol redesign. TCP has had problems with congestion handling from day one, that have necessitated a million and one hacks and workarounds, because it stupidly conflates packet loss with congestion... Some links will have packet loss without any congestion, and others (like these with huge buffers) will have congestion without (immediate) packet loss. It was a bad design decision.
What's worse is that IP was designed correctly to begin with. The original design has ICMP control messages (eg. source-quench) to signal congestion, much like many other networking protocols. The real problem was that the specifics were vague, and there was no exact standard on how much to slow down, how it affects higher level protocols, etc., so it became a prisoner's dilemma, and highly unfair, and was deprecated.
Of course, this problem could occur with TCP's congestion control just as easily if any particular implementations reduced the rate of exponential backoff, so there's nothing fundamentally wrong with the original congestion control design, just the lack of consistent implementation.
Controlling congestion by dropping packets is like controlling freeway traffic by randomly pushing cars off the road with a bulldozer.
Re: (Score:1)
wouldn't it be smarter just to update TCP to take advantage of new tech advances?
But then you'll have a TCPv6 situation where no-one can communicate between the 2 different internets.
Re: (Score:2)
Re:Cringely again... (Score:4, Informative)
Push or Pull (Score:5, Funny)
To configure your active queue management, the first thing I need to know is: do you have a push system, or a pull system?
Neither, sir, we have a suck system.
Re: (Score:1, Funny)
Re:SPAM (Score:5)
Maybe posting a new article on an issue that was also an issue a year ago is not a "dupe", but an acceptable and possibly even normal thing for a news site to do?
Re:SPAM (Score:4, Insightful)
And just maybe some of us are interested in how research has progressed since the last article...
really? calling Jim Gettys a spammer? (Score:3)
Feel free to try it out yourself...I have and the problem is real.
I've definitely noticed it on my DSL (Score:5, Interesting)
Re: (Score:2)
I've had similar problems a few years back, but either TCP implementations are better or the buffer sizes in DSLAMs are better tuned these days because it doesn't seem to happen. Yes, latency goes up if I saturate the connection with a big download, but not so much that all other connections grind to a halt.
Not like a few years back when a big download would cause latency to go go up to ~ 2 seconds and you could kiss goodbye to doing anything else with the connection at the same time, at least at tolerable
Re: (Score:3)
Re: (Score:1)
article's analogy is like (Score:5, Funny)
This analogy is like a bathtub, full of spiders, and on fire. It sounds dangerous, but it's self limiting.
Re: (Score:2)
That does sound like a dangerous analogy. What if the curtains catch?
Buffer bloat animation (Score:5, Informative)
The bad Bufferbloat setup is on the left (yellow dots), and the 'good' setup (i.e. how things used to be configured about 10-20 years ago when RAM was more expensive!) is on the right (cyan/blue dots).
Both sides start off okay, but notice how the left side 'queues' (tall yellow dot columns) keep on growing over time, while the right side blue columns stop short because of the small buffer size. As they stop short, some data 'packets' must be dropped, and this gets reported back to the upload site that it's shoving data to the user too fast. As a result, the upload site temporarily slows the sending of data, and thus the system self-corrects.
Meanwhile, on the left side, these packets of data never get dropped, so the giant bloated yellow buffers get filled more and more, but the computer at the upload site doesn't realise the carnage of these giant queues further down the line, and instead thinks "All is okay, let's keep sending data fast!".
Finally, when a smaller piece of data needs to be sent to the user (see 2:30+ signified by red dots on the left and dark blue dots on the right), the left side shows the red dots (which could be say, a small email) wading through giant queues to reach their destination, really slowly. Furthermore these tiny bits of data often need special 'emergency' treatment as they hold up other larger data associated with it. On the good right side, the dark blue dots have no such giant queues.
Lag-o-Meter-of-Internet-Doom (Score:5, Interesting)
If you look at buffers allocated to fast multi-gigabit interfaces at the core of the network they are simply not large enough compared to forwarding rates involved to be able to induce the kinds of delays needed to cause Internet wide problems.
You can argue they may not be ideal for real time voice, game or video communication when these links are oversubscribed but no doomsday is possible.
Today buffer bloat effects are mostly observed at the edge even though they need not always be.
Failure of a congestion control algorithm to control link saturation does not translate into congestive collapse of the larger network. It just results in *your* network connection turning to shit. When netalyzer runs it intentionally saturates your link at that time. In the real world only a few portions of the edge are ever saturated to the extent congestion control failure becomes an issue leading to more packets through core routers. The number of edge machines in this category would need to be significant to cause a rerun of previous issues.
That condition can not be met due to self feedbacks. If everyone maxed their pipes at once the core would saturate self-limiting edge saturation due to gross over-provisioning of available edge bandwidth in relation to core bandwidth which would ensure congestion control algorithms function properly.
I'm not arguing there is not a problem or more can't be done. I'm just arguing the doomsday congestive collapse scenario is bullshit.
Re: (Score:2)
There are significant networks that do not look like the consumer edge Internet, one of which reportedly collapsed in a nasty way (not necessarily in the same way as 1986). Don't presume that the network that may collapse is the global internet (though time based self synchronizing phenomena are a worry there). One of the functions of AQM algorithms is to ensure that TCP flows don't synchronize their behavior. And those AQM algorithms are MIA on many networks today.
Those of us who lived through the 1986 c
solution (Score:2)
Easy solution? (Score:1)
This government has literally spent billions through DHS to fehn internet safety. I may be simple, but that same overreaching government could give $500m to 4 private non-profits to actively deal with this and other "infrastructure" issues like IPv6, more rural broadband, urban wireless, and other issues.
It has been done in the past through Red Cross for disaster relief, in floods, earthquakes, epidemics and the like. Society impact issues.
NGO's have far more efficient resource allocation than the governm
Re: (Score:2)
What Jim and the bufferbloat.net's group of volunteers have accomplished in a year - on nearly no money - boggles my mind.
Today's commentary on slashdot is a hundred times more clueful than it was last year - and a few days back Byte Queue Limits went into linux's net-next tree, which fixes much of the bloat problems that exist at the ethernet driver layer.
What has been discussed as 'Time in Queue' limits in the higher level schedulers is still awaiting a clean way to avoid layer violations. I've been too d
Kind of pointless when you think about it. (Score:1)
IMHO. If ISP's would build out their networks instead of relying on buffers there would not be an argument here. My attention would be on fixing dumb wireless devices and drivers that ignore every attempt of making them play nice.
Classic problem (Score:2)
Article in LinuxPro Magazine June 2011 (Score:2)
Disclaimer I'm the author. I covered this in my June 2011 column: http://www.linuxpromagazine.com/Issues/2011/127/Security-Lessons-Bufferbloat/%28kategorie%29/0 [linuxpromagazine.com] direct link to the PDF http://www.linux-magazine.com/w3/issue/127/058-059_kurt.pdf [linux-magazine.com]. In a nutshell: my link latency at home is usually ~50ms to seifried.org, but with one single outbound file transfer to saturate my uplink ping times go to over 1000ms (1 second) reliably (which completely breaks VOIP/games/etc.).
How can I improve my own connection? (Score:4, Interesting)
What can I do with my own laptop and wifi router to make my own situation better?
Re:How can I improve my own connection? (Score:5, Interesting)
As an end-user there are only a few things you can do:
1) Reduce the outgoing tcp queue size.
2) Reduce the tx ring buffer size in the network device driver
3) Set your router's upstream quality-of-service settings to throttle your upstream data transmission rate to just less than your upstream bandwidth.
Alternately, if you only have one heavy user of upstream bandwidth you could do something like what is described at "http://wanners.net:8000/blog/2011/05/zapping-upload-bufferbloat-with-one-command/". Basically throttling the upstream bandwidth directly on the machine in question rather than on the router.
net neutrality wont help (Score:1)
The proposed solution of actve queue management is exactly the sort of discrimination the net neutrality folks want to forbid, no?
Re: (Score:2)
No, it isn't based on who you are talking to....
Re: (Score:1)
Re:Alarmism? (Score:5, Informative)
Except it is an alarmist. The current situation isn't optimal but being optimal and having a critical issue are two different things. The crux of the problem is basically "Long delays from bufferbloat are frequently attributed incorrectly to network congestion, and this misinterpretation of the problem leads to the wrong solutions being proposed." That means is the administrators *might* mistake large buffer slow downs for other causes of network congestion. Idealy, it should definitely be dealt with better but it's hardly a collapse of the network.
A network buffer acts just as that, a buffer to smooth out traffic spikes. A buffer does this at the cost of latency. If a buffer is large AND consistently full, that means that network link is always being fully utilized to where a large buffer isn't needed which basically induces large latency on top of waiting for the link to clear for no benefits (the extra latency *may* confuse administrators is basically the "danger"). On the other hand, if the link is under utilized the majority of times, the a large buffer is beneficial to deal with spike traffic. The majority of networks are the latter and hence designed as such. Two solutions, get faster links or deal with it more intelligently.
Re: (Score:3)
But that's the point, the buffers smooth the link, but not the streams going across them. At enough of a buffer bloat, the buffers actually make the link have to retransmit the same data multiple times due to the design of TCP congestion avoidance.
Re: (Score:2)
The problem is *we don't know* what will happen.
The network is operating in a regime it wasn't intended to... And we lack the instrumentation we need to understand how the internet is operating.
The buffers are now so large that we've defeated the basic congestion avoidance algorithms (slow start and congestion avoidance). Technically, TCP congestion avoidance is operating, but the time gets so long that TCP thinks the path changed, and probes more aggressively for a new operating point, as explained in the
Re: (Score:2)
fine then, ignore the warnings, but don't come crying to me when you can't download pornography one day.
Re: (Score:2)
...he said, over the internet.
Re: (Score:1, Offtopic)