Comcast Reduced 'Working Latency' By 90% with AQM. Is This the Future? (apnic.net) 119
Long-time Slashdot reader mtaht writes:
Comcast fully deployed bufferbloat fixes across their entire network over the past year, demonstrating 90% improvements in working latency and jitter — which is described in this article by by Comcast Vice President of Technology Policy & Standards. (The article's Cumulative Distribution Function chart is to die for...) But: did anybody notice? Did any other ISPs adopt AQM tech? How many of y'all out there are running smart queue management (sch_cake in linux) nowadays?
But wait — it gets even more interesting...
The Comcast official anticipates even less latency with the newest Wi-Fi 6E standard. (And for home users, the article links to a page recommending "a router whose manufacturer understands the principles of bufferbloat, and has updated the firmware to use one of the Smart Queue Management algorithms such as cake, fq_codel, PIE.")
But then the Comcast VP looks to the future, and where all of this is leading: Currently under discussion at the IETF in the Transport Area Working Group is a proposal for Low Latency, Low Loss Scalable Throughput. This potential approach to achieve very low latency may result in working latencies of roughly one millisecond (though perhaps 1-5 milliseconds initially). As the IETF sorts out the best technical path forward through experimentation and consensus-building (including debate of alternatives), in a few years we may see the beginning of a shift to sub-5 millisecond working latency. This seems likely to not only improve the quality of experience of existing applications but also create a network foundation on which entirely new classes of applications will be built.
While we can certainly think of usable augmented and virtual reality (AR and VR), these are applications we know about today. But what happens when the time to access resources on the Internet is the same, or close to the time to access local compute or storage resources? What if the core assumption that developers make about networks — that there is an unpredictable and variable delay — goes away? This is a central assumption embedded into the design of more or less all existing applications. So, if that assumption changes, then we can potentially rethink the design of many applications and all sorts of new applications will become possible. That is a big deal and exciting to think about the possibilities!
In a few years, when most people have 1 Gbps, 10 Gbps, or eventually 100 Gbps connections in their home, it is perhaps easy to imagine that connection speed is not the only key factor in your performance. We're perhaps entering an era where consistently low working latency will become the next big thing that differentiates various Internet access services and application services/platforms. Beyond that, factors likely exceptionally high uptime, proactive/adaptive security, dynamic privacy protection, and other new things will likely also play a role. But keep an eye on working latency — there's a lot of exciting things happening!
But wait — it gets even more interesting...
The Comcast official anticipates even less latency with the newest Wi-Fi 6E standard. (And for home users, the article links to a page recommending "a router whose manufacturer understands the principles of bufferbloat, and has updated the firmware to use one of the Smart Queue Management algorithms such as cake, fq_codel, PIE.")
But then the Comcast VP looks to the future, and where all of this is leading: Currently under discussion at the IETF in the Transport Area Working Group is a proposal for Low Latency, Low Loss Scalable Throughput. This potential approach to achieve very low latency may result in working latencies of roughly one millisecond (though perhaps 1-5 milliseconds initially). As the IETF sorts out the best technical path forward through experimentation and consensus-building (including debate of alternatives), in a few years we may see the beginning of a shift to sub-5 millisecond working latency. This seems likely to not only improve the quality of experience of existing applications but also create a network foundation on which entirely new classes of applications will be built.
While we can certainly think of usable augmented and virtual reality (AR and VR), these are applications we know about today. But what happens when the time to access resources on the Internet is the same, or close to the time to access local compute or storage resources? What if the core assumption that developers make about networks — that there is an unpredictable and variable delay — goes away? This is a central assumption embedded into the design of more or less all existing applications. So, if that assumption changes, then we can potentially rethink the design of many applications and all sorts of new applications will become possible. That is a big deal and exciting to think about the possibilities!
In a few years, when most people have 1 Gbps, 10 Gbps, or eventually 100 Gbps connections in their home, it is perhaps easy to imagine that connection speed is not the only key factor in your performance. We're perhaps entering an era where consistently low working latency will become the next big thing that differentiates various Internet access services and application services/platforms. Beyond that, factors likely exceptionally high uptime, proactive/adaptive security, dynamic privacy protection, and other new things will likely also play a role. But keep an eye on working latency — there's a lot of exciting things happening!
The same thing that always happened: (Score:5, Interesting)
But what happens when the time to access resources on the Internet is the same, or close to the time to access local compute or storage resources? What if the core assumption that developers make about networks â" that there is an unpredictable and variable delay â" goes away?
The same thing that always happened when access to distant resources became acceptably fast. Back in the very early days of the internet and mailnet there was this little rhyme about how, as locality became less critical, people tended to select hosts without regard to locality:
"A host is a host, from coast to coast
And nobody talks to a host that's close,
Unless the host that isn't close
Is busy, hung, or dead!"
Re: (Score:2)
These days most hosts are close thanks to CDNs. The user doesn't select a host, it's done automatically for them.
Re: The same thing that always happened: (Score:2)
You're never going to see sub millisecond latency as an end user for those types of applications. What Comcast is doing is finding ways to reduce additional latency that gets introduced at network "hops" as a result of buffer bloat.
Re: (Score:2)
Re: The same thing that always happened: (Score:2, Interesting)
For games, VR, and interactive computer use, the situation is indeed grim. Even 5-10ms (equivalent to a single video frame of latency at 200 to 100fps) creates enough perceptible lag to feel like the scene is 'sloshing'.
Somewhere around 2-3ms (~400-500fps, with one frame of latency), it becomes fast enough to not LOOK like blatant slosh... but then you fall off the cliff into the uncanny valley, because your BRAIN still knows something isn't quite right.
The problem is with peripheral vision. If you're focus
Who's going to provide those speeds? Comcast? Hah. (Score:2)
Comcast goes out of its way to never spend any money in regions where they have an effective monopoly. Their infrastructure has been crumbling, while their fees have been skyrocketing.
Oh, and AT&T ("DSL") isn't much better. We get saturation advertising for "Gigabit Fiber", but 95% of the SF Bay can't get anything faster than 2 or 3 Mbps.
Re: (Score:3, Interesting)
Can you prove that with data, or are you just making stuff up because you have an axe to grind?
I work for Comcast, and I can tell you that, you are indeed quite wrong. Comcast operates one of the worlds most advanced cable networks, that doesnt happen by ignoring it. Remember when everyone started working from home in 2020? Network held up pretty good didnt it? Yes.
Thank you, have a nice day.
Re: (Score:1)
Your automated customer service needs work though. The simple task of registering the new modem took several hours since the system doesn’t understand that task.
Re: (Score:2)
TF are you talking about?
Re: (Score:2)
They gradually removed all or almost all subscriber control over this functionality, so it's always on and you can't really do anything about it unless you supply your own cable modem.
Re: Who's going to provide those speeds? Comcast? (Score:2)
Re: (Score:1)
Re: (Score:2)
Re: Who's going to provide those speeds? Comcast? (Score:2)
Re: (Score:2)
I believe s/he is complaining about the fact that most comcast modems come with a separate wireless SSID outside of your home network that any comcast subscriber can log in and use if they're nearby.
They gradually removed all or almost all subscriber control over this functionality, so it's always on and you can't really do anything about it unless you supply your own cable modem.
As an added bonus, you can setup phishing wifis using their SSID.
Re: (Score:1)
Re: (Score:1)
Re: Who's going to provide those speeds? Comcast? (Score:2)
Re: Who's going to provide those speeds? Comcast? (Score:3)
Re: (Score:3)
Of course, my only real option is AT&T U-verse that is really DSL here, and like 4M down and 1-ish meg up. If any real compe
Re: (Score:1)
Re: Who's going to provide those speeds? Comcast? (Score:2)
Is that 60 in kbps or Mbps? My friendâ(TM)s Comcast never goes down, but itâ(TM)s only about 200 kbps so itâ(TM)s only a little faster than what you have.
Re: (Score:1)
AQM helps on DSL too (Score:2)
Re: (Score:2)
Can you prove that with data, or are you just making stuff up because you have an axe to grind?
I work for Comcast, and I can tell you that, you are indeed quite wrong. Comcast operates one of the worlds most advanced cable networks, that doesnt happen by ignoring it. Remember when everyone started working from home in 2020? Network held up pretty good didnt it? Yes.
s/cable network/pots/ and reread what you just said because this is effectively current reality. Comcast is in denial no different than local carriers who thought ADSL would be good enough and therefore didn't have to invest in new last mile technology.
Where I live some tiny mid sized ISP few have heard of went thru town getting everyone hooked up with fiber. You better believe I switched immediately. I am paying half what I was before (normal non-intro pricing) for 4x more down, 60x more up than Comcas
Re: Who's going to provide those speeds? Comcast? (Score:2)
I have 384 kbps right now in Seattle. This story is just pure PR. We need to finally increase speeds since theyâ(TM)re only about an order of magnitude faster than dialup from twenty+ years ago.
Re: (Score:2)
My ISP does not need AQM (Score:2)
No buffers, no buffer-bloat. It is cheap-ass ISPs that have this problem.
Re: My ISP does not need AQM (Score:3)
Re: (Score:2)
Re: My ISP does not need AQM (Score:2)
I think if you've got a seriously overprovisioned network, it doesn't matter what algorithm you use to manage buffers since they're always empty.
Re: (Score:2)
I think if you've got a seriously overprovisioned network, it doesn't matter what algorithm you use to manage buffers since they're always empty.
Well, yes. But it is well-known that TCP/IP _needs_ a seriously overprovisioned network to work well. This is not a new insight in any way. Cheap-ass ISPs just try to squeeze more money out of their customers by skimping on bandwidth. With predictable results.
Re:My ISP does not need AQM (Score:5, Informative)
No buffers just mean that you can't handle even the smallest burst over your connection rate without dropping packets. That really sounds like a cheap-ass ISP to me.
Buffers are not there to fix your low connection speed. They are there to handle bursts, and bursts in traffic can happen at any connection speed.
As long as you have 3 identical ports, it is always possible that two of them are sending to the third at the same time. No amount of over provisioning can fix that problem.
Buffers are great, and easy, when you have small bursts because then the buffers empty themselves naturally. Buffer bloat arises when you have sustained traffic at max rate because then the buffers won't get a natural chance to empty, even if your traffic is no longer above your connection speed.
AQM is supposed to let you handle big bursts but at the same time make sure the buffers do not stay full for long periods of time.
If it works well, it is much better than "no buffers".
Re: (Score:2)
Re: (Score:3)
Of course it is no magic solution, but there is no realistic way to provision in such a way that the receiver capacity in one node is greater than the total sender capacity of all the nodes it is connected to. This is of course depending on your specific network scenario, and some problems are solved with just adding more capacity, but in general it is not possible to solve all your network problems with just "more bandwidth". When downloading files it is almost always so that you have more send capacity th
Re: (Score:3)
no realistic way to provision in such a way that the receiver capacity in one node is greater than the total sender capacity of all the nodes it is connected to
Nobody's asking for a 1:1 contention ratio. The only requirement is to provision enough capacity to not have regular congestion. The point is that AQM doesn't "fix" congestion. It just throttles bandwidth selectively to hide it. The customers get less than the paid-for bandwidth, but low latency is nice, I guess. The situation you claim to fix is not a burst but congestion: The senders can't cause a problem by filling up big buffers that aren't there. They slow down to the capacity of the receiver, which is
Re: (Score:2)
The situation you claim to fix is not a burst but congestion
Correct. The problem is that you want big buffers to handle the bursts. If you just throw out the buffers you lose the benefit that they provide.
The goal of AQM is to restore this property by selectively throttling traffic so that some traffic doesn't see congestion at all while other traffic is slowed down below the advertised bandwidth
If it gets slowed down significantly below advertised bandwidth, then it has been poorly implemented.
The point is that AQM doesn't "fix" congestion.
Correct, it hides the problems caused by congestion. Your "solution" tries to fix the congestion problem by making the burst problem worse.
For senders to slow down, their packets need to be dropped, not buffered
Not true. https://en.wikipedia.org/wiki/... [wikipedia.org]
With ECN you can slow down the sender before you need to drop packets.
the reason why TCP doesn't start full blast but takes some time to reach the full capacity of the connection
And everyone a
Re: (Score:2)
All ISPS and most modern technologies need AQM (Score:2)
You *are* correct in that a right-sized buffer (say, between 20 and 60ms) operates well at a fixed speed offered. Ethernet (except w
Re: (Score:2)
Unfortunately, the underlying bandwidth available in *shared mediums* like wifi, lte, cable, many gpon services, starlink, even dsl, fluctuates, often quite widely, and the only way to right-size the buffer to the rate is via an AQM technology.
Yes, some of the effects of the congestion you describe can be mitigated by throttling. It's just more marketing friendly if you describe it your way*. If you have a congested access network and don't do anything about it, it's worse than AQM. If you solve the congestion, you don't need AQM.
*) I'll give you the benefit of the doubt and assume that you understand that the point of AQM is to slow down transmissions to a lower speed than the customer paid for in order to prevent a buffer build up.
Re: (Score:2)
Not to be too pointed but when you say this " I'll give you the benefit of the doubt and assume that you understand that the point of AQM is to slow down transmissions to a lower speed than the customer paid for in order to prevent a buffer build up."
I am mostly inclined to try and point you at the
Re: (Score:2)
Re: (Score:2)
All networks have a bottleneck. The natural behavior of TCP is to saturate that, and
Re: (Score:2)
There's no "throttling"
Of course there is. Dropping packets is a signal indicating to the sender to slow down. If an ISP drops packets before I saturate the bandwidth I pay for, I'm not getting what I pay for. An ISP can take advantage of the fact that not everybody uses the bandwidth to the fullest at the same time, which is why contention ratios exist, but ultimately the ISP has to provide enough bandwidth to satisfy the bandwidth demand. They took the money and must provide the agreed-upon service. Trying to improve the latenc
Re: (Score:2)
You are correct. If I go above the 1Gpbs symmetrical I have, packets get lost. You cannot squeeze more data through a fiber than it can carry.
Seriously, I have done TCP/IP networking for 35 years now, including some large-scale traffic analysis stuff. Don't presume to lecture me. The article is about buffers in the network, not end-system side buffers. Apparently you missed that little detail. Apparently you also have no clue what the long-standing "buffer bloat" issue really is about. Buffers have no place
all ISPs and most modern tech needs AQM (Score:2)
Starlink (Score:3)
Is this the thing that dude a while back was complaining Starlink didn't do but needed to?
Re: (Score:3)
Latency is irrelevant (Score:2)
Re: (Score:3)
Iffy.
I have either very short duration outages of pretty horrific latency. It's hard to say which is which.
Speed is reasonable, if overpriced.
But low latency means VoiP becomes actually usable instead of some curiosity, as well as a whole host of communications I can imagine.
Various types of networking become more efficient... there is a lot of promise here.
Re: (Score:1)
Re: (Score:1)
Re: Latency is irrelevant (Score:2)
Cringely warned us. (Score:2)
Re: (Score:2)
Security agencies monitoring and censorship (Score:2)
Most governments will want to have the same kind of capability to spy on and censor Internet activity that they have today. Doing so with the applications of the future would be insanely resource intensive, even if advances in artificial intelligence make it technically feasible. Will governments step in with legislation to prevent the development of applications that they fear would be outside of their control?
Re: Security agencies monitoring and censorship (Score:2)
Can you expand on how reduced latency and more cloud applications reduces the government's ability for surveillance or censorship? Or is this something specific to bufferbloat?
Re: (Score:3)
The issue is not to do with various kinds of performance improvements themselves, but the applications they will facilitate in the future. Blocking access to websites your government does not want you to see, and intercepting all email activity is pretty much a solved problem. On the other hand, it is already difficult and resource intensive to spy on audio and video conversations, identifying and recording automatically those of potential interest. Recording and analysing all the activity occurring in virt
Yeah right. (Score:3)
> in a few years, when most people have 1 Gbps
hahahhahahhhahahhahahahah ahahahahhahahahahaaah hahahahahhahaa hahahahhahahhahahahahha ahhahahahahahahahhahahahaha hahahahha!!!!
Re: (Score:2)
This was my exact reaction. I don't forsee having even 0.1 Gbps within the next several years, let alone full 1 Gbps.
I think it's stretching things to even say that "most people will have 1 Gbps easily available in their area," let alone actually have it, within the next couple years.
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
Topsham checking in. I am sitting at .831Gbps on the 800Mbps residential plan according to an Ookla speed test.
I had similar speeds in Bowdoinham with Comcast.
Are there places in Maine that have crap broadband? Yes, but the areas served by Comcast, which is only around 10 towns, have been pretty good and reliable.
Full disclosure, not all of Bowdoinham is serviced by Comcast. There are areas in that town that up until recently had no Cable or Broadband options.
Considering Comcast has a total of 10 towns in M
Re: (Score:2)
On one hand, your laughter may be warranted. Certainly the majority of people in the world won't have 1 Gbps by then, but what about in developed countries, unlike the USA?
OK but seriously folks, even in the USA we're starting to get pretty good access to Gbps internet for people who live in urban centers, and even for people who just live near them. However, pretty much all last mile internet in the USA is massively oversubscribed (read: woefully underprovisioned) so you can only get those speeds when your
Re: (Score:1)
Re: (Score:2)
Well, I'm using DOCSIS 3.0 and I have 400 Mbps service, and sometimes I see it and sometimes I don't so I have no confidence that my ISP (Suddenlink) could provide 1 Gbps reliably, let alone 10.
I'm happy when I can get 400 Mbps, I can only actually use about 200 on my desktop because I don't have a wire to the router right now so I'm using another router as a bridge, the rest is left for the rest of the household. I wouldn't mind gig but I'm not really willing to pay more to get it.
Re: (Score:2)
> in a few years, when most people have 1 Gbps
hahahhahahhhahahhahahahah ahahahahhahahahahaaah hahahahahhahaa hahahahhahahhahahahahha ahhahahahahahahahhahahahaha hahahahha!!!!
I'm confused, do you not live in a first world country?
Are they just ignoring that distance thing? (Score:2)
Re: (Score:2)
Re: (Score:2)
I have just tried pinging a number of servers in Universities in the UK from a server located in data centre in a UK university. Yeah they are all within a few ms of the speed of light in glass on the straight line distance between them. I am not sure how this is going to help reduce latency. You canny change the laws of physics captain. Which is oddly appropriate as the server I was pinging from is in Scotland.
Re: (Score:3)
It's still relevant since the latency that's added by the networking hardware can add up to significant amounts. So what this does is shave off some overhead that's added by the network IF everyone in between you and that other person 1/3 of the planet away adheres to this, bringing the delay closer to what is technically possible over optical fiber and or copper with around 2/3 of the speed of light.
Hollow-core fibre could be the future and
Like I'm a 6 year-old (Score:4, Insightful)
Could one of you please explain to me in the simplest terms what "AQM" means? I'm tired and can't bring myself to RTFA. Some of you sound like you know what you're talking about, so it would be a big help.
TIA
Re: (Score:2)
Re: (Score:2)
this [wordpress.com] seems like a good link.
As I understand it, if there is a multi-hop path to a destination and packets get buffered waiting to traverse one of the hops then the systems at each end can't do much to detect the issue and modify their behavior. But if the ISP drops packets instead of allowing a queue to build (or has some other congestion notification mechanism) then the endpoints can detect the packet loss and, slow their sending rate, reducing the load on the link
I suppose if the endpoints knew an expect
Re: Like I'm a 6 year-old (Score:5, Informative)
Re: Like I'm a 6 year-old (Score:2)
Thank you for the explanation.
On a general note it's interesting how many methods to handle congestion don't actually work out that well in real life. Ethernet pause frames comes to mind.
Re: (Score:3)
Active Queue Management.
Normally routers use a basic FIFO, and when the buffer gets full they start dropping packets until there is space. Devices sending packets to the router notice that packets are being dropped and reduce the transmission rate in response.
The problem with the FIFO is that if the buffer is large then packets get stuck in there for a long time, and latency increases. If the buffer is small it limits the maximum speed at which the router can route packets, and causes a lot of packet drops.
Measuring latency under ISP load and WiFi issues (Score:2)
Great TOOL! (Score:2)
Re: (Score:2)
Re: (Score:2)
Bufferbloat is a symptom of congestion (Score:2)
Re: (Score:2)
sub 1ms ping to goog (Score:2)
Re: (Score:2)
I am seeing a LOT of 100mbit fiber networks that are seriously overbuffered on uploads, and underbuffered at 1gbit (a *target* of 5ms is better than a 5ms buffer)
Re: (Score:2)
Re: (Score:2)
On the shell I get 0.4ms ping times to Google. This is with fiber tth, a PC with wired Gbit/s to a switch and then a router to the fibre transceiver.
If this is true, then both your physical location and your routing are extremely unusual:
0.4 ms at the speed of light is equal to 120 km, or a distance of 60 km (ping is round trip) to "Google"
(whatever that actually means).
Probably even less, since light in glass is quite a bit slower than light in vacuum.
Re: (Score:2)
Kudos to Comcast (Score:2)
I've been a Comcast customer for the past decade.
Running the DSL Reports test in the past always gave an abysmal score for bufferbloat, but I had not run it in ages.
I ran it just now and got a bufferbloat score of "A".
Re: (Score:2)
Light speed doesn't change (Score:2)
even if we fix the jitter and increase predictability, light speed is till throwing a wrench in the works. Typical wired network latencies from south america to the US are around 150-180ms on a good day. You need servers everywhere to fake low latency, which is expensive.
Even at lightspeed in the vacuum, you cannot get a roundtrip under 46ms, on fiber is closer to 70ms, add a few hops and you get to the number we have today.
If you make an applica
Re: (Score:2)
Glad EditorDavid read the whole piece (Score:2)
Most Comcast customers need AQM (Score:2)
This is great news for Comcast customers because most of them have very low upstream bandwidth.
I suffered from bufferbloat when I was on legacy DSL, VDSL, and even symmetric 50 Mbps fiber. While not entirely gone, the worst part of bufferbloat as tested by dslreports.com and waveform.com largely went away, going from F to B, on symmetric 300 Mbps fiber overprovisioned to 360 Mbps.
Re:Most ISP customers need AQM (Score:2)
That said *even at gigE*, most users will see some benefit from quality fair queuing, especially, coupled with AQM. FQ in parti
Meanwhile on AT&T Fiber (Score:2)
Iâ(TM)m getting 25-28 ms ping to their gateway on their their FTTH service. Clearly not all companies understand or care about something as important as latency.
You need more (Score:2)
You need a backbone capable of handling near-peak loads, tier 2 nets that have redundancy and at minimum the capacity to handle the rms of the actual peak loads.
To avoid congestion caused by a few users hammering the net, have bandwidth guarantees and then any load above that on a best effort.
If the router can't handle the load, get a better router. Former highend gear can get reused or cannibalised.
If you've dark fibre and can't afford to light it all the time, light it up at peak times.
Also, do less. At t
dropping packets (Score:2)
Dropping more packets and sooner. Sounds like great service.
^ Mod parent up. (Score:2)
Comcast is effectively a monopoly in my area as well - and has been for decades. While it's mostly reliable here, the prices increase with their threats and caps. Fuck them.
cake (Score:2)
I have high hopes something like cake will be widely adopted by ISPs one day, and not just advanced users, also.
As to your que