Slashdot Log In
Better Bandwidth Utilization
Posted by
michael
on Wed Mar 05, 2003 09:46 AM
from the neat-hack dept.
from the neat-hack dept.
jtorin writes "Daniel Hartmeier (of OpenBSD fame) has written a short but interesting article which explains how to better utilize available bandwidth. In short it gives priority to TCP ACKs over other types of traffic, thereby making it possible to max both upload and download bandwidth simultaenously. Be sure to check ot the nice graphs! Also note the article on OpenBSD Journal. OpenBSD 3.3 beta is now stable enough for daily use, so why not download a snapshot from one of the mirrors and try it out?"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
How ironic (Score:4, Funny)
Re:How ironic (Score:5, Funny)
Parent
Re:How ironic (Score:3, Funny)
How to better utilize bandwidth (Score:4, Funny)
Oh, that's where OpenBSD is... (Score:4, Funny)
Now if only I could find that Linux thing...
like wondershaper does for months now? (Score:5, Informative)
Re:like wondershaper does for months now? (Score:5, Informative)
Parent
Re:like wondershaper does for months now? (Score:5, Informative)
Parent
Re:like wondershaper does for months now? (Score:5, Interesting)
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
Parent
Re:like wondershaper does for months now? (Score:4, Informative)
"To make sure that uploads don't hurt downloads,
we also move ACK packets to the front of the queue."
It's pretty cool, it throttles your speeds to just under what the maximum should be, so that queueing will only happen on your linux box, and then you can prioritize what you want.
Parent
Re:like wondershaper does for months now? (Score:4, Informative)
Parent
This will be of most use to ... (Score:4, Insightful)
Corporate networks are already optimized under 100 or gigabit ethernet with Cisco routers which automatically handle collisions and error corrections.
Re:This will be of most use to ... (Score:5, Informative)
Parent
Uh, no, I don't think so (Score:5, Informative)
Someone far more knowledgable than myself will get to correct me, but I seem to recall there was a process of-
Send some stuff-wait for ACK.
When you get the ACK, send some more.
By turbocharging the ACKs, you are reducing that lag time
Parent
Re:Uh, no, I don't think so (Score:5, Informative)
When you get the ACK, send some more.
By turbocharging the ACKs, you are reducing that lag time
Not quite. TCP streams use pipelining: you send N packets (N is the "window size"), and each time you get an ACK you send one more. So in the ideal case there's no lag, because the ACK for packet 3 lets you go ahead and send packet 10 (if N=7).
When a packet (or its ACK) gets dropped, TCP assumes the network is congested, and cuts N in half, and very slowly increases it back to where it was. So after each dropped packet or ACK you have a while during which you're not using the full link. Several drops in a row can reduce your throughput by a factor of 100 or more.
Prioritizing ACKs doesn't reduce the lag time. It reduces the likelihood that TCP will overreact and reduce its sending rate due to perceived congestion.
Parent
Re: Kinda been done with TFTP (Score:3, Informative)
Take FTP and strip the overhead error checking and if something doesn't come out right, refresh and download it again.
For streaming, you get more throughput, and every now and them you might miss a frame in exchange for the higher quality you can obtain with the lower overhead
TCP Daytona (Score:4, Informative)
The technique you suggest is one of several proposed by Stefan Savage in TCP Congestion Control with a Misbehaving Receiver [washington.edu]. He called it TCP Daytona. :)
Parent
Re:Uh, no, I don't think so (Score:3, Interesting)
Why design a stateless protocol and then put it on top of TCP, requiring a connection to be set up and torn down for each HTTP request?
Because you want reliability. Unfortunately, reliable UDP (or transactional TCP) is not widely supported.
Also, because many HTTP responses don't fit in a single UDP packet.
Re:same bandwidth in relation to what ? (Score:2)
A T1 to T1 connection usually gets me no better gameplay or internet page render speed than a cable modem connection or DSL connection or WiFi connection.
Re:This will be of most use to ... (Score:2)
We figured this before (Score:5, Interesting)
The problem is (Score:2, Interesting)
Re:The problem is (Score:5, Informative)
Parent
Interesting (Score:3, Interesting)
Linux solution (Score:3, Informative)
The Linux Advanced Routing & Traffic Control HOWTO [lartc.org] discuss how to achieve the same thing on linux using QoS. See section 9.2.2.2 [lartc.org](Sample configuration)
Re:Linux solution (Score:5, Informative)
It is a differend solution to a different problem caused by the same thing....
The cause is the big cache in the modem, it results in a delay on outgoing traffic.
One problem is that interactive traffic gets, well, less interactive (e.g. the echo characters in a remote shell have a delay). This is solved in the HOWTO you refered to.
Another problem is that the downstream acks get delayed resulting in less downstream data. This is solved in the mentioned article.
A combination of the two would be really great and could probably be done in both linux and openbsd.
Jeroen
Parent
My solution: (Score:5, Funny)
Put lower priorities on p0rn, MP3s, Windows viruses, and Slashdot referrals. That should speed everything else up by about two orders of magnitude.
Re:My solution: (Score:5, Funny)
Parent
Note the article is all about low bandwidth setups (Score:5, Insightful)
A little off topic but I always find it interesting that people with hicap gear (Foundry, Cisco, etc.) are always talking about QOS when it really only makes sense most times on low bandwidth lines. So his work is really important when you look at where it is in scheme of things - out at the end users line.
Not just low bandwidth (Score:5, Interesting)
A fair number of protocols do transmit windows of a certain size. They'll send a certain amount of data, and not send more data until the oldest packest in the window gets an ACK back. You therefore only have so much data "in-flight" at any one time. Strongly asynchronous link (like aDSL and cablemodems) can require strikingly different window sizes than synchronous links.
The right amount of in-flight data is dependent on the speed of your pipe, obviously, but a lot of applications still use defaults set for low-bandwidth pipes. You can argue that the proper solution for this is to change the defaults, but if you just give ACKs priority, you don't need to worry about it, and the less you force users to change, the better. (The transmit window size has to be a user setting, directly or indirectly, either by asking a window size, or by asking "what kind of pipe do you have?" and guessing a window size from that.)
This is dependent on the protocol, true, but giving ACKs priority is actually a decent generic solution to what many consider an application-specific problem.
QOS is also often about bandwidth guarantees, not necessarily throughput. You have a 155mbit link shared among several applications, and an application that *requires* 45mbit. So you use QOS to guarantee that application gets 45mbit if it wants it, and everything else shares the remainder. If the app isn't going, then that 45mbit it requires can be made available to other apps until it is required.
Parent
Very Usefull (Score:2, Interesting)
Is this related to another article? (Score:3, Funny)
Ask Slashdot: What percentage of internet traffic is pr0n? [slashdot.org]
Daniels original email (Score:5, Informative)
http://marc.theaimsgroup.com/?l=openbsd-pf&m
It contains a little more of the pf rules than the article does, and has all the relevant information you need except for the nice
Working Link (Score:3, Insightful)
link [theaimsgroup.com]
Trip down memory lane... (Score:3, Interesting)
Re:Trip down memory lane... (Score:4, Informative)
A) Zmodem is still around, at least in the *nix world. You can get lrzsz from here [www.ohse.de].
Some telnet clients still support Zmodem, and you can use lrzsz to transfer files via telnet. Personally, I'd rather use ssh as it's a lot more secure, but in cases where either you can only use telnet or when you are on network you can trust (i.e., not the Internet), you can still use Zmodem.
b) Zmodem is not, nor has it ever been a bidirectional protocol -- you can't upload and download at the same time unless you have two different connections. There *were* protocols that would let you do this (Puma comes to mind), but you most decidedly could NOT do this with Zmodem.
Parent
.CX Domain (Score:4, Funny)
Security hole (Score:3, Funny)
Or maybe it was flooded with SYN's? Damn. I can't remember.
Re:Security hole (Score:5, Interesting)
For each SYN packet you send, you eat up a little bit more memory and CPU time on the victim. Do it enough times, and the system runs out of memory or processor time, and the system becomes unable to perform its regular operations. Effectively causing a Denial of Service.
If you're smart, you'll form the SYN packets to have source addresses that differ from your real IP, otherwise a) you're traceable; and b) your machine will be flooded with SYN/ACKS. If you are even smarter, you'll use an IP that, while valid and routable, belongs to a host that either doesn't exist, or is currently off. Otherwise the 2nd level victim recieving the SYN/ACKs from your initial target will send RSETs for every SYN/ACK, since it never requested to initial the connection. When your target gets the RSET for the SYN/ACK, it will close the session, freeing up the memory and CPU time that you are desparately trying to fill. Essentially, a non-existant host will never respond to a SYN/ACK, so the target system has to wait for a timeout duration before closing the session, which makes it easier for you to eat up CPU and memory. Unfortunately though, the fake spirce IP on your SYN packets will likely have to be within your ISP's network range, as all smart ISP network administrators perform egress packet filtering to prevent such attacks from originating within their network.
Better tactics include sending the SYNs from multiple machines that have different providers. Thus preventing load from the SYN/ACKs from filling your ISPs pipe. This effectively makes the attack a DDoS, rather than a DoS.
Either way, you can't really perform these attacks in much safety, as competent network administrators will have sniffers in place to detect these attacks as they cross their network. So #1) if your ISP admin is smart, you're busted by them regardless; and #2) if the chain of smart admins follows you all the way back to your sources, you're busted by the authorities (which if you cross state lines means the Feds, which will suck quite adamently).
So, that is how it works, but I wouldn't recommend trying it.
Parent
Slashdotted - Mirror (Score:5, Informative)
Try it (Score:5, Funny)
"OpenBSD 3.3 beta is now stable enough for daily use, so why not download a snapshot [openbsd.org] from one of the mirrors [openbsd.org]and try it out?"
Windows XP is now stable enough for daily use, so why not download a snapshot [kazaalite.tk] from one of the mirrors [sharereactor.com] and try it out?"
(intended as a joke)
Broadband implications (Score:3, Insightful)
Then again, since when have most broadband providers really ever cared about supplying good speeds when the user maxes out the outrageously capped upstream...
this may break TCP flow control! (Score:5, Interesting)
So if the network is congested and an ACK SHOULD time out but doesn't, TCP will keep on flooding the network, ruining the pool for everyone.(see: Tragedy of the commons [dieoff.com])
Yes, I agree that this is a big-O style worse case scenario, but its something to consider.
Re:this may break TCP flow control! (Score:5, Informative)
So if the network is congested and an ACK SHOULD time out but doesn't, TCP will keep on flooding the network, ruining the pool for everyone.
No. If the downstream is flooded, the packets won't be received, and no ACK will be sent. ACKs have higher priority, but even that can't make them appear out of thin air.
Parent
W. Richard Stevens TCP/IP Illustrated, Volume 1 (Score:5, Informative)
It seems to me that a great many
Reading that book will give you a foundation to understanding how a single endpoint behaves in an IP network. If you want some understanding of the guts of a large scale internetwork I'd suggest the Cisco Press IP Quality of Service book.
There are a great many things near and dear to
If you're impatient you can look at my journal - I've covered some of the issues there.
Server got /.'ed before 0 comments... (Score:4, Informative)
http://www.benzedrine.cx/ackpri-norm.jpg
http://www.benzedrine.cx/ackpri-priq.jpg
benzedrine.cx - Prioritizing empty TCP ACKs with pf and ALTQ Prioritizing empty TCP ACKs with pf and ALTQ
Introduction ALTQ is a framework to manage queueing disciplines on network interfaces. It manipulates output queues to enforce bandwidth limits and priorize traffic based on classification.
While ALTQ was part of OpenBSD and has been enabled by default since several releases, the next release will merge the ALTQ and pf configuration into a single file and let pf assign packets to queues. This both simplifies the configuration and greatly reduces the cost of queue assignment.
This article presents a simple yet effective example of what the pf/ALTQ combination can be used for. It's meant to illustrate the new configuration syntax and queue assignment. The code used in this example is already available in the -current OpenBSD source branch.
Problem I'm using an asymmetric DSL with 512 kbps downstream and 128 kbps upstream capacity (minus PPPoE overhead). When I download, I get transfer rates of about 50 kB/s. But as soon as I start a concurrent upload, the download rate drops significantly, to about 7 kB/s.
Explanation Even when a TCP connection is used to send data only in one direction (like when downloading a file through ftp), TCP acknowledgements (ACKs) must be sent in the opposite direction, or the peer will assume that its packets got lost and retransmit them. To keep the peer sending data at the maximum rate, it's important to promptly send the ACKs back.
When the uplink is saturated by other connections (like a concurrent upload), all outgoing packets get delayed equally by default. Hence, a concurrent upload saturating the uplink causes the outgoing ACKs for the download to get delayed, which causes the drop in the download throughput.
Solution The outgoing ACKs related to the download are small, as they don't contain any data payload. Even a fast download saturating the 512 kbps downstream does not require more than a fraction of upstream bandwidth for the related outgoing ACKS.
Hence, the idea is to priorize TCP ACKs that have no payload. The following pf.conf fragment illustrates how to set up the queue definitions and assign packets to the defined queues:
ext_if="kue0"
altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)
pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)
pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)
First, a macro is defined for the external interface. This makes it easier to adjust the ruleset when the interface changes.
Next, altq is enabled on the interface using the priq scheduler, and the upstream bandwidth is specified.
I'm using 100 kbps instead of 128 kbps as this is the real maximum I can reach (due to PPPoE encapsulation overhead). Some experimentation might be needed to find the best value. If it's set too high, the priority queue is not effective, and if it's set too low, the available bandwidth is not fully used.
Then, two queues are defined with (arbitrary) names q_pri and q_def. The queue with the lower priority is made the default.
Finally, the rules passing the relevant connections (statefully) are extended to specify what queues to assign the matching packets to. The first queue specified in the parentheses is used for all packets by default, while the second (and optional) queue is used for packets with ToS (type of service) 'lowdelay' (for instance interactive ssh sessions) and TCP ACKs without payload.
Both incoming and outgoing TCP connections will pass by those two rules, create state, and all packets related to the connections will be assigned to either the q_def or q_pri queues. Packets assigned to the q_pri queue will have priority and will get sent before any pending packets in the q_def queue.
Result The following test was performed first without and then with the ALTQ rules explained above:
The first graphs shows the results of the test without ALTQ, and the second one with ALTQ:
Image 1, ACK PRI Normal [benzedrine.cx]
Image 2, ACK PRI PRIq [benzedrine.cx]
The improvement is quite significant, the saturated uplink no longer delays the outgoing empty ACKs, and the download rate doesn't drop anymore.
This effect is not limited to asymmetric links, it occurs whenever one direction of the link is saturated. With an asymmetric link this occurs more often, obviously.
Related links
How does this compare to a DDR Fairness? (Score:4, Informative)
See "QoS for Modems and Remote Access" at this KB article [microsoft.com].
ACK Shaping (Score:4, Informative)
Unlike most conventional traffic shapers which queue and control the data rate on the outgoing channel, PacketShaper controls the rate of acknowledgements on the reverse channel.
This is usually used to *slow* traffic. I.e., instead of having the router drop packets (thereby wasting resources until the source TCP understands that the net is congested and reduces load) it just slows the ACKs and the sender automatically reduces its sending rate.
Anyway, the real nice thing about the OpenBSD implementation is that they merge their packet filter (pf) with the ALTQ queuing code. Now this is really powerful.
Sounds like a good time for all BSDs to adopt this new combination instead of relying on less-capable mechanisms. E.g. FreeBSD has ipfw for filtering and dummynet for queue management. I don't know how pf compares with ipfw but ALTQ is definitely better than dummynet.
Nimrod.
throttled (Score:3, Informative)
Title is correct! (Score:5, Interesting)
The article is
The bandwidth is there, it;s just under utilised. By prioritisng the ACK's, so that they get boosted through, it becomes possible to saturate both upstream and downstream pipes at once, at peak efficency, rather than one of the coasting along, waithing for the other.
Note that this only applies for TCP/IP and similar, reliable, protocols. If you had a UDP app (e.g. media streaming done properly), then this trick won't affect it at all, as it doesn't wait for an ACK.
Parent