Slashdot Log In
Boosting Socket Performance on Linux
Posted by
CmdrTaco
on Thu Jan 19, 2006 03:55 PM
from the everyone-likes-more dept.
from the everyone-likes-more dept.
Cop writes "The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high performance -- or inhibit it. This article explores four ways to use the Sockets API to squeeze the greatest performance out your application and to tune the GNU/Linux® environment to achieve the best results."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Be aware (Score:4, Funny)
Re:Be aware (Score:4, Insightful)
Exactly... especially with things like these, it's usually best for the entire internet if you just stick with the defaults... they are defaults for a reason, it might not be the best for you, but it's most likely the best for the internet as a whole.
Reminds me of those people tweaking firefox settings to hammer all kind of webservers... sure, your browsing might be a slight bit faster, at the expense of the browsing of lots of other people...
Parent
Re:Be aware (Score:3, Interesting)
Re:Be aware (Score:2, Interesting)
Re:Be aware (Score:3, Interesting)
are you sure?
From a paper written by Phil Dykstra, back in 1999.
"A recent example comes from the Pacific Northwest Gigapop in Seattle which is based on a collection of Foundry gigabit ethernet switches. At Supercomputing '99, Microsoft and NCSA demonstrated HDTV over TCP at over 1.2 Gbps from Redmond to Portland. In order to achieve that performance they used 9000 byte packets and thus had to bypass the switches at the NAP! Let's hope that in the future NAPs don't place 1500
Re:Be aware (Score:2)
Re:Be aware (Score:4, Informative)
Parent
slashdotted? (Score:5, Funny)
Re:slashdotted? (Score:2)
Judging by the response time from IBM's web server, it looks like they have yet to put their advice into practice.
... or too many Slashdot visitors already did that exact thing... :-)
Re:slashdotted? (Score:2)
Boost socket performance on Linux
Four ways to speed up your network applications
M. Tim Jones (mtj@mtjones.com), Senior Principal Software Engineer, Emulex
17 Jan 2006
The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high perf
GNU/Linux®...A lessefficent way to say Linux (Score:2, Funny)
Hello 1995 (Score:5, Insightful)
Re:Hello 1995 (Score:4, Interesting)
In the same line - where is the discussion of different FD table polling mechanisms? select() versus poll(), and wheres the writeup about Linux's epoll(). I would have been interested in an epoll() article, especially how it compares to FreeBSD's kqueue().
Parent
Re:Hello 1995 (Score:5, Informative)
For the overview, you want Dan Kegel's c10k page:
http://www.kegel.com/c10k.html [kegel.com]
Parent
Hello 2003. (Score:5, Interesting)
Documentation like this is great and extremely valuable. It would be much more valuable, however, if it remained current. For example, can the ABISS [sourceforge.net] project (which improves block I/O) be used at all? What do the numbers look like, when using profiling tools like Web100 [web100.org] (which profiles TCP communications)?
Has anyone run the Linux or one of the *BSD kernels through DAKOTA [sandia.gov], KOJAK [fz-juelich.de] or PAPI [utk.edu] to determine where, precisely, bottlenecks are within the kernels? It's easy to theorise, but isn't it cleaner to measure?
Now, I'm not saying these things aren't being done. They probably are, somewhere, by someone, but if the results aren't getting published we don't really know what impact what changes are going to have. The current method of evolving Operating System code in general is often a mix of personal theory and subjective experience based on non-random samples of activity. That can't really be a good way to do things, can it?
If I'm wrong, feel free to say. If I'm right, then maybe it would be a good thing if someone (possibly me) put together some kind of testing kit for measuring Linux kernel performance and actually measured the stats for Linux kernels on some kind of regular basis.
Parent
Re:Hello 1995 (Score:2)
Vuja de rules!
Re:Hello 1995 (Score:2)
Actually, given that it's 2006, I would have thought that the socket layer would be smart enough to perform these sorts of "optimizations" for you automatically, by analyzing your usage patterns. There's no reason the programmer should have to deal with any of this crap, except maybe by providing a broad hint such as "Maximize throughput" or "Minimize latency."
Re:Hello 1995 (Score:2)
To a certain degree, they are optimized. Since most network activity occurs through a higher level networking API (e.g. HTTP), the network performance is already optimized by the library. It's not all that often that you have to open a direct socket unless you happen to be writing such a library or server.
Which just further points out how much this a
Re:Hello 1995 (Score:2)
BDP = link_bandwidth * RTT
100MBps * 0.050 sec / 8 = 0.625MB = 625KB
Note: I divide by 8 to convert from bits to bytes communicated.
So, set your TCP window to the BDP, or 1.25MB. Where does
Is this a typo, or am I missing something in the calculation?
Re:Hello 1995 (Score:2)
Re:Hello 1995 (Score:4, Informative)
From the MAN page [linuxmanpages.com]:
The article could have better explained that in context. For the most part it's automatic though, so don't worry about it.
Parent
Re:Hello 1995 (Score:2)
Re:Hello 1995 (Score:3, Insightful)
Now if only the article actually covered something specific to Linux, I'd agree with you. About the most useful thing it does is tell you the location of the same parameters that you muck with on every other system in existence. This info has only been around for Linux for, oh, more than a decade. Pick up any bo
Summary ripped directly from article (again) (Score:2, Informative)
Here is the summary:
The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high performance -- or inhibit it. This article explores four ways to use the Sockets API to squeeze the greatest performance out your application and to tune the GNU/Linux® environment to achieve the best results.
Here is the first paragraph of the article:
The Sockets
Re:Summary ripped directly from article (again) (Score:2)
-l
Re:Summary ripped directly from article (again) (Score:2)
No wonder he was un
No mention of alternatives to select? (Score:5, Informative)
depends on how it's used (Score:2)
Re:depends on how it's used (Score:2)
True enough. But how many applications nowadays are written expecting no more than ~32 simultaneous connections?
Re:No mention of alternatives to select? (Score:2)
Ok, the issue is how many fds you can pass. With select() you are limited to a bitmaps worth. And performance has never been much of an issue.
Of course, poll() is a different matter -- if you are passing 100s or thousands of fds.
But, what has this got to do with the tcp connection? Not much.
So, you speed up poll() and still write small packets, and nagle won't write them out immediately... That's about the only connection here.
Ratboy.
Re:No mention of alternatives to select? (Score:2)
Agreed. But the topic of his article is "Boost socket performance on Linux," not "How to optimize TCP layer use on Linux." And the article deals almost entirely with API-level settings. It just seems odd to me that he'd not mention issues with some of the classic BSD functions.
Re:No mention of alternatives to select? (Score:3, Informative)
http://www.kegel.com/c10k.html [kegel.com]
Very awesome paper. How do _you_ make a server that handles 10,000 connections?
--jeff++
Re:No mention of alternatives to select? (Score:5, Informative)
http://www.xmailserver.org/linux-patches/nio-impr
The website is hideous, but there used to be benchmarks against different polling/selecting methods. If I remember correctly, its kinda trial and error, YMMV, kind of stuff. Its worth a look.
Parent
Re:Code Portability (Score:5, Informative)
Parent
GNU/Linux®? (Score:2)
Re:GNU/Linux®? (Score:3, Informative)
Re:GNU/Linux®? (Score:2)
Duh?
IBM is getting some good Linux content... (Score:5, Interesting)
Folks who are thinking about writing something technical - give dW a shot. The editors are savvy folks and there's lots of good stuff up there already.
Oh, and book plug [pmdapplied.com]!
I've always wanted to know if it is possible (Score:2)
Re:I've always wanted to know if it is possible (Score:2)
Re:I've always wanted to know if it is possible (Score:5, Funny)
It ignores you except at feeding time, and pees in your shoes when it's mad at you?
Parent
Re:I've always wanted to know if it is possible (Score:5, Informative)
Is that what you're looking for?
Parent
Nagle's algorithm (Score:5, Interesting)
To get around the above problems, I came up with the following scheme: Leave Nagle's algorithm enabled, but create a FlushSocket() function that merely disables Nagle on the socket, then calls send() on the socket with a 0-byte buffer, then enables Nagle again. This apparently forces the TCP stack to immediately send any data that it may have accumulated in its Nagle-buffer. Therefore the only thing the calling code has to remember to do is to call FlushSocket() whenever it has called send() one or more times and doesn't think it will be sending any more data any time soon.
The above technique seems to work pretty well under Linux, Windows, and OS/X (and is more portable than Linux-specific flags like TCP_CORK, etc), but I haven't seen it documented anywhere. Is that simply an oversight, or is there some nasty downside to this technique that I'm overlooking?
Re:Nagle's algorithm (Score:3, Interesting)
calls you have to pay for?
if you have some knowledge about the natural grouping of data,
it would be better to just turn nagle off and do buffering
in user space (collect up enough data and send it all in one
go)
Re:Nagle's algorithm (Score:3, Informative)
Math error in paper? (Score:3, Informative)
throughput = window_size / RTT
110KB / 0.050 = 2.2MBps
If instead you use the window size calculated above, you get a whopping 31.25MBps, as shown here:
625KB / 0.050 = 31.25MBps
That's funny, I get 12.5MBps
???
Re:somewhat old... (Score:2)
I agree though, nothing earth shaking. Nagle's algorithm is discussed in depth in most TCP/IP books, and so is how to turn it off. Wake me up when they post something new.
Re:flush( sd ) would be nice (Score:2)
See my previous post above ("Nagle's Algorithm") for a way to do it.
Re:flush( sd ) would be nice (Score:3, Interesting)
I don't know if that really makes sense for networking though... the reason you'd want fflush() to block until the data makes it to disk is so that once your call to fflush() returns you know that your written data is safe in the event of a crash or power failure. (Although with too-clever hard drive firmware I'm not so sure even that's true anymore!). With networking on the ot
The trouble with the Nagle algorithm (Score:5, Interesting)
Here's the real problem, and its solution.
The concept behind delayed ACKs is to bet, when receiving some data from the net, that the local application will send a reply very soon. So there's no need to send an ACK immediately; the ACK can be piggybacked on the next data going the other way. If that doesn't happen, after a 500ms delay, an ACK is sent anyway.
The concept behind the Nagle algorithm is that if the sender is doing very tiny writes (like single bytes, from Telnet), there's no reason to have more than one packet outstanding on the connection. This prevents slow links from choking with huge numbers of outstanding tinygrams.
Both are reasonable. But they interact badly in the case where an application does two or more small writes to a socket, then waits for a reply. (X-Windows is notorious for this.) When an application does that, the first write results in an immediate packet send. The second write is held up until the first is acknowledged. But because of the delayed ACK strategy, that acknowledgement is held up for 500ms. This adds 500ms of latency to the transaction, even on a LAN.
The real problem is that 500ms unconditional delay. (Why 500ms? That was a reasonable response time for a time-sharing system of the 1980s.) As mentioned above, delaying an ACK is a bet that the local application will reply to the data just received. Some apps, like character echo in Telnet servers, do respond every time. Others, like X-Windows "clients" (really servers, but X is backwards about this), only reply some of the time.
TCP has no strategy to decide whether it's winning or losing those bets. That's the real problem.
The right answer is that TCP should keep track of whether delayed ACKs are "winning" or "losing". A "win" is when, before the 500ms timer runs out, the application replies. Any needed ACK is then coalesced with the next outgoing data packet. A "lose" is when the 500ms timer runs out and the delayed ACK has to be sent anyway. There should be a counter in TCP, incremented on "wins", and reset to 0 on "loses". Only when the counter exceeds some number (5 or so), should ACKs be delayed. That would eliminate the problem automatically, and the need to turn the "Nagle algorithm" on and off.
So that's the proper fix, at the TCP internals level. But I haven't done TCP internals in years, and really don't want to get back into that. If anyone is working on TCP internals for Linux today, I can be reached at the e-mail address above. This really should be fixed, since it's been annoying people for 20 years and it's not a tough thing to fix.
The user-level solution is to avoid write-write-read sequences on sockets. write-read-write-read is fine. write-write-write is fine. But write-write-read is a killer. So, if you can, buffer up your little writes to TCP and send them all at once. Using the standard UNIX I/O package and flushing write before each read usually works.
John Nagle
Parent