login
Search
So I got a netgear switch and it works fine. But my tests are busted.
Catching netdev up, I'm trying to optimize traffic to a server that has
a gbit interface; I moved to a 24 port netgear that is all 10/100/1000
and I have a pile of clients to act as load generators.
I can do this on each of the clients
dd if=/dev/zero bs=1024000 | rsh work "dd of=/dev/null"
and that cranks up to about 47K packets/second which is about 70MB/sec.
One of my clients also has gigabit so I played around with just that
one and it (itanium running hpux w/ broadcom gigabit) can push the load
as well. One weird thing is that it is dependent on the direction the
data is flowing. If the hp is sending then I get 46MB/sec, if linux is
sending then I get 18MB/sec. Weird. Linux is debian, running
Linux work 2.6.18-5-k7 #1 SMP Thu Aug 30 02:52:31 UTC 2007 i686
and dual e1000 cards:
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
I wrote a tiny little program to try and emulate this and I can't get
it to do as well. I've tracked it down, I think, to the read side.
The server sources, the client sinks, the server looks like:
11689 accept(3, {sa_family=AF_INET, sin_port=htons(49376), sin_addr=inet_addr("10.3.1.38")}, [16]) = 4
11689 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
11689 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [1048576], 4) = 0
11689 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7ddf708) = 11694
11689 close(4) = 0
11689 accept(3, <unfinished ...>
11694 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576
11694 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576
11694 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576
11694 write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., ..... This is exactly what I'd expect if the machine is *not* under excessive load. The system calls are fast enough that the latency for the TCP stack is roughly on the same scale as the time it takes to receive one new packet, so since a socket read will always return when it has any data (not until it has filled the whole buffer), you get exactly that "one or two packets" pattern. If you'd be really CPU-limited or under load from other programs, you'd have more packets come in while you're in the read path, and you'd get bigger reads. But do a tcpdump both ways, and see (for example) if the TCP window is much bigger going the other way. Linus -
That's fine, but why is it that my trivial program can't do as well as dd | rsh dd? A short summary is "can someone please post a test program that sources and sinks data at the wire speed?" because apparently I'm too old and clueless to write such a thing. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
From: lm@bitmover.com (Larry McVoy) You're not showing us your test program so there is no way we can help you out. My initial inclination, even without that critical information, is to ask whether you are setting any socket options in way? In particular, SO_RCVLOWAT can have a large effect here, if you're setting it to something, that would explain why dd is doing better. A lot of people link to "helper libraries" with interfaces to setup sockets with all sorts of socket option settings by default, try not using such things if possible. You also shouldn't dork at all with the receive and send buffer sizes. They are adjusted dynamically by the kernel as the window grows. But if you set them to specific values, this dynamic logic is turned off. -
The only one I was playing with was SO_RCVBUF/SO_SNDBUF and I tried disabling that and I tried playing with the read/write size. Didn't Agreed. That was my first thought as well, I must have been doing something that messed up the defaults. But you did get the strace Yeah, dorking with those is left over from the bad old days of '95 when lmbench was first shipped. But I turned that all off and no difference. So feel free to show me where I'm an idiot in the code, but if you can't, then what would rock would be a little send.c / recv.c that demonstrated filling the pipe. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com
Here's a simple reference tcp source/sink that's I've used for years. For example, on a couple gigabit machines: $ ./tcpsend -t10 dew Sent 1240415312 bytes in 10.033101 seconds Throughput: 123632294 B/s -John
WRT the different speeds in each direction talking with HP-UX, perhaps there is an interaction between the Linux TCP stack (TSO perhaps) and HP-UX's ACK avoidance heuristics. If that is the case, tweaking tcp_deferred_ack_max with ndd on the HP-UX system might yield different results. I don't recall if the igelan (broadcom) driver in HP-UX attempts to auto-tune the interrupt throttling. I do believe the iether (intel) driver in HP-UX does. That can be altered via lanadmin -X mumble... commands. Later (although later than a 2.6.18 kernel IIRC) e1000 drivers do try to auto-tune the interrupt throttling and one can see oscilations when an e1000 driver is talking to an e1000 driver. I think that can only be changed via the InterruptThrotleRate e1000 module parameter in that era of kernel - not sure if the Intel folks have that available via ethtool on contemporary kernels now or not. WRT the small program making a setsockopt(SO_*BUF) call going slower than the rsh, does rsh make the setsockopt() call, or does it bend itself to the will of the linux stack's autotuning? What happens if your small program does not make setsockopt(SO_*BUF) calls? Other misc observations of variable value: *) depending on the quantity of CPU around, and the type of test one is running, results can be better/worse depending on the CPU to which you bind the application. Latency tends to be best when running on the same core as takes interrupts from the NIC, bulk transfer can be better when running on a different core, although generally better when a different core on the same chip. These days the throughput stuff is more easily seen on 10G, but the netperf service demand changes are still visible on 1G. *) agreement with the observation that the small recv calls suggest that the application is staying-up with the network. I doubt that SO_&BUF settings would change that, but perhaps setting watermarks might (wild ass guess). The watermarks will do nothing on ...
I doubt it because I see the same sort of behaviour when I have a group of Linux clients talking to the server. The HP box is in the mix simply because it has a gigabit card and that makes driving the load simpler. But if I do several loads from 100Mbit clients I get the same I haven't tracked down if rsh does that but I've tried doing it with These are fast CPUs and they are running at 93% idle while running the test. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
I doubt it, the same test works fine in one direction and poorly in the other. Wouldn't the flow control squelch either way? -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
From: lm@bitmover.com (Larry McVoy) HW controls for these things are typically: 1) Generates flow control flames 2) Listens for them So you can have flow control operational in one direction and not the other. -
While I am often guilty of it, a wise old engineer tried to teach me that the proper spelling is ass-u-me :) I wouldn't count on it hitting in both directions, depends on the specifics of the situation. WRT the HP-UX ACK avoidance heuristic, the default HP-UX socket buffer/window is 32768, and tcp_deferred_ack_max defaults to 22. That isn't really all that good a combination - with a window of 32768 11 for the deferred ack would be better. You could also go ahead and try it with a value of 2. Or, bump the window size defaults - tcp_recv_hiwater_def and tcp_xmit_hiwater_def - to say 65535 or 128K or something - or use the setsockopt() calls to effect that. rick jones -
Tangential aside: Interesting. I was going to say that I've generally had the opposite experience when it comes to bulk data transfers, which is what I would expect due to CPU caching effects, but that perhaps it's motherboard/NIC/ driver dependent. But in testing I just did I discovered it's even MTU dependent (most of my normal testing is always with 9000-byte jumbo frames). With Myricom 10-GigE NICs, NIC interrupts on CPU 0 and nuttcp app running on CPU 1 (both transmit and receive sides), and using 9000-byte jumbo frames: [root@lang2 ~]# nuttcp -w10m 192.168.88.16 10078.5000 MB / 10.02 sec = 8437.5396 Mbps 100 %TX 99 %RX With Myricom 10-GigE NICs, and both NIC interrupts and nuttcp app on CPU 0 (both transmit and receive sides), again using 9000-byte jumbo frames: [root@lang2 ~]# nuttcp -w10m 192.168.88.16 11817.8750 MB / 10.00 sec = 9909.7537 Mbps 100 %TX 74 %RX Same tests repeated with standard 1500-byte Ethernet MTU: With Myricom 10-GigE NICs, NIC interrupts on CPU 0 and nuttcp app running on CPU 1 (both transmit and receive sides), and using standard 1500-byte Ethernet MTU: [root@lang2 ~]# nuttcp -M1460 -w10m 192.168.88.16 5685.9375 MB / 10.00 sec = 4768.0951 Mbps 99 %TX 98 %RX With Myricom 10-GigE NICs, and both NIC interrupts and nuttcp app on CPU 0 (both transmit and receive sides), again using standard 1500-byte Ethernet MTU: [root@lang2 ~]# nuttcp -M1460 -w10m 192.168.88.16 4974.0625 MB / 10.03 sec = 4161.6015 Mbps 100 %TX 100 %RX Now back to your regularly scheduled programming. :-) -Bill -
First of all check the CPU load on both sides to see if either of them is saturating. If the CPU's fine then look at the tcpdump output to see if both receivers are using the same window settings. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt -
tcpdump is a good idea, take a look at this. The window starts out at 46 and never opens up in my test case, but in the rsh case it starts out the same but does open up. Ideas? 08:08:06.033305 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: S 2756874880:2756874880(0) win 32768 <mss 1460,wscale 0,nop> 08:08:06.033335 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: S 3360532803:3360532803(0) ack 2756874881 win 5840 <mss 1460,nop,wscale 7> 08:08:06.047924 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 1 win 32768 08:08:06.048218 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 1:2921(2920) ack 1 win 46 08:08:06.048426 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 1461 win 32768 08:08:06.048446 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 2921:5841(2920) ack 1 win 46 08:08:06.048673 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 4381 win 32768 08:08:06.048684 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 5841:10221(4380) ack 1 win 46 08:08:06.049047 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 8761 win 32768 08:08:06.049057 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 10221:16061(5840) ack 1 win 46 08:08:06.049422 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 14601 win 32768 08:08:06.049429 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: P 16061:18981(2920) ack 1 win 46 08:08:06.049462 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 18981:20441(1460) ack 1 win 46 08:08:06.049484 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 20441:23361(2920) ack 1 win 46 08:08:06.049924 IP hp-ia64.bitmover.com.49614 > work-cluster.bitmover.com.31235: . ack 21901 win 32768 08:08:06.049943 IP work-cluster.bitmover.com.31235 > hp-ia64.bitmover.com.49614: . 23361:32121(8760) ack 1 win 46 08:08:06.050549 IP ...
Interesting data point. My test case is like this: server bind listen while (newsock = accept...) transfer() client connect transfer If the server side is the source of the data, i.e, it's transfer is a write loop, then I get the bad behaviour. If I switch them so the data flows in the other direction, then it works, I go from about 14K pkt/sec to 43K pkt/sec. Can anyone else reproduce this? I can extract the test case from lmbench so it is standalone but I suspect that any test case will do it. I'll try with the one that John sent. Yup, s/read/write/ and s/write/read/ in his two files at the appropriate places and I get exactly the same behaviour. So is this a bug or intentional? -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
For whatever it is worth, I believed that we used to get better performance from the same hardware. My guess is that it changed somewhere between 2.6.15-1-k7 and 2.6.18-5-k7. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
On Tue, 2 Oct 2007 09:25:34 -0700 For the period from 2.6.15 to 2.6.18, the kernel by default enabled TCP Appropriate Byte Counting. This caused bad performance on applications that did small writes. -- Stephen Hemminger <shemminger@linux-foundation.org> -
It's doing 1MB writes. Is there a sockopt to turn that off? Or /proc or something? -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
On Tue, 2 Oct 2007 09:49:52 -0700 sysctl -w net.ipv4.tcp_abc=0 -- Stephen Hemminger <shemminger@linux-foundation.org> -
On Mon, 15 Oct 2007 14:40:25 +0200 Read the RFC3465 for explanation of TCP ABC. What happens is that applications that do multiple small writes will end up using up their window. Typically these applications are not streaming enough data to grow the congestion window so they get held after 4 writes until an ACK comes back. The fix for the application (which also helps on all OS's and TCP versions as well) is to use a call like writev() or sendmsg() to aggregate the small header blocks together into a single send. -- Stephen Hemminger <shemminger@linux-foundation.org> -
I have a more complex configuration & application, but I don't see this problem in my testing. Using e1000 nics and modern hardware I can set up a connection between two machines and run 800+Mbps in both directions, or near line speed in one direction if the other direction is mostly silent. I am purposefully setting the socket send/rx buffers, as well has twiddling with the tcp and netdev related tunables. If you want, I can email these tweaks to you. NICs and busses have a huge impact on performance, so make sure those are good. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com -
Ben sent those to me, see below, they didn't make any difference. I tried diddling the socket send/recv buffers to 10MB, that didn't help. The defaults didn't help. 1MB didn't help and 64K didn't help. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
I'm currently on 2.6.20, and have also tried 10gbe nics on 2.6.23 with good results. At least for my app, performance has been pretty steady at least as far back as the .18 kernels, and probably before.... I do 64k or smaller writes & reads, and non-blocking IO (not sure if that would matter..but I do :) Have you tried something like ttcp, iperf, or even regular ftp? Checked your nics to make sure they have no errors and are negotiated to full duplex? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com -
> I'm currently on 2.6.20, and have also tried 10gbe nics on 2.6.23 with Yeah, I've factored out the code since BitKeeper, my test program, and John's test program all exhibit the same behaviour. Also switched Yup and yup. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
On Tue, 2 Oct 2007 10:21:55 -0700 Make sure you don't have slab debugging turned on. It kills performance. -- Stephen Hemminger <shemminger@linux-foundation.org> -
> Make sure you don't have slab debugging turned on. It kills performance. It's a stock debian kernel, so unless they turn it on it's off. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
Sounds like accept() possibly initializes slightly different socket parameters than connect() does. On the other hand, different network cards will simply have different behaviour (some due to hardware, some due to driver differences), so I hope you also switched the processes around and/or used identically configured machines (and the port configuration on switches could matter, of course, so it's really best to switch the processes around, to make sure that the *only* difference is whether the socket was set up by Sounds like a bug to me, modulo the above caveat of making sure that it's not some hw/driver/switch kind of difference. Linus -
Isn't this something so straightforward that you would have tests for it? This is the basic FTP server loop, doesn't someone have a big machine with Pretty unlikely given that we've changed the switch, the card works fine in the other direction, and I'm 95% sure that we used to get better perf before we switched to a more recent kernel. I'll try and find some other gig ether cards and try them. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
From: lm@bitmover.com (Larry McVoy) Nobody is really doing this, or they aren't talking about it. Sometimes the crash fixes and other work completely consumes us. Add in travel to conferences and real life, and it's no surprise stuff like this slips through the cracks. We absolutely depend upon people like you to report when there are anomalies like this. It's the only thing that scales. FWIW I have a t1000 Niagara box and an Ultra45 going through a netgear gigabit switch. I'm getting 85MB/sec in one direction and 10MB/sec in the other (using bw_tcp from lmbench3). Both are using identical broadcom tigon3 gigabit chips and identical current kernels so that is a truly strange result. I'll investigate, it may be the same thing you're seeing. -
Well cool, finally doing something useful :) Is this issue no test setup? Because this does seem like something we'd Note that bw_tcp mucks with SND/RCVBUF. It probably shouldn't, it's been Wow, sounds very similar. In my case I was seeing pretty close to 3x consistently. You're more like 8x, but I was all e1000 not broadcom. And note that sky2 doesn't have this problem. Does the broadcom do TSO? And sky2 not? I noticed a much higher CPU load for sky2. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
From: lm@bitmover.com (Larry McVoy) Yes the broadcoms (the revisions I have) do TSO and it is enabled on both sides. Which makes the mis-matched performance even stranger :) -
Stranger still, with a mix of a 2.6.23-rc5ish kernel and a net-2.6.24 one (pulled oh middle of last week?) I get link-rate and I see no asymmetry between TCP_STREAM and TCP_MAERTS over an "e1000" link with no switch or tg3 with a ProCurve on my rx2660's. I can also run bw_tcp from lmbench 3.0a8 and get 106 MB/s. I don't have a netgear switch to try in all this... rick jones -
From: Rick Jones <rick.jones2@hp.com> I'm starting to have a theory about what the bad case might be. A strong sender going to an even stronger receiver which can pull out packets into the process as fast as they arrive. This might be part of what keeps the receive window from growing. -
I can back you up on that. When I straced the receiving side that goes slowly, all the reads were short, like 1-2K. The way that works the reads were a lot larger as I recall. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
From: lm@bitmover.com (Larry McVoy) My issue turns out to be hardware specific too. The two Broadcom 5714 onboard NICs on my Niagara t1000 give bad packet receive performance for some reason, the other two which are Broadcom 5704's are perfectly fine. I'll figure out what the problem is, probably some misprogramed register in either the chip or the bridge it's behind. The UDP stream test of netperf is great for isolating TCP/TSO vs. hardware issues. If you can't saturate the pipe or the cpu with the UDP stream test, it's likely a hardware issue. The cpu utilization and service demand numbers provided, on both send and receive, are really useful for diagnosing problems like this. Rick deserves several beers for his work on this cool toy. :) -
Indeed I was getting more like 8K on each recv() call per netperf's -v 2 stats,
but the system was more than fast enough to stay ahead of the traffic. On the
hunch that it was the interrupt throttling which was keeping the recv's large
rather than the speed of the system(s) I nuked the InterruptThrottleRate to 0
and was able to get between 1900 and 2300 byte recvs on the TCP_STREAM and
TCP_MAERTS tests and still had 940 Mbit/s in each direction.
hpcpc106:~# netperf -H 192.168.7.107 -t TCP_STREAM -v 2 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.7.107
(192.168.7.107) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
87380 87380 87380 10.02 940.95 10.75 21.65 3.743 7.540
Alignment Offset Bytes Bytes Sends Bytes Recvs
Local Remote Local Remote Xfered Per Per
Send Recv Send Recv Send (avg) Recv (avg)
8 8 0 0 1.179e+09 87386.29 13491 1965.77 599729
Maximum
Segment
Size (bytes)
1448
hpcpc106:~# netperf -H 192.168.7.107 -t TCP_MAERTS -v 2 -c -C
TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.7.107
(192.168.7.107) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
87380 87380 87380 10.02 940.82 20.44 10.61 7.117 3.696
Alignment Offset Bytes Bytes Recvs Bytes Sends
Local Remote Local Remote Xfered Per ...(Binary tcpdumps are always better than ascii.) The window on the sender (linux box) starts at 46. It doesn't open up, but it's not receiving data so it doesn't matter, and you don't expect it to. The HP box always announces a window of 32768. Looks like you have TSO enabled. Does it behave differently if it's disabled? I think Rick Jones is on to something with the HP ack avoidance. Looks like a pretty low ack ratio, and it might not be interacting well with TSO, especially at such a small window size. -John -
It cranks the interrupts/sec up to 8K instead of 5K. No difference in I sincerely doubt it. I'm only using the HP box because it has gigabit so it's a single connection. I can produce almost identical results by doing the same sorts of tests with several linux clients. One direction goes fast and the other goes slow. 3x performance difference depending on the direction of data flow: # Server is receiving, goes fast $ for i in 22 24 25 26; do rsh -n glibc$i dd if=/dev/zero|dd of=/dev/null & done load free cach swap pgin pgou dk0 dk1 dk2 dk3 ipkt opkt int ctx usr sys idl 0.98 0 0 0 0 0 0 0 0 0 30K 15K 8.1K 68K 12 66 22 0.98 0 0 0 0 0 0 0 0 0 29K 15K 8.2K 67K 11 64 25 0.98 0 0 0 0 0 0 0 0 0 29K 15K 8.2K 67K 12 66 22 # Server is sending, goes slow $ for i in 22 24 25 26; do dd if=/dev/zero|rsh glibc$i dd of=/dev/null & done load free cach swap pgin pgou dk0 dk1 dk2 dk3 ipkt opkt int ctx usr sys idl 1.06 0 0 0 0 0 0 0 0 0 5.0K 10K 4.4K 8.4K 21 17 62 0.97 0 0 0 0 0 0 0 0 0 5.1K 10K 4.4K 8.9K 2 15 83 0.97 0 0 0 0 0 0 0 0 0 5.0K 10K 4.4K 8.6K 21 26 53 $ for i in 22 24 25 26; do rsh glibc$i cat /etc/motd; done | grep Welcome Welcome to redhat71.bitmover.com, a 2Ghz Athlon running Red Hat 7.1. Welcome to glibc24.bitmover.com, a 1.2Ghz Athlon running SUSE 10.1. Welcome to glibc25.bitmover.com, a 2Ghz Athlon running Fedora Core 6 Welcome to glibc26.bitmover.com, a 2Ghz Athlon running Fedora Core 7 $ for i in 22 24 25 26; do rsh glibc$i uname -r; done 2.4.2-2 2.6.16.13-4-default 2.6.18-1.2798.fc6 2.6.22.4-65.fc7 No HP in the mix. It's got nothing to do with hp, nor to do with rsh, it has everything to do with the direction the data is flowing. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
More data, we've conclusively eliminated the card / cpu from the mix. We've got 2 ia64 boxes with e1000 interfaces. One box is running linux 2.6.12 and the other is running hpux 11. I made sure the linux one was running at gigabit and reran the tests from the linux/ia64 <=> hp/ia64. Same results, when linux sends it is slow, when it receives it is fast. And note carefully: we've removed hpux from the equation, we can do the same tests from linux to multiple linux clients and see the same thing, sending from the server is slow, receiving on the server is fast. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
I think I'm still missing some basic data here (probably because this thread did not originate on netdev). Let me try to nail down some of the basics. You have a linux ia64 box (running 2.6.12 or 2.6.18?) that sends slowly, and receives faster, but not quite a 1 Gbps? And this is true regardless of which peer it sends or receives from? And the behavior is different depending on which kernel? How, and which kernel versions? Do you have other hardware running the same kernel that behaves the same or differently? Have you done ethernet cable tests? Have you tried measuring the udp sending rate? (Iperf can do this.) Are there any error counters on the interface? -John -
just got off the phone with Linus and he thinks it is the side that does the accept is the problem side, i.e., if you are the server, you do the accept, and you send the data, you'll go slow. But as I'm writing this I realize he's wrong, because it is the combination of accept & send. accept & recv goes fast. A trivial way to see the problem is to take two linux boxes, on each apt-get install rsh-client rsh-server set up your .rhosts, and then do dd if=/dev/zero count=100000 | rsh OTHER_BOX dd of=/dev/null rsh OTHER_BOX dd if=/dev/zero count=100000 | dd of=/dev/null See if you get balanced results. For me, I get 45MB/sec one way, and 15-19MB/sec the other way. I've tried the same test linux - linux and linux - hpux. Same results. The test setup I have is work: 2ghz x 2 Athlons, e1000, 2.6.18 ia64: 900mhz Itanium, e1000, 2.6.12 hp-ia64:900mhz Itanium, e1000, hpux 11 glibc*: 1-2ghz athlons running various linux releases all connected through a netgear 724T 10/100/1000 switch (a linksys showed identical results). I tested work <-> hp-ia64 work <-> ia64 ia64 <-> hp-ia64 and in all cases, one direction worked fast and the other didn't. It would be good if people tried the same simple test. You have to use rsh, ssh will slow things down way too much. Alternatively, take your favorite test programs, such as John's, and make a second pair that reverses the direction the data is sent. So one pair is server sends, the other is server receives, try both. That's where we started, BitKeeper, my stripped down test, and John's test all exhibit the same behavior. And the rsh test is just a really simple way to demonstrate it. Wayne, Linus asked for tcp dumps from just one side, with the first 100 packets and then wait 10 seconds or so for the window to open up, and then a snap shot of the another 100 packets. Do that for both directions and send them to the list. Can you do that? I want to get lunch, I'm starving. -- --- Larry McVoy ...
OK attached are 4 raw tcpdumps of 1000 packets each. One from the start and one from steady state. The slow set was done like this: on ia64: netcat -l -p8888 > /dev/null on work: netcat ia64 8888 < /dev/zero the traces were done of work with slow1 started right before the netcat on work was exectued. And slow2 started after it acheved steady state at 18MB/s. The fast set was done like this: on work: netcat -l -p8888 > /dev/null on ia64: netcat ia64 8888 < /dev/zero the traces were done of work with fast1 started right before the netcat on ia64 was exectued. And fast2 started after it acheved steady state at 42MB/s. -Wayne
That sounds wrong. Larry claims the slow case is when the side that did This one is guaranteed wrong too, since you have the listener reading (fine), but the sener now doesn't go over the network at all, but sends to itself. That said, let's assume that only your description was bogus, the TCP dumps themselves are ok. I find the window scaling differences interesting. This is the opening of the fast sequence from the receiver: 13:35:13.929349 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: S 2592471184:2592471184(0) ack 3363219397 win 5792 <mss 1460,sackOK,timestamp 174966955 3714830794,nop,wscale 7> 13:35:13.929702 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 1449 win 68 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929712 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 2897 win 91 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929724 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 4345 win 114 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929941 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 5793 win 136 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929951 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 7241 win 159 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929960 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 8689 win 181 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929970 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 10137 win 204 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929981 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 11585 win 227 <nop,nop,timestamp 174966955 3714830795> 13:35:13.929992 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 13033 win 249 <nop,nop,timestamp 174966955 3714830795> 13:35:13.930331 IP 10.3.1.1.ddi-tcp-1 > 10.3.1.10.58415: . ack 14481 win 272 <nop,nop,timestamp 174966955 3714830795> ... ie we use a window scale of 7, and we started with a window of 5792 bytes, and after ten packets it has grown to 272<<7 (34816) bytes. The slow case is 13:34:16.761034 IP 10.3.1.10.ddi-tcp-1 > ...
Netperf TCP_STREAM - server receives. TCP_MAERTS (STREAM backwards) - server sends: [root@hpcpc106 ~]# netperf -H 192.168.2.107 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.107 (192.168.2.107) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 87380 87380 10.17 941.46 [root@hpcpc106 ~]# netperf -H 192.168.2.107 -t TCP_MAERTS TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.107 (192.168.2.107) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 87380 87380 10.15 941.35 The above took all the defaults for socket buffers and such. [root@hpcpc106 ~]# uname -a Linux hpcpc106.cup.hp.com 2.6.18-8.el5 #1 SMP Fri Jan 26 14:16:09 EST 2007 ia64 ia64 ia64 GNU/Linux [root@hpcpc106 ~]# ethtool -i eth2 driver: e1000 version: 7.2.7-k2-NAPI firmware-version: N/A bus-info: 0000:06:01.0 between a pair of 1.6 GHz itanium2 montecito rx2660's with a dual-port HP A9900A (Intel 82546GB) in slot 3 of the io cage on each. Connection is actually back-to-back rather than through a switch. I'm afraid I've nothing older installed. sysctl settings attached Where I do have things connected via a switch (HP ProCurve 3500 IIRC, perhaps a 2724) is through the core BCM5704: [root@hpcpc106 netperf2_work]# netperf -H hpcpc107 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to hpcpc107.cup.hp.com (16.89.84.107) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 87380 87380 10.03 941.41 [root@hpcpc106 netperf2_work]# netperf -H hpcpc107 -t TCP_MAERTS TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to hpcpc107.cup.hp.com (16.89.84.107) ...
Can you tcpdump both cases and send snippets (both of steady-state, and the initial connect)? Linus -
From: Linus Torvalds <torvalds@linux-foundation.org> Another thing I'd like to see is if something more recent than 2.6.18 also reproduces the problem. It could be just some bug we've fixed in the past year :) -
I don't think that's an issue, since you only send one way. The window opening up only matters for the receiver. Also, you missed the "wscale=7" at the beginning, so the window of "46" looks like it actually is 5888 (ie fits four segments - and it's not grown because it never gets any data). However, I think this is some strange TSO artifact: ... We see a single packet containing 16060 bytes, which seems to be because of TSO on the sending side (you did your tcpdump on the sender, no?), so it will actually be broken up into 11 1460-byte regular frames by the network card, since they started out agreeing on a standard 1460-byte MSS. So the above is not a jumbo frame, it just kind of looks like one when you capture it on the sender side. And maybe a 32kB window is not big enough when it causes the networking code to basically just have a single packet outstanding. I also would have expected more ACK's from the HP box. It's been a long time since I did TCP, but I thought the rule was still that you were supposed to ACK at least every other full frame - but the HP box is acking roughly every 16K (and it's *not* always at TSO boundaries: the earlier ACK's in the sequence are at 1460-byte packet boundaries, but it does seem to end up getting into that pattern later on). So I'm wondering if we get into some bad pattern with the networking code trying to make big TSO packets for e1000, but because they are *so* big that there's only room for two such packets per window, you don't get into any smooth pattern with lots of outstanding packets, but it starts stuttering. Larry, try turning off TSO. Or rather, make the kernel use a smaller limit for the large packets. The easiest way to do that should be to just change the value in /proc/sys/net/ipv4/tcp_tso_win_divisor. It defaults to 3, try doing echo 6 > /proc/sys/net/ipv4/tcp_tso_win_divisor and see if that changes anything. And maybe I'm just whistling in the dark. In fact, it looks like ...
From: Linus Torvalds <torvalds@linux-foundation.org> We fixed a lot of bugs in TSO last year. It would be really great to see numbers with a more recent kernel than 2.6.18 -
More data, sky2 works fine (really really fine, like 79MB/sec) between Linux dylan.bitmover.com 2.6.18.1 #5 SMP Mon Oct 23 17:36:00 PDT 2006 i686 Linux steele 2.6.20-16-generic #2 SMP Sun Sep 23 18:31:23 UTC 2007 x86_64 So this is looking like a e1000 bug. I'll try to upgrade the kernel on the ia64 box and see what happens. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
A few notes to the discussion. I've seen one e1000 "bug" that ended up being a crappy AMD pre-opteron SMP chipset with a totally useless PCI bus implementation, which limited performance quite a bit-totally depending on what you plugged in and in which slot. 10e milk-and-bread-store 32/33 gige nics actually were better than server-class e1000's in those, but weren't that great either. A few things worth trying out is using recv(.., MSG_TRUNC ) on the receiver, that tests the theoretical sender maximum performance much better (but memory bandwidth vs. GigE is much higher these days than it was in 2001 so maybe not that useful anymore). Check your interrupt rates for the interface. You shouldn't be getting anywhere near 1 interrupt/packet. If you are, something is badly wrong :). Running getsockopt(...TCP_INFO) every few secs on the socket and printing that out can be useful too. That gives you both sides' idea on what the tcp windows etc. are. My favourite tool is a home-made thing called yantt btw. ( http://www.ee.oulu.fi/~pp/yantt.tgz . Needs lots of cleanup love, it mucks with the window sizes by default, since in the 2.4 days you really had to do that to get any kind of performance and the help text is wrong. But it's pretty easy to hack to try out new ideas, use sendfile/MSG_TRUNC/TCP_INFO etc. Netperf is the kitchen sink of network benchmark tools. But trying out a few tiny things with it is not fun at all, I tried and quickly decided to write my own tool for my master's thesis work ;-) Oh. Don't measure CPU usage with top. Use a cyclesoaker (google for cyclesoak, I included akpm's with yantt) :-) And yes. TCP stacks do have bugs, especially when things get outside the equipment most people have. Having a dedicated transatlantic 2.5Gbps connection found a really fun one a long time ago ;) -- Pekka Pietikainen -
That could well be my problem, this is a dual processor (not core) athlon The acks (because I'm sending) are about 1.5 packets/interrupt. When this box is receiving it's moving about 3x ass much data and has a _lower_ (absolute, not per packet) interrupt load. -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com -
If it's AMD760/768MPX, here's some relevant discussion: http://lkml.org/lkml/2002/7/18/292 http://www.ussg.iu.edu/hypermail/linux/kernel/0307.1/1109.html http://www.ussg.iu.edu/hypermail/linux/kernel/0307.1/1154.html http://www.ussg.iu.edu/hypermail/linux/kernel/0307.1/1212.html Probably not a problem then, since those acks probably cover many sent packets. Current interrupt mitigation schemes are pretty dynamic, balancing between latency and bulk performance so the acks might be fine (thousands vs. tens of thousands/sec) -- Pekka Pietikainen -
> It would be really great to see numbers with a more recent kernel > than 2.6.18 FWIW Debian has binaries for 2.6.21 in testing and for 2.6.22 in unstable so it should be very easy for Larry to try at least those. - R. -
Drift... The RFC's say "SHOULD" (emphasis theirs) rather than "MUST." Both HP-UX and Solaris have rather robust ACK avoidance heuristics to cut-down on the CPU overhead of bulk transfers. (That they both have them stems from their being cousins, sharing a common TCP stack ancestor long ago - both of course have been diverging since then). rick jones -
