The sendfile() facility allows a regular file to be sent out to a stream socket. The system call was first implemented in FreeBSD 3.0. This provides many performance benefits for various server appliances. Andre Opperman has implemented an improved sendfile() facility for FreeBSD that has so far shown 45% less CPU usage without TCP segmentation offload and 83% less CPU usage with TCP segmentation offload. This is a great improvement over the previous implementation.

From:   Andre Oppermann [email blocked]
To:  	freebsd-current, freebsd-net

The recent addition of TSO (TCP Segmentation Offload) has highlighted some shortcommings in our 
sendfile(2) kernel implementation.  The current code simply loops over the file, turns each 4K 
page into an mbuf and sends it off.  This has the effect that TSO can only generate 2 packets 
per send instead of up to 44 at its maximum of 64K.

I have rewritten kern_sendfile() to work in two loops, the inner which turns as many pages into 
mbufs as it can up to the free send socket buffer space. The outer loop then drops the whole 
mbuf chain into the send socket buffer, calls tcp_output() on it and then waits until 50% of the 
socket buffer are free again to repeat the cycle.  This way tcp_output() gets the full amount of 
data to work with and can issue up to 64K sends for TSO to chop up in the network adapter without 
using any CPU cycles.  Thus it gets very efficient especially with the readahead the VM and I/O 
system do.

Looking at the benchmarks we see some very nice improvements (95% confidence):
45% less cpu (or 1.81 times better) with new sendfile vs. old sendfile (non-TSO)

83% less cpu (or 5.7 times better) with new sendfile vs. old sendfile (TSO)

The sender is an AMD Opteron 852 (2.6GHz) with em(4) PCI-X-133 interface and the receiver is a 
DELL Poweredge SC1425 P-IV Xeon 3.2GHz with em(4) LOM connected back to back at 1000Base-TX 
full duplex.

The patch is available here:
 http://people.freebsd.org/~andre/sendfile-20060920.diff

Any testing and heavy (code) beating and reviews welcome.