Well, I'm not sure what experiment you are refering to but I'll assume
its the network threading, which works quite well actually. The protocol
threads can be matched against the toeplitz function and in that case
the entire packet stream operates lockless. Even without the matching
we still get good benefits from batching (e.g. via ether_input_chain())
which drops the IPI and per-packet switch overhead basically to zero.
We have other issues but the protocol threads aren't one of them.
In anycase, the lesson to learn with batching to a thread is that you
don't want the thread to immediately preempt the sender (if it happens
to be on the same cpu), or to generate an instant IPI (if going between
cpus). This creates a degenerate case where you wind up with a
thread switch on each message or an excessive messaging interrupt
rate... THAT is what seriously screws up performance. The key is to
be able to batch multiple messages per thread switch when under load
and to be able to maintain a pipeline.
A single user-process test case will always have a bit more latency
and can wind up being inefficient for a variety of other reasons
(e.g. whether the target thread is on the same cpu or not),
but that becomes less relevant when the machine is under load so
its a self-correcting problem for the most part.
Once the machine is under load batching becomes highly efficient.
That is, latency != cpu cycle cost under load. When the threads
have enough work to do they can pick up the next message without the
cost of entering a sleep state or needing a wakeup (or needing to
generate an actual IPI interrupt, etc). Plus you can run lockless
and you get excellent cache locality. So as long as you ensure these
optimal operations become the norm under load you win.
Getting the threads to pipeline properly and avoid unnecessary
tsleeps and wakeups is the hard part.
...
