In message <20070221234224.GF26468@netbsd.org>,
Bill Studenmund writes:
Andrew, just what will trigger the blocking (expensive) case?
And how expensive is it?
Can I offer the following example? Suppose for the sake of discussion,
that I'm running IPsec traffic with FAST_IPSEC, and handling a modest
50,000 packets per second. Suppose further that there's also always
userland processes ready to run.
My first guess (perhaps overly naive) on how thread-per-interrupt
would work is something like this:
(...new IPsec'd packet comes in, asserts NIC interrupt)
1. switch from user to NIC interrupt thread
(... NIC calls ether_input which demuxes packet, enqueues on
protocol input routine, requests softint processing, blocks)
2. switch from NIC hardware thread to softint thread
(... OCF submits job, blocks ...)
3. switch from softint to user
(... crypto hardware finishes, requests interrupt...)
4. switch from user to crypto-interrupt thread
(... crypto driver calls OCF which wakes up softint processing...)
5. switch to softint thread, process cleartext packet
(... done with local kernel packet processing, softint thread ...)
6. switch back to user.
[[Let's ignore, for now, whether there'd be a single softint thread,
or a separate thread for sofnet, or multiple softnet threads.]]
Is the above roughly right? If it is, then (to steal a comment I made
to Sam Leffler early in FAST_IPSEC development): 300,000 context
switches/sec are _not_ your friend. And even if the NIC->softint and
crypto->softint are a same-VM-context switch (and thus cheap, or at
least optimizable as kthread-to-kthread), the cost of rescheduling is
high relative to the work done per packet. And remember, the cost is
not just instructions, but the additional cache-footprint (and thus
evictions) of all these switches. Ditto for TLB state; doubly so for
arches with software-filled TLBs.
So I assume I'm missing something; can you clue me in?
Bill, you mean "about as good as NetBSD is now", right?
