On Wed, Feb 21, 2007 at 05:57:41PM -0800, jonathan@dsg.stanford.edu wrote:
quoted text
>=20
> In message <20070221234224.GF26468@netbsd.org>,
> Bill Studenmund writes:
>=20
> >On Wed, Feb 21, 2007 at 10:51:03PM +0000, Andrew Doran wrote:
>=20
> >> What I want to do will on x86 add 29 arithmetic instructions to the
> >> interrupt path (as of now). That's means in the common, non blocking c=
ase,
quoted text
> >> it's ~free. I'd be pretty bummed if we undo some of that and the reaso=
ning
quoted text
> >> for doing so is to make the system a better fit for ~20 year old syste=
ms.
quoted text
>=20
> Andrew, just what will trigger the blocking (expensive) case?
> And how  expensive is it?

I'm not Andy, but as I understand it (and I mention this so I'll be=20
corrected if I'm wrong), the blocking case would be where another CPU=20
holds the mutex you need to acess the hardware. i.e. CPU 3 is configuring=
=20
a piece of hardware then CPU 2 receives an interrupt from that device.

quoted text
> Can I offer the following example? Suppose for the sake of discussion,
> that I'm running IPsec traffic with FAST_IPSEC, and handling a modest
> 50,000 packets per second.  Suppose further that there's also always
> userland processes ready to run.
>=20
> My first guess (perhaps overly naive) on how thread-per-interrupt
> would work is something like this:
>=20
> 	(...new IPsec'd packet comes in, asserts NIC interrupt)
> 1. switch from user to NIC interrupt thread
> 	(...  NIC calls ether_input which demuxes packet, enqueues on
> 	      protocol input routine, requests softint processing, blocks)
> 2. switch from NIC hardware thread to softint thread
> 	(... OCF submits job, blocks ...)
> 3. switch from softint to user=09
> 	(... crypto hardware finishes, requests interrupt...)
> 4.  switch from user to crypto-interrupt thread
> 	(... crypto driver calls OCF which wakes up softint processing...)
> 5.	switch to softint thread, process cleartext packet
> 	(... done with local  kernel packet processing, softint thread  ...)
> 6.	switch back to user.=20
>=20
> [[Let's ignore, for now, whether there'd be a single softint thread,
> or a separate thread for sofnet, or multiple softnet threads.]]
>=20
> Is the above roughly right? If it is, then (to steal a comment I made
> to Sam Leffler early in FAST_IPSEC development): 300,000 context
> switches/sec are _not_ your friend.  And even if the NIC->softint and
> crypto->softint are a same-VM-context switch (and thus cheap, or at
> least optimizable as kthread-to-kthread), the cost of rescheduling is
> high relative to the work done per packet.  And remember, the cost is
> not just instructions, but the additional cache-footprint (and thus
> evictions) of all these switches.  Ditto for TLB state; doubly so for
> arches with software-filled TLBs.
>=20
> So I assume I'm missing something; can you clue me in?

I don't think that's a function of interrupts-as-threads. I think our=20
current code will do the same thing now, as will anything where one packet=
=20
=3D=3D one interrupt.

I think this is a reason for interrupt mitigation in NICs and, if=20
possible, in crypto cards. And/or for doing something completely different=
=20
for handling packet reception, as Jason hinted at.

quoted text
> >Ok, that sounds good. How do we make this work on SPARC, m68k, and VAX?
> >Right now, interrupts-as-threads sounds like a total loss for them. If we
> >can turn it into something that's "about as good" then I think you will
> >just get a green light. :-)
>=20
> Bill, you mean "about as good as NetBSD is now", right?=20

Exactly.

Take care,

Bill