On Thu, Feb 22, 2007 at 04:49:40PM -0800, jonathan@dsg.stanford.edu wrote:
quoted text
> In message <20070222174642.GB24922@netbsd.org>, Bill Studenmund writes:
>=20
> [...]
>=20
> >I think the problem is you've assumed an implementation, and specifically
> >you've assumed one other than what Andy was suggesting.
>=20
> I came to that conclusion after re-reading Andrew's reply.  i think
> the problem is terminology.  I'm not used to seeing taking an
> interrupt called a "context switch" instead of, well, taking an
> interrupt or a trap.

Well, it is. :-) It is one form of context switch, and all of the=20
context-switchy things it does are a main part of its expense.

quoted text
> >My understanding is that Andy has figured out a way to have, at least on
> >x86, the interrupt handler borrow the context of the interrupted thread.
> >So the interrupt context switch is also the context switch to the thread.
> >
> >That's why he said it was the same as what we do now.
>=20
> That's an unfortunate choice of terminology. it'S inviting confusion
> to say that a new approach does just what we did before, when it
> doesn't, and the details matter.

True.

For the common case, we will have almost the same behavior as now, except=
=20
that we can take mutexes in an interrupt handler.

As I understand it, the difficulty comes in when the mutex is held by a=20
thread that is not running. In that case, the interrupt handler blocks. In=
=20
that case, the interrupt handler has to be blocked, and the interrupt has=
=20
to be disabled/ignored until serviced. For systems with a PIC, we disable=
=20
the interrupt and cope.

For systems without a PIC, like many m68k systems (and VAX as I understand=
=20
matt), we would need to disable said interrupt in the CPU (set the chip's=
=20
interrupt level) until the service routine completes.

As noted, this will hopefully be a rare occurrence, where a thread that=20
holds an interrupt mutex is no longer on a processor. We can structure our=
=20
code so that this is very unlikely, if not impossible.

Note that in the case of the mutex being locked but the thread being on=20
another processor, we will just spin-wait. That's a feature of the locking=
=20
we copied from Solaris. I expect that this _will_ happen, but we then get=
=20
the exact same behaviors as if we had spinlocks.

quoted text
> >Let's try it and see! Rather than discuss hypotheticals, let's get #s.
> >Either the numbers will show little change, which will make ALL of us
> >happy, or the numbers will show a marked decrease, which will displease
> >ALL of us. :-)
>=20
> Numbers would be interesting.  But what kind of numbers?  Single-CPU?
> Multi-CPU?  Or CPUs with (gack) virtually-indexed caches and TLBs
> without address-space-IDs?  I don't see how Andrew's scheme would
> really work there, but maybe I'm missing something.

Single ande multi-CPU to be sure. But as for the other stuff, it's mostly=
=20
what we have now. So I don't see how we will have radially-different=20
results.

We should test stuff over time to make sure we don't do something stupid,=
=20
to be sure!

quoted text
> The other issue I see is that Andrew is assuming that blocking (in his
> sense, meaning doing the full work of the deferred context switch) is
> a rare case.  I don't buy that, not for networking.  Look at the
> resources put into making FreeBSD a fine-grained kernel, and notice
> that FreeBSD-6 still has one big lock around the network stack and
> socket-layer code.

I don't think that will mater for interupt handlers. Yes, I think there's=
=20
a lot of work to do for the networking stack. But I think it'll be=20
different work.

As above, trying to get the lock while a thread running on another CPU has=
=20
it turns into a spin wait. That's not the slow case. So as long as we=20
don't hold a mutex we take in interrupt context while we do something else=
=20
that can block, we will NOT trigger the slow path in an interrupt handler.=
=20
We don't sleep while holding SPL now (or we aren't supposed to), so a very=
=20
direct 1:1 translation should be fine.

We can run into issues, of course, and we need to look for them. Like a
case where the process that we interrupted holds a mutex that the thread
that holds the interrupt mutex is waiting for. That's a deadlock if we
don't detect it & trigger slow-path. It can also be argued that that's a
bad design. :-)

While I think we need to think about some architectures more, I believe=20
this is a good direction to go in.

Also, I remember you wanting to work on MP-ing the network stack. newlock2=
=20
+ interrupt handling will be a definite springboard for that!

Take care,

Bill