After a great of pondering, I've concluded that interrupt threads are  
an extremely bad idea.

I think that hard interrupts should simply invoke the handler (at the  
appropriate IPL), the handler will disable the cause of the  
interrupt, optionally it may use a SPIN mutex to gain control over a  
shared section, do something to the device, release the mutex, or it  
may just schedule a continuation to run (either via software  
interrupt or via a kernel lwp workqueue, the former can't sleep/ 
block, the latter can).

While IPL will need to continue to exist for talking to hardware, I  
think software priority levels will eventually disappear as they  
currently exist.  They will transition to actual real-time priorities  
used by the scheduler to run kernel lwps (< SPL_SCHED) or become  
mutexes (>= SPL_SCHED).