In response to a recent bug report on the lkml, Jeff Garzik, the maintainer of the Serial ATA subsystem [story] and network device drivers, had some harsh words for kernel preemption. In the bug report it was mentioned that kernel preemption was enabled to which Jeff replied, "I strongly recommend disabling kernel preemption. It is a hack that hides bugs". The comment lead into an interesting discussion.

Jeff's two main points were that kernel preemption potentially hides long latency code paths that should instead be fixed, and that it potentially introduces bugs. Andrea Arcangeli [interview], who back in March also had some sharp criticisms on the usefulness of kernel preemption [story] replied to the first point, "with any decent profiler, places where we're wasting cpus will show-up immediately. [...] So I disagree with your claim that preempt risks to hide inefficient code, there are many other (probably easier) ways to detect inefficient code than to check the latencies." 2.6 maintainer Andrew Morton [interview] replied to the second point, "what driver bugs are apparent with preemption which are not already SMP bugs? [The] only thing I can think of is unguarded use of per-cpu data, and we have runtime debug checks for that now."

From: Jeff Garzik [email blocked]
Subject: Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA)
Date: 	Tue, 05 Oct 2004 20:54:14 -0400

Gianluca Cecchi wrote:
> Eventually I can provide config. Btw, I'm using udev and kernel preemption.

I strongly recommend disabling kernel preemption.  It is a hack that 
hides bugs.



> ata1: dev 0 ATA, max UDMA/133, 240121728 sectors:
> ata1: dev 0 configured for UDMA/100
> and the same for ata2?

The controller maximum is set to UDMA/100.



> But taking in parallel top with refresh 1s in another window, during the
> 80
> seconds elapsed above, I had two freezes in top session: one of 20sec and
> another
> of 15 sec: about the half of the time of the total I/O operation!
> During this time there was one cpu used at about 30%, no other cpu constraints.
> I repeated the operation with similar behaviour. Sometimes also 50 contiguos
> secs
> of freeze, until I/O operation finished.
> The freeze is not about all system itself, but only certian things.
> For example I tried also a vmstat session in another window with 3secs of
> delay
> and it had no problems while top was blocked:

Make sure you have the latest BIOS update for your system, and latest 
BIOS update for your card.

	Jeff



From: Roland Dreier [email blocked]
Subject: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 18:00:16 -0700
 Obscurity, linux)

    Jeff> I strongly recommend disabling kernel preemption.  It is a
    Jeff> hack that hides bugs.

Why do you say that?  Preempt seems to be the cleanest way to low
latency, and if anything it exposes locking bugs and races rather than
hiding anything.

Thanks,
  Roland



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 21:11:59 -0400

Roland Dreier wrote:
>     Jeff> I strongly recommend disabling kernel preemption.  It is a
>     Jeff> hack that hides bugs.
> 
> Why do you say that?  Preempt seems to be the cleanest way to low
> latency, and if anything it exposes locking bugs and races rather than
> hiding anything.

Clean?  Hardly.  It breaks up code paths that were never written to be 
broken up.  The proper fix is to locate and fix high latency code paths. 
  Preempt does nothing but hide those high latency code paths, and 
discourage people from seeking a better solution.

Fix the drivers, rather than bandaid over them with preempt.

If all code paths in the kernel were low latency, then you would not 
need preempt at all.

	Jeff



From: Nick Piggin [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 11:28:20 +1000

Jeff Garzik wrote:
> Roland Dreier wrote:
> 
> Clean?  Hardly.  It breaks up code paths that were never written to be 
> broken up.

But lots of things change. Unserialising the kernel broke code paths.

>  The proper fix is to locate and fix high latency code paths. 
>  Preempt does nothing but hide those high latency code paths, and 
> discourage people from seeking a better solution.
> 
> Fix the drivers, rather than bandaid over them with preempt.
> 
> If all code paths in the kernel were low latency, then you would not 
> need preempt at all.
> 

When you say low latency, you mean small lock hold times, *and*
cond_rescheds placed everywhere - it is this second requirement
that isn't the cleanest way of doign things.

With preempt, sure you still need small lock hold times. No big
deal.




From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 21:40:03 -0400

Nick Piggin wrote:
> When you say low latency, you mean small lock hold times, *and*
> cond_rescheds placed everywhere - it is this second requirement
> that isn't the cleanest way of doign things.
> 
> With preempt, sure you still need small lock hold times. No big
> deal.

And with preempt you're still hiding stuff that needs fixing.  And when 
it gets fixed, you don't need preempt.

Therefore, preempt is just a hack that hides stuff that wants fixing anyway.

	Jeff



From: Robert Love [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 21:52:55 -0400

On Tue, 2004-10-05 at 21:40 -0400, Jeff Garzik wrote:

> And with preempt you're still hiding stuff that needs fixing.  And when 
> it gets fixed, you don't need preempt.
> 
> Therefore, preempt is just a hack that hides stuff that wants fixing anyway.

This actually sounds like the argument for preempt, and against
sprinkling cond_resched() hacks all over the kernel.

	Robert Love



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 5 Oct 2004 21:55:15 -0400

On Tue, Oct 05, 2004 at 09:52:55PM -0400, Robert Love wrote:
> On Tue, 2004-10-05 at 21:40 -0400, Jeff Garzik wrote:
> 
> This actually sounds like the argument for preempt, and against

As opposed to fixing drivers???  Please fix the drivers and code first.


> sprinkling cond_resched() hacks all over the kernel.

cond_resched() is not the only solution.

	Jeff



From: Robert Love [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 22:07:39 -0400

On Tue, 2004-10-05 at 21:55 -0400, Jeff Garzik wrote:

> As opposed to fixing drivers???  Please fix the drivers and code first.

No, definitely not, dude.  Fixes for anything--drivers include--is never
superseded by anything else, even the eternal quest for "low latency." 

> > sprinkling cond_resched() hacks all over the kernel.
> 
> cond_resched() is not the only solution.

Indeed.  Most other solutions (fixing algorithms, lowering lock hold
time) have "automatic" benefits with kernel preemption, though, and that
has been what I have always advocated.

	Robert Love



From: Nick Piggin [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 12:02:48 +1000

Jeff Garzik wrote:
> On Tue, Oct 05, 2004 at 09:52:55PM -0400, Robert Love wrote:
> 
>>On Tue, 2004-10-05 at 21:40 -0400, Jeff Garzik wrote:
>>
>>
>>>And with preempt you're still hiding stuff that needs fixing.  And when 
>>>it gets fixed, you don't need preempt.
>>>
>>>Therefore, preempt is just a hack that hides stuff that wants fixing anyway.

What is it hiding exactly?

>>
>>This actually sounds like the argument for preempt, and against
> 
> 
> As opposed to fixing drivers???  Please fix the drivers and code first.
> 

I thought you just said preempt should be turned off because it
breaks things (ie. as opposed to fixing the things that it breaks).

But anyway, yeah obviously fixing drivers always == good. I don't
think anybody advocated otherwise.



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 5 Oct 2004 22:07:34 -0400

On Wed, Oct 06, 2004 at 12:02:48PM +1000, Nick Piggin wrote:
> 
> What is it hiding exactly?

Bugs and high latency code paths that should instead be fixed.


> 
> But anyway, yeah obviously fixing drivers always == good. I don't
> think anybody advocated otherwise.

By _definition_, when you turn on preempt, you hide the stuff I just
described above.

Hiding that stuff means that users and developers won't see code paths
that need fixing.  If users and developers aren't aware of code paths
that need fixing, they don't get fixed.

Therefore, by advocating preempt, you are advocating a solution _other
than_ actually making the necessary fixes.

	Jeff



From: Andrea Arcangeli [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 6 Oct 2004 05:17:26 +0200

Hello,

On Tue, Oct 05, 2004 at 10:07:34PM -0400, Jeff Garzik wrote:
> Hiding that stuff means that users and developers won't see code paths
> that need fixing.  If users and developers aren't aware of code paths

with any decent profiler places where we're wasting cpus will showup
immediatatly, one obvious example that I can recall is the pte_chains
rmap code, that wasn't necessairly high latency but still it was very
wasteful in terms of cpu, was always at the top of the profiling.

So I disagree with your claim that preempt risks to hide inefficient
code, there are many other (probably easier) ways to detect inefficient
code than to check the latencies.

The one argument I've against preempt is that the claim that preempt
doesn't spread cond_resched all over the place is false. It can spread
even more of them as implicit ones. They're not visible to the developer
but they're visible to the cpu. So disabling preempt and putting
finegriend cond_resched should allow us to optimize the code better, and
actually _reduce_ the number of cond_resched (cond_resched as the ones
visible to the cpu, not the ones visible to the kernel developer).

I wonder if anybody ever counted the number of implicit cond_resched
placed by preempt and compared them to the number of explicit
cond_resched needed without preempt.



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 23:27:06 -0400

Andrea Arcangeli wrote:
> So I disagree with your claim that preempt risks to hide inefficient
> code, there are many other (probably easier) ways to detect inefficient
> code than to check the latencies.


You're ignoring the argument :)

If users and developers are presented with the _impression_ that long 
latency code paths don't exist, then nobody is motivated to profile them 
(with any tool), much less fix them.

	Jeff



From: Andrea Arcangeli [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 6 Oct 2004 06:03:23 +0200

On Tue, Oct 05, 2004 at 11:27:06PM -0400, Jeff Garzik wrote:
> You're ignoring the argument :)
> 
> If users and developers are presented with the _impression_ that long 
> latency code paths don't exist, then nobody is motivated to profile them 
> (with any tool), much less fix them.

well, you are assuming those latencies are visible with eyes. they might
be in extreme cases, but normally they're not (what people notices
normally are disk latencies, and few people uses an RT userspace
anyways which means they cannot claim the problem to be a lack of
cond_resched, but more likely they want shorter timeslices in the
scheduler etc..). So my point is that you need a measurement tool anyways...



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 00:08:47 -0400

Andrea Arcangeli wrote:
> 
> well, you are assuming those latencies are visible with eyes. they might
> be in extreme cases, but normally they're not (what people notices
> normally are disk latencies, and few people uses an RT userspace
> anyways which means they cannot claim the problem to be a lack of
> cond_resched, but more likely they want shorter timeslices in the
> scheduler etc..). So my point is that you need a measurement tool anyways...


I do agree with that.

I don't think that implies preempt is useful for anything except hiding 
stuff that should be fixed anyway, though...

Preempt will always be something I ask people to turn off when reporting 
driver bugs; it just adds too much complicated mess for zero gain.

	Jeff



From: Andrew Morton [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 5 Oct 2004 21:46:05 -0700

Jeff Garzik [email blocked] wrote:
>
> Preempt will always be something I ask people to turn off when reporting 
>  driver bugs; it just adds too much complicated mess for zero gain.

What driver bugs are apparent with preemption which are not already SMP bugs?

Only thing I can think of is unguarded use of per-cpu data, and we have
runtime debug checks for that now.



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 02:04:27 -0400

Andrew Morton wrote:
> Jeff Garzik [email blocked] wrote:
> 
>>Preempt will always be something I ask people to turn off when reporting 
>> driver bugs; it just adds too much complicated mess for zero gain.
> 
> 
> What driver bugs are apparent with preemption which are not already SMP bugs?

If your implied answer is true, then we wouldn't need 
preempt_{en,dis}able() sprinkled throughout the code so much.

	Jeff



From: Andrew Morton [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 5 Oct 2004 23:16:42 -0700

Jeff Garzik [email blocked] wrote:
>
> Andrew Morton wrote:
> > Jeff Garzik [email blocked] wrote:
> > 
> >>Preempt will always be something I ask people to turn off when reporting 
> >> driver bugs; it just adds too much complicated mess for zero gain.
> > 
> > 
> > What driver bugs are apparent with preemption which are not already SMP bugs?
> 
> If your implied answer is true, then we wouldn't need 
> preempt_{en,dis}able() sprinkled throughout the code so much.
> 

Where?

`grep -r preempt_disable drivers' points at one bodgy scsi driver.

`grep -r preempt_disable fs' finds two instances of per-cpu data.

`grep -r preempt_disable mm' finds three instances (wtf is vmalloc_to_page
trying to do?  Looks redundant)

`grep -r preempt_disable ipc' is empty

`grep -r preempt_disable net' is empty

`grep -r preempt_disable include' gets a few.

It's less than I expected, actually.




From: Nick Piggin [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 13:43:53 +1000

Jeff Garzik wrote:
> 
> You're ignoring the argument :)
> 
> If users and developers are presented with the _impression_ that long 
> latency code paths don't exist, then nobody is motivated to profile them 
> (with any tool), much less fix them.
> 

But even without preempt you'd still have to profile the latency.

If anyone with !preempt notices unacceptable latency, then they can
report and/or profile and fix it. If not, then !preempt latency must
be acceptable.



From: Jeff Garzik [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Tue, 05 Oct 2004 23:59:07 -0400

Nick Piggin wrote:
> 
> But even without preempt you'd still have to profile the latency.

You're making my point for me.  If the bandaid (preempt) is not active, 
then the system can be accurated profiled.  If preempt is active, then 
it is potentially hiding trouble spots.

The moral of the story is not to use preempt, as it
* potentially hides long latency code paths
* potentially introduces bugs, as we've seen with net stack and many 
other pieces of code
* is simply not needed, if all code paths are fixed

	Jeff



From: Nick Piggin [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 14:22:57 +1000

Jeff Garzik wrote:
> Nick Piggin wrote:
> 
>> Jeff Garzik wrote:
>>

>>
>> But even without preempt you'd still have to profile the latency.
> 
> 
> You're making my point for me.  If the bandaid (preempt) is not active, 
> then the system can be accurated profiled.  If preempt is active, then 
> it is potentially hiding trouble spots.

No. It can still be accurately profiled. You could profile theoretical
!preempt latency on a running preempt kernel.

*You* are ignoring my point :) *Nothing* has to be fixed if !preempt
users aren't seeing unacceptable latency. By definition, right?

> 
> The moral of the story is not to use preempt, as it
> * potentially hides long latency code paths
> * potentially introduces bugs, as we've seen with net stack and many 
> other pieces of code

So does CONFIG_SMP. For better or worse, it is in the kernel and
therefore a preempt bug is a bug and not a reason to turn preempt
off.

> * is simply not needed, if all code paths are fixed
> 

Jeff, the entire point of preempt is to not have to put cond_resched
everywhere. So yes, you can fix it by putting in lots of cond_rescheds,
or turning on preempt. What's more, with preempt, those that don't care
about 100us latency don't have to be executing cond_resched 10000 times
per second.

I'd actually say that the code needs fixing if the !PREEMPT case is doing
that much work to try to achieve insanely low latencies.



From: Aleksandar Milivojevic [email blocked]
Subject: Re: Preempt? (was Re: Cannot enable DMA on SATA drive (SCSI-libsata, VIA SATA))
Date: 	Wed, 06 Oct 2004 10:16:08 -0500

Jeff Garzik wrote:
> You're making my point for me.  If the bandaid (preempt) is not active, 
> then the system can be accurated profiled.  If preempt is active, then 
> it is potentially hiding trouble spots.
> 
> The moral of the story is not to use preempt, as it
> * potentially hides long latency code paths
> * potentially introduces bugs, as we've seen with net stack and many 
> other pieces of code
> * is simply not needed, if all code paths are fixed

One can also look onto it from another angle:
* conviniently resolves long latency code paths that can't be avoided
* uncovers bugs that need to be fixed
* implicitly fixes code paths

It seems to me that you are mixing latency of the system, efficiency of 
the driver functions, and performance of the system in the way it suits 
your arguments.

Those three influent each other, but should be looked at and solved 
separately.

Preempt is a fix for latency.  It doesn't (and can't) fix efficiency and 
performace.  If you are using latency as a measure for efficiency and 
performance, you are mixing apples and oranges with bananas.

Unefficient driver function (or code path) will not become efficient if 
you sprinkle it with cond_resched (it will only reduce the latency of 
the system).  As you conviniently said, you are just putting band aid on 
the problem, instead of fixing it.  Not different than using preept 
kernel, really.  Only more time spent on it by developer, that might be 
used better somewhere else (like making code path more efficient).

Performance of the system might be a bit lower with preempt kernel.  But 
most of those that would notice or care (0.1% of users?  probably less) 
would probably be happier without cond_resched executed thousands a time 
per second too, and would happily sacrifice latency to high performance.

Finally, the bugs.  Bugs need to be fixed.  Period.  If bug goes away 
when somebody turns off preempt on uniprocessor system, it may as well 
hit back when you move to non-preempt SMP system in even more obscure 
ways (because than you really have code paths executed in parallel). 
Telling somebody to try with non-preempt kernel should be debugging 
step, not the solution.

-- 
Aleksandar Milivojevic [email blocked]    Pollard Banknote Limited
Systems Administrator                           1499 Buffalo Place
Tel: (204) 474-2323 ext 276                     Winnipeg, MB  R3T 1L7

Related Links:

• Archive of above thread
• KernelTrap interview with Nick Piggin
• KernelTrap interview with Robert Love
• KernelTrap interview with Andrea Arcangeli
• KernelTrap interview with Andrew Morton