"People who think SD was 'perfect' were simply ignoring reality," Linus Torvalds began in a succinct explanation as to why he chose the CFS scheduler written by Ingo Molnar instead of the SD scheduler written by Con Kolivas. He continued, "sadly, that seemed to include Con too, which was one of the main reasons that I never [entertained] the notion of merging SD for very long at all: Con ended up arguing against people who reported problems, rather than trying to work with them." He went on to stress the importance of working toward a solution that is good for everyone, "that was where the SD patches fell down. They didn't have a maintainer that I could trust to actually care about any other issues than his own." He then offered some praise to Ingo, "as a long-term maintainer, trust me, I know what matters. And a person who can actually be bothered to follow up on problem reports is a *hell* of a lot more important than one who just argues with reporters." Linus went on to note a comparison between the two schedulers:

"I realize that this comes as a shock to some of the SD people, but I'm told that there was a university group that did some double-blind testing of the different schedulers - old, SD and CFS - and that everybody agreed that both SD and CFS were better than the old, but that there was no significant difference between SD and CFS."

Con Kolivas maintained the -ck Linux kernel patchset which aimed at improving the desktop experience since 2002, originally for the 2.4 kernel. Shortly after the decision to merge the CFS scheduler instead of his SD scheduler, and without an official response about merging his swap prefetch patch, he announced his decision to stop working on the Linux kernel. More information about his contributions and recent decision can be found in this interview on apcmag.com.

Ingo Molnar wrote the original CFS scheduler within a 62 hour window of time starting on April 11'th, 2007. Early reports on the CFS scheduler suggested it was an improvement over the old scheduler, but perhaps not over the SD scheduler. Ingo determinedly followed up on all bug and regression reports, rapidly improving the scheduler and addressing all known issues. It was merged into the 2.6.23 kernel on July 9'th, three months after it was written.

From:	Kasper Sandberg [email blocked]
To:	Linus Torvalds [email blocked]
Subject: Re: Linus 2.6.23-rc1
Date:	Sat, 28 Jul 2007 04:04:39 +0200

(sorry for repost, but there seemed to have been some troubles..)

On Sun, 2007-07-22 at 14:04 -0700, Linus Torvalds wrote:
> Ok, right on time, two weeks afetr 2.6.22, there's a 2.6.23-rc1 out there.
> 
> And it has a *ton* of changes as usual for the merge window, way too much 
> for me to be able to post even just the shortlog or diffstat on the 
> mailing list (but I had many people who wanted to full logs to stay 
> around, so you'll continue to see those being uploaded to kernel.org).
> 
> Lots of architecture updates (for just about all of them - x86[-64], arm, 
> alpha, mips, ia64, powerpc, s390, sh, sparc, um..), lots of driver updates 
> (again, all over - usb, net, dvb, ide, sata, scsi, isdn, infiniband, 
> firewire, i2c, you name it).
> 
> Filesystems, VM, networking, ACPI, it's all there. And virtualization all 
> over the place (kvm, lguest, Xen).
> 
> Notable new things might be the merge of the cfs scheduler, and the UIO 
> driver infrastructure might interest some people.
> 
Im still not so keen about this, Ingo never did get CFS to match SD in
smoothness for 3d applications, where my test subjects are quake(s),
world of warcraft via wine, unreal tournament 2004. And this is despite
many patches he sent me to try and tweak it. As far as im concerned, i
may be forced to unofficially maintain SD for my own systems(allthough
lots in the gaming community is bound to be interrested, as it does make
games lots better)


<snip>



From:	Linus Torvalds [email blocked]
To:	Kasper Sandberg [email blocked]
Subject: Re: Linus 2.6.23-rc1
Date:	Fri, 27 Jul 2007 19:35:58 -0700 (PDT)



On Sat, 28 Jul 2007, Kasper Sandberg wrote:
>
> Im still not so keen about this, Ingo never did get CFS to match SD in
> smoothness for 3d applications, where my test subjects are quake(s),
> world of warcraft via wine, unreal tournament 2004. And this is despite
> many patches he sent me to try and tweak it.

You realize that different people get different behaviour, don't you? 
Maybe not.

People who think SD was "perfect" were simply ignoring reality. Sadly, 
that seemed to include Con too, which was one of the main reasons that I 
never ended entertaining the notion of merging SD for very long at all: 
Con ended up arguing against people who reported problems, rather than 
trying to work with them.

Andrew also reported an oops in the scheduler when SD was merged into -mm, 
so there were other issues.

> As far as im concerned, i may be forced to unofficially maintain SD for 
> my own systems(allthough lots in the gaming community is bound to be 
> interrested, as it does make games lots better)

You know what? You can do whatever you want to. That's kind of the point 
of open source. Keep people honest by having alternatives.

But the the thing is, if you want to do a good job of doing that, here's a 
big hint: instead of keeping to your isolated world, instead of just 
talking about your own machine and ignoring other peoples machines and 
issues and instead of just denying that problems may exist, and instead of 
attacking people who report problems, how about working with them?

That was where the SD patches fell down. They didn't have a maintainer 
that I could trust to actually care about any other issues than his own.

So here's a hint: if you think that your particular graphics card setup is 
the only one that matters, it's not going to be very interesting for 
anybody else. 

[ I realize that this comes as a shock to some of the SD people, but I'm 
  told that there was a university group that did some double-blind 
  testing of the different schedulers - old, SD and CFS - and that 
  everybody agreed that both SD and CFS were better than the old, but that 
  there was no significant difference between SD and CFS. You can try 
  asking Thomas Gleixner for more details. ]

I'm happy that SD was perfect for you. It wasn't for others, and it had 
nobody who was even interested in trying to solve those issues. 

As a long-term maintainer, trust me, I know what matters. And a person who 
can actually be bothered to follow up on problem reports is a *hell* of a 
lot more important than one who just argues with reporters.

			Linus




From:	Grzegorz Kulewski [email blocked]
To:	Linus Torvalds [email blocked]
Subject: Re: [ck] Re: Linus 2.6.23-rc1
Date:	Sat, 28 Jul 2007 09:09:06 +0200 (CEST)

On Fri, 27 Jul 2007, Linus Torvalds wrote:
> On Sat, 28 Jul 2007, Kasper Sandberg wrote:
>>
>> Im still not so keen about this, Ingo never did get CFS to match SD in
>> smoothness for 3d applications, where my test subjects are quake(s),
>> world of warcraft via wine, unreal tournament 2004. And this is despite
>> many patches he sent me to try and tweak it.
>
> You realize that different people get different behaviour, don't you?
> Maybe not.
>
> People who think SD was "perfect" were simply ignoring reality. Sadly,
> that seemed to include Con too, which was one of the main reasons that I
> never ended entertaining the notion of merging SD for very long at all:
> Con ended up arguing against people who reported problems, rather than
> trying to work with them.

I don't really want to keep all that -ck flamewar going but this sum-up is 
a little strange for me:

If Con was thinking SD was "perfect" why he released 30+ versions of it? 
And who knows how many versions of his previous scheduler?

Besides Con always tried to help people and improve his code if some bugs 
or problems were reported. Archives of this list prove that. I reported 
several problems (on list and privately) and all were fixed very fast and 
with very kind responses. I had run -ck for months and years and it was 
always very stable (I remember one broken "stable" version).

I don't know what exactly are you refering to when you say about those 
unaddressed reports but maybe it depends on who was asking, how and to do 
what (for example - purely theoretical one, I don't remember exact emails 
you refering to so I am not saying it happened - stating at the beginning 
that the whole design is unacceptable and interactivity hacks are a 
must-have won't make a friend from any maintainer and for sure lowers his 
desire to get anything fixed for that guy). Or maybe Con had some bad day 
or was depressed. Happens. But I really don't remember Con ignoring too 
many valuable user reports in last 3 years...

And no - I am not thinking that SD was "perfect". Nothing is perfect, 
especially not software. But it was based on months and years of Con's
experience with desktop and gaming workloads and extensively tested in 
similar uses by _many_ others. In nearly all possible desktop 
configurations, with most games and all video drivers. This is why it was 
perfectly designed and tuned for such workloads while still being general 
enough and without any ugly hacks. And because of these tests and Con's 
believe that the desktop is very (most?) important all bugs and problems 
in this area were probably killed long ago. I think even design was 
changed and tuned a little at the early stages to help solve such 
interactivity/dekstop/gaming problems.

So it does not surprise me that CFS is worse in such workloads (at least 
for some people) because I strongly suspect that the number of people who 
played games with current version of CFS is limited to about 5, maybe 10. 
And I also suspect that you (and Ingo) will get many regression reports 
when 2.6.23 is released (and months later too... or maybe you won't 
because users will be to "scared" to report such hard to mensure and 
reproduce "unimportant" bugs). Hopefully such problems when reported will 
be addressed as soon as they can. And hopefully they will be easy enough 
to solve without rewriting or redesigning CFS and causing that way even 
more regressions in other areas. If not people will probably be patching 
O(1) scheduler back privately...


Thanks,

Grzegorz Kulewski



From:	Linus Torvalds [email blocked]
Subject: Re: [ck] Re: Linus 2.6.23-rc1
Date:	Sat, 28 Jul 2007 10:12:32 -0700 (PDT)



On Sat, 28 Jul 2007, Jonathan Jessup wrote:
> 
> Linus, there is a complaint about the Linux kernel, this complaint is that
> the Linux kernel isn't giving priorities to desktop interactivity and
> experience. The response on osnews.com etc have shown that there is public
> demand for it too.

No, the response on osnews.com only shows that there are a lot of armchair 
complainers around.

People are suggesting that you'd have a separate "desktop kernel". That's 
insane. It also shows total ignorance of maintainership, and reality. And 
I bet most of the people there haven't tested _either_ scheduler, they 
just like making statements.

The fact is, I've _always_ considered the desktop to be the most important 
part. And I suspect that that actually is true for most kernel developers, 
because quite frankly, that's what 99% of them ends up using. If a kernel 
developer uses Windows for his day-to-day work, I sure as hell wouldn't 
want to have him developing Linux. That has nothing to do with anything 
anti-windows: but the whole "eat your own dogfood" is a very fundamental 
thing, and somebody who doesn't do that shouldn't be allowed to be even 
_close_ to a compiler!

So the whole argument about how kernel developers think that the desktop 
isn't important is totally made-up crap by Con, and then parrotted by 
osnews and other places.

The fact is, most kernel developers realize that Linux is used in 
different places, on different machines, and with different loads. You 
cannot make _everybody_ happy, but you can try to do as good a job as 
possible. And doing "as good a job as possible" very much includes not 
focusing on any particular load.

And btw, "the desktop" isn't actually one single load. It's in fact a lot 
of very different loads, and different people want different things. What 
makes the desktop so interesting is in fact that it shows more varied 
usage than any other niche - and no, 3D gaming isn't "it". 

> Maybe once or twice Con couldn't help or fix an issue but isn't that what
> open source software is all about anyway?

That's not the issue. 

Con wass fixated on one thing, and one thing only, and wasn't interested 
in anythign else - and attacked people who complained. Compare that to 
Ingo, who saw that what Con's scheduler did was good, and tried to solve 
the problems of people who complained.

The ck mailing list is/was also apparently filled with people who all had 
the same issues, which is seriously the *wrong* thing to do. It means that 
any "consensus" coming out of that kind of private list is totally 
worthless, because the people you ask are already in agreement - you have 
a so-called "selection bias", and they just reinforce their own opinions.

Which is why I don't trust mailing lists with a narrow topic. They are 
_useless_. If you cannot get many different people from _different_ areas 
to test your patches, and cannot see the big picture, the end result won't 
likely be very interesting to others, will it?

The fact is, _any_ scheduler is going to have issues. I will bet you 
almost any amount of money that people are going to complain about Ingo's 
scheduler when 2.6.23 is released. That's not the issue: the issue is that 
the exact same thing would have happened with CK too.

So if you are going to have issues with the scheduler, which one do you 
pick: the one where the maintainer has shown that he can maintain 
schedulers for years, can can address problems from _different_ areas of 
life? Or the one where the maintainer argues against people who report 
problems, and is fixated on one single load?

That's really what it boils down to. I was actually planning to merge CK 
for a while. The _code_ didn't faze me.

			Linus



From:	Kasper Sandberg [email blocked]
To:	Linus Torvalds [email blocked]
Subject: Re: Linus 2.6.23-rc1
Date:	Sat, 28 Jul 2007 11:44:08 +0200

On Fri, 2007-07-27 at 19:35 -0700, Linus Torvalds wrote:
> 
> On Sat, 28 Jul 2007, Kasper Sandberg wrote:
> >
> > Im still not so keen about this, Ingo never did get CFS to match SD in
> > smoothness for 3d applications, where my test subjects are quake(s),
> > world of warcraft via wine, unreal tournament 2004. And this is despite
> > many patches he sent me to try and tweak it.
> 
> You realize that different people get different behaviour, don't you? 
> Maybe not.

Sure.

> 
> People who think SD was "perfect" were simply ignoring reality. Sadly, 
> that seemed to include Con too, which was one of the main reasons that I 
> never ended entertaining the notion of merging SD for very long at all: 
> Con ended up arguing against people who reported problems, rather than 
> trying to work with them.

Im not saying its perfect, not at all, neither am i saying CFS is bad,
surely CFS is much better than the old one, and i agree with what that
university test you mentioned on kerneltrap says, that CFS and SD is
basically impossible to feel difference in, EXCEPT for 3d under load,
where CFS simply can not compete with SD, theres no but, this is how it
has acted on every system ive tested, and YES, others reported it too,
whether you choose to see it or not. and others people who run games on
linux tells me the exact same thing, and i have had quite a few people
try this.

> 
> Andrew also reported an oops in the scheduler when SD was merged into -mm, 
> so there were other issues.

And whats the point here? If you are trying to pull the old "Con just
runs away", forget it, its a certainty that he would have put the
required time into fixing whatever issues arise.

> 
> > As far as im concerned, i may be forced to unofficially maintain SD for 
> > my own systems(allthough lots in the gaming community is bound to be 
> > interrested, as it does make games lots better)
> 
> You know what? You can do whatever you want to. That's kind of the point 
> of open source. Keep people honest by having alternatives.

True that

> 
> But the the thing is, if you want to do a good job of doing that, here's a 
> big hint: instead of keeping to your isolated world, instead of just 
> talking about your own machine and ignoring other peoples machines and 
First off, i've personally run tests on many more machines than my own,
i've had lots of people try on their machines, and i've seen totally
unrelated posts to lkml, plus i've seen the experiences people are
writing about on IRC. Frankly, im not just thinking of myself.

> issues and instead of just denying that problems may exist, and instead of 
> attacking people who report problems, how about working with them?

As i recall, there was only 1 persons reports that were attacked, and
that was because the person repeatedly reported the EXPECTED behavior as
broken, simply because it was FAIRLY allocating the cpu time, and this
did not meet with the dudes expectations. And it was after multiple
mails he was "attacked"

> 
> That was where the SD patches fell down. They didn't have a maintainer 
> that I could trust to actually care about any other issues than his own.

You may not have been able to trust Con, but thats because you havent
taken the time to actually really see whats been going on, if you just
read the threads for SD you'd realize that he was more than willing to
maintain it, after all, why do you think he wrote and submitted it? you
think he just wrote it to piss you off by having it merged and leave?

> 
> So here's a hint: if you think that your particular graphics card setup is 
> the only one that matters, it's not going to be very interesting for 
> anybody else.

as explained earlier, its not just my particular setup, but actually
that of alot of people, with lots of different hardware.

>  
> 
> [ I realize that this comes as a shock to some of the SD people, but I'm 
>   told that there was a university group that did some double-blind 
>   testing of the different schedulers - old, SD and CFS - and that 
>   everybody agreed that both SD and CFS were better than the old, but that 
>   there was no significant difference between SD and CFS. You can try 
>   asking Thomas Gleixner for more details. ]
> 
> I'm happy that SD was perfect for you. It wasn't for others, and it had 
> nobody who was even interested in trying to solve those issues. 
> 
> As a long-term maintainer, trust me, I know what matters. And a person who 
> can actually be bothered to follow up on problem reports is a *hell* of a 
> lot more important than one who just argues with reporters.

Okay, i wasnt going to ask, but ill do it anyway, did you even read the
threads about SD? Con was extremely polite to everyone, and he did work
with a multitude of people, you seem to be totally deadlocked into the
ONE incident with a person that was unhappy with SD, simply for being a
fair scheduler.



From:	Linus Torvalds [email blocked]
To:	Kasper Sandberg [email blocked]
Subject: Re: Linus 2.6.23-rc1
Date:	Sat, 28 Jul 2007 10:50:48 -0700 (PDT)



On Sat, 28 Jul 2007, Kasper Sandberg wrote:
>
> First off, i've personally run tests on many more machines than my own,
> i've had lots of people try on their machines, and i've seen totally
> unrelated posts to lkml, plus i've seen the experiences people are
> writing about on IRC. Frankly, im not just thinking of myself.

Ok, good. Has anybody tried to figure out why 3D games seem to be such a 
special case? 

I know Ingo looked at it, and seemed to think that he found and fixed 
something. But it sounds like it's worth a lot more discussion.

> Okay, i wasnt going to ask, but ill do it anyway, did you even read the
> threads about SD?

I don't _ever_ go on specialty mailing lists. I don't read -mm, and I 
don't read the -fs mailing lists. I don't think they are interesting. 

And I tried to explain why: people who concentrate on one thing tend to 
become this self-selecting group that never looks at anything else, and 
then rejects outside input from people who hadn't become part of the "mind 
meld". 

That's what I think I saw - I saw the reactions from where external people 
were talking and cc'ing me.

And yes, it's quite possible that I also got a very one-sided picture of 
it. I'm not disputing that. Con was also ill for a rather critical period, 
which was certainly not helping it all.

> Con was extremely polite to everyone, and he did work
> with a multitude of people, you seem to be totally deadlocked into the
> ONE incident with a person that was unhappy with SD, simply for being a
> fair scheduler.

Hey, maybe that one incident just ended up being a rather big portion of 
what I saw. Too bad. That said, the end result (Con's public gripes about 
other kernel developers) mostly reinforced my opinion that I did the right 
choice.

But maybe you can show a better side of it all. I don't think _any_ 
scheduler is perfect, and almost all of the time, the RightAnswer(tm) ends 
up being not "one or the other", but "somewhere in between".

It's not like we've come to the end of the road: the baseline has just 
improved. If you guys can show that SD actually is better at some loads, 
without penalizing others, we can (and will) revisit this issue.

So what you should take away from this is that: from what I saw over the 
last couple of months, it really wasn't much of a decision. The difference 
in how Ingo and Con reacted to peoples reports was pretty stark. And no, I 
haven't followed the ck mailing list, and so yes, I obviously did get just 
a part of the picture, but the part I got was pretty damn unambiguous.

But at the same time, no technical decision is ever written in stone. It's 
all a balancing act. I've replaced the scheduler before, I'm 100% sure 
we'll replace it again. Schedulers are actually not at all that important 
in the end: they are a very very small detail in the kernel.

				Linus

• Archive of above thread