Some of the concerns expressed about the Completely Fair Scheduler [1] were reports that it might not handle 3D games as well as the SD scheduler [2]. In a recent thread, Ingo Molnar noted, "people are regularly testing 3D smoothness, and they find CFS good enough [3] and that matches my experience as well (as limited as it may be). In general my impression is that CFS and SD are roughly on par when it comes to 3D smoothness." He noted that all known regressions were reported against earlier versions of CFS that had long since been fixed, and that he was very interested in any new reports of regressions against the current version of the code, "what is more interesting (to me) is not the positive CFS feedback but negative CFS feedback (although positive feedback certain _feels_ good so don't hold it back intentionally ;-)," adding, "there are no open 3D related regressions for CFS at the moment." Ingo then offered benchmarks illustrating the improved 3D performance of CFS, with numbers showing it to perform as well and in some cases considerably better than the SD scheduler.
Linus Torvalds noted, "I don't think _any_ scheduler is perfect, and almost all of the time, the RightAnswer(tm) ends up being not 'one or the other', but 'somewhere in between'." He noted that he was confident that he'd made the right decision in merging CFS, then added, "but at the same time, no technical decision is ever written in stone. It's all a balancing act. I've replaced the scheduler before, I'm 100% sure we'll replace it again. Schedulers are actually not at all that important in the end: they are a very very small detail in the kernel."
From: Kasper Sandberg [email blocked] Subject: SD still better than CFS for 3d (was Re: 2.6.23-rc1) Date: Fri, 27 Jul 2007 13:43:30 +0200 On Sun, 2007-07-22 at 14:04 -0700, Linus Torvalds wrote: > Ok, right on time, two weeks afetr 2.6.22, there's a 2.6.23-rc1 out there. > > And it has a *ton* of changes as usual for the merge window, way too much > for me to be able to post even just the shortlog or diffstat on the > mailing list (but I had many people who wanted to full logs to stay > around, so you'll continue to see those being uploaded to kernel.org). > > Lots of architecture updates (for just about all of them - x86[-64], arm, > alpha, mips, ia64, powerpc, s390, sh, sparc, um..), lots of driver updates > (again, all over - usb, net, dvb, ide, sata, scsi, isdn, infiniband, > firewire, i2c, you name it). > > Filesystems, VM, networking, ACPI, it's all there. And virtualization all > over the place (kvm, lguest, Xen). > > Notable new things might be the merge of the cfs scheduler, and the UIO > driver infrastructure might interest some people. Im still not so keen about this, Ingo never did get CFS to match SD in smoothness for 3d applications, where my test subjects are quake(s), world of warcraft via wine, unreal tournament 2004. And this is despite many patches he sent me to try and tweak it. As far as im concerned, i may be forced to unofficially maintain SD for my own systems(allthough lots in the gaming community is bound to be interrested, as it does make games lots better) <snip>
From: Ingo Molnar [4] [email blocked] Subject: Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Sun, 29 Jul 2007 19:06:41 +0200 * Kasper Sandberg [email blocked] wrote: > Im still not so keen about this, Ingo never did get CFS to match SD in > smoothness for 3d applications, where my test subjects are quake(s), > world of warcraft via wine, unreal tournament 2004. [...] here's an update: checking whether Wine could be a factor in your problem i just tested latest CFS against latest SD with a 3D game running under Wine: v2.6.22-ck1 versus v2.6.22-cfsv19 (to get the most comparable kernel), using Quake 3 Arena Demo under Wine (0.9.41). Here are the results in a pretty graph: http://people.redhat.com/mingo/misc/cfs-vs-sd-wine-quake.jpg [5] or, in text: 2.6.22-ck1 2.6.22-cfs-v19 ------------------------ ------------------------ quake + 0 loops | 41 fps quake + 0 loops | 41 fps quake + 1 loop | 3 fps quake + 1 loop | 41 fps quake + 2 loops | 2 fps quake + 2 loops | 32 fps quake + 3 loops | 1 fps quake + 3 loops | 24 fps quake + 4 loops | 0 fps quake + 4 loops | 20 fps quake + 5 loops | 0 fps quake + 5 loops | 16 fps Quake3-under-Wine behavior under SD/-ck: framerate breaks down massively during any kind of load. The game is completely unusable with 1 CPU loop running already! Quake3-under-Wine behavior under CFS: framerate goes down gently with load, gameplay remains smooth. Framerate is still pretty acceptable and the game is playable even with a 500% CPU overload. The graph looks good and the framerate reduction goes roughly along the expected 1/n 'fairness curve' - so it all looks pretty healthy. [Note: quake3 keeps its fully 41 fps even with 1 competing loop running on the CPU due to "sleeper fairness".] [ i've re-tested this using other SD and ck versions and other CFS versions such as v2.6.23-rc1 and the results are the same. To get the fps result i started a simple game scene: Single Player / Q3DM1 / I Can Win, turned on the fps display of Quake3, and did not move the player at all, just looked at the framerate that is displayed. (i also tried other scenes and other gameplay sections and they all behave consistently with the above results.) The system was otherwise completely idle. While i trust these numbers take them with a grain of salt, i'm obviously not neutral in this thing :-) ] so Kasper, i'll definitely need your help in tracking down your 3D smoothness problem under CFS. I have the feeling that it could be some odd factor that only hits your system, and once we've tracked that down there will be a simple solution that does not affect the totality of the scheduler. So far only you have reported any 3D game smoothness problem against recent CFS versions. (all 3D feedback has been positive, and that includes a number of gamers as well. Most of the 3D smoothness problems were fixed in CFS v13..v15 and it has not been reported to have regressed since then.) Ingo
From: Ingo Molnar [6] [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Sun, 29 Jul 2007 22:47:16 +0200 * John [email blocked] wrote: > Ingo- > > Why not perform the same test using the native linux Q3 client to > compare numbers to wine? [...] I regularly test native Linux games on CFS, and they all behave well. While waiting for more detailed data from Kasper i was looking for atypical stuff in Kasper's description about what his workload involves, and what looked a bit atypical was that Kasper's workload also involved gaming under Wine: > > > my test subjects are quake(s), world of warcraft via wine, unreal > > > tournament 2004. [...] and Wine is a more complex server/client scenario instead of a single (and simple) standalone Quake3 binary that the Linux binary does. So it looked more interesting from a scheduler workload (and scheduler regression) POV. In any case i'll need more info from Kasper. Ingo
From: Ingo Molnar [7] [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 13:46:49 +0200 * John [email blocked] wrote: > On 7/29/07, Ingo Molnar [email blocked] wrote: > > > > > > * John [email blocked] wrote: > > > > > Ingo- > > > > > > Why not perform the same test using the native linux Q3 client to > > > compare numbers to wine? [...] > > > > I regularly test native Linux games on CFS, and they all behave well. > > While waiting for more detailed data from Kasper i was looking for > > atypical stuff in Kasper's description about what his workload involves, > > and what looked a bit atypical was that Kasper's workload also involved > > gaming under Wine: > > I understand that, I was just wondering if the FPS scales the same > natively vs. Wine as I typically only run native games. [...] people are regularly testing 3D smoothness, and they find CFS good enough: http://bhhdoa.org.au/pipermail/ck/2007-June/007816.html [8] and that matches my experience as well (as limited as it may be). In general my impression is that CFS and SD are roughly on par when it comes to 3D smoothness. The Wine+Quake3 numbers i posted yesterday are so bad under SD that they must be some artifact in SD (possibly related to yield - i've strace-ed the tasks under SD today and they are blocking in yield), so they are not really representative of the general quality of SD (unless you are being hit by that particular regression). Still it is kind of ironic that when i tried to find a 3D regression in CFS i found a 3D regression in SD. What is more interesting (to me) is not the positive CFS feedback but negative CFS feedback (although positive feedback certain _feels_ good so dont hold it back intentionally ;-), and i cannot possibly give you any definitive answer: at this point CFS could still have artifacts and bugs, so "check and see yourself" is the best answer. All i can tell you is that there are no open 3D related regressions for CFS at the moment. > [...] I have been hesitant to move over to CFS due to reports of 3D > issues and wanted to see if you had numbers in regards to CFS vs. SD. i have no numbers now, other than the trivial native 'ppracer' game where SD and CFS have roughly the same framerate under load: SD CFS 0: 38.1 0: 38.1 1: 24.0 1: 24.2 2: 16.6 2: 16.1 3: 11.9 3: 12.3 4: 9.9 4: 9.7 5: 8.2 5: 8.1 which i'd have expected, ppracer is quite CPU-intense on my test-system, and the fairness model of SD and CFS is similar for CPU-bound tasks. But ... numbers from _me_ are suspect by definition, i wrote a good chunk of the CFS code :-) So it would be much more interesting if others provided more numbers. Would you be interested in trying CFS and doing some numers perhaps? It requires some work: you have to start up your favorite game in a way that gives a reliable framerate number. (many games allow the display of FPS in-game) In Quake3 i simply started the game and did not move the player - that is something easy to reproduce. then create load the following way, by entering this into a shell: while :; do :; done & that will cause a shell to just loop infinitely, hogging the CPU. This is the "1 loop" case in the numbers i posted. Start several of them to get more. (Type 'killall bash' in the same terminal to get rid of them.) Monitor how the FPS of your game changes when you start more and more CPU hogs, and note the numbers. Repeat it under SD and CFS as well, and please post the results into this thread. and note that CPU hogs are just one type of 'load' that a system can experience - IO load or networking load could impact your in-game experience just as much. If you see any artifact or FPS reduction under CFS i'll give you further info about how to debug it (were you interested in debugging it). Ingo
From: Kenneth Prugh [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 13:54:59 -0400 Ingo Molnar wrote: > <large snip> Hello, I have a gaming rig and would love to help benchmark with my copy of UT2004(E6600 Core2 and a 7950GTO card). Or if you have anything else that would better serve as a benchmark I could grab it and try. The only problem is I don't know what 2 kernels I should be using to test the schedulers. I assume 2.6.23-rc1 for CFS, but what about SD? -- Kenneth Prugh
From: Ingo Molnar [9] [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 21:10:29 +0200 * Kenneth Prugh [email blocked] wrote: > Ingo Molnar wrote: > > <large snip> > > Hello, I have a gaming rig and would love to help benchmark with my > copy of UT2004(E6600 Core2 and a 7950GTO card). Or if you have > anything else that would better serve as a benchmark I could grab it > and try. > > The only problem is I don't know what 2 kernels I should be using to > test the schedulers. I assume 2.6.23-rc1 for CFS, but what about SD? .22-ck1 includes it, so that should be fine: http://ussg.iu.edu/hypermail/linux/kernel/0707.1/0318.html [10] Ingo
From: Kenneth Prugh [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 17:24:27 -0400 Ingo Molnar wrote: > * Kenneth Prugh [email blocked] wrote: > >> Ingo Molnar wrote: >>> <large snip> >> Hello, I have a gaming rig and would love to help benchmark with my >> copy of UT2004(E6600 Core2 and a 7950GTO card). Or if you have >> anything else that would better serve as a benchmark I could grab it >> and try. >> >> The only problem is I don't know what 2 kernels I should be using to >> test the schedulers. I assume 2.6.23-rc1 for CFS, but what about SD? > > .22-ck1 includes it, so that should be fine: > > http://ussg.iu.edu/hypermail/linux/kernel/0707.1/0318.html [11] > > Ingo > Alright, Just got done with some testing of UT2004 between 2.6.23-rc1 CFS and 2.6.22-ck1 SD. This series of tests was run by spawning in a map while not moving at all and always facing the same direction, while slowing increasing the number of loops. CFS generally seemed a lot smoother as the load increased, while SD broke down to a highly unstable fps count that fluctuated massively around the third loop. Seems like I will stick to CFS for gaming now. Below you will find the results of my test with the average number of FPS. CFS | SD UT2004 + 0 loops | 200 FPS UT2004 + 0 loops | 190 FPS UT2004 + 1 loops | 195 FPS UT2004 + 1 loops | 190 FPS UT2004 + 2 loops | 190 FPS UT2004 + 2 loops | 190 FPS UT2004 + 3 loops | 189 FPS UT2004 + 3 loops | 136 FPS UT2004 + 4 loops | 150 FPS UT2004 + 4 loops | 137 FPS UT2004 + 5 loops | 145 FPS UT2004 + 5 loops | 136 FPS UT2004 + 6 loops | 145 FPS UT2004 + 6 loops | 105 FPS UT2004 + 7 loops | 118 FPS UT2004 + 7 loops | 104 FPS UT2004 + 8 loops | 97 FPS UT2004 + 8 loops | 104 FPS UT2004 + 9 loops | 94 FPS UT2004 + 9 loops | 89 FPS UT2004 + 10 loops | 92 FPS UT2004 + 10 loops | 91 FPS -- Kenneth Prugh
From: Miguel Figueiredo [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 22:34:21 +0100 can you apply the patch [1] that changes the behaviour of sched_yield on SD and report the results? SD should scale a lot better after the patch. 1 - http://bhhdoa.org.au/pipermail/ck/2007-July/008297.html [12] -- Com os melhores cumprimentos/Best regards, Miguel Figueiredo http://www.DebianPT.org [13]
From: Kenneth Prugh [email blocked] To: Miguel Figueiredo [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Mon, 30 Jul 2007 18:45:02 -0400 Miguel Figueiredo wrote: > > can you apply the patch [1] that changes the behaviour of sched_yield on SD > and report the results? > > SD should scale a lot better after the patch. > > 1 - http://bhhdoa.org.au/pipermail/ck/2007-July/008297.html [14] > I Applied the patch. SD Seemed a bit smoother over the loads, although that could be a placebo effect. It wasn't until the 8 or 9th loop running that I could really notice that the fps were fluctuating in the map without looking at the fps counter. SD-Patched UT2004 + 0 loops | 202 FPS UT2004 + 1 loops | 201 FPS UT2004 + 2 loops | 199 FPS UT2004 + 3 loops | 143 FPS UT2004 + 4 loops | 145 FPS UT2004 + 5 loops | 145 FPS UT2004 + 6 loops | 112 FPS UT2004 + 7 loops | 110 FPS UT2004 + 8 loops | 108 FPS UT2004 + 9 loops | 90 FPS UT2004 + 10 loops | 89 FPS -- Kenneth Prugh - Ken69267 Gentoo AMD64 Arch Tester
From: Ingo Molnar [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Tue, 31 Jul 2007 11:45:26 +0200 * Kenneth Prugh wrote: > Alright, Just got done with some testing of UT2004 between 2.6.23-rc1 > CFS and 2.6.22-ck1 SD. This series of tests was run by spawning in a > map while not moving at all and always facing the same direction, > while slowing increasing the number of loops. > > CFS generally seemed a lot smoother as the load increased, while SD > broke down to a highly unstable fps count that fluctuated massively > around the third loop. Seems like I will stick to CFS for gaming now. > > Below you will find the results of my test with the average number of > FPS. Thanks Kenneth for the testing! I've created a graph out of your numbers: http://people.redhat.com/mingo/misc/cfs-sd-ut2004-perf.jpg [15] (it also includes the SD numbers you got with the turn-yield-into-NOP hack applied.) Ingo
From: Matthew Hawkins [email blocked] Subject: Re: [ck] Re: SD still better than CFS for 3d ?(was Re: 2.6.23-rc1) Date: Tue, 31 Jul 2007 23:16:09 +1000 On 7/31/07, Ingo Molnar wrote: > * Kenneth Prugh wrote: > > CFS generally seemed a lot smoother as the load increased, while SD > > broke down to a highly unstable fps count that fluctuated massively > > around the third loop. Seems like I will stick to CFS for gaming now. My experience was quite similar. I noticed after launching the second loop that the FPS stuck down to 15 for about 20 seconds, then climbed back up to 48. After that it went rapidly downhill. This is similar to other benchmarks I've done of SD versus CFS in the past. At a "normal" load they're fairly similar but SD breaks down under pressure. The only other thing of interest is that the -ck kernel had the WM menus appear in about 3 seconds rather than 5-8 under the other two. Game: Nexuiz 2.3 OpenGL 2.0 shaders on Vertex Buffer Objects on Show FPS on ultimate quality 1024x768 2.6.23-git 0 48 1 48 2 48 3 48 4 40 5 38 6 33 7 28 8 22 9 22 10 18 2.6.22.1-ck 0 48 1 48 2 48 3 12 4 6 5 6 6 5 7 4 8 3 9 3 10 2 2.6.22.1-cfs-v19.1+ckbits [*] 0 48 1 48 2 48 3 46 4 45 5 43 6 36 7 32 8 25 9 24 10 24 [*] This kernel has the cfq-* and mm-* patches from -ck applied, and the above-background-load function from pre-SD ck patchsets (or 2.6.23-git) -- Matt
From: Kasper Sandberg [email blocked] Subject: Re: Linus 2.6.23-rc1 Date: Sat, 28 Jul 2007 11:44:08 +0200 On Fri, 2007-07-27 at 19:35 -0700, Linus Torvalds wrote: > > On Sat, 28 Jul 2007, Kasper Sandberg wrote: > > > > Im still not so keen about this, Ingo never did get CFS to match SD in > > smoothness for 3d applications, where my test subjects are quake(s), > > world of warcraft via wine, unreal tournament 2004. And this is despite > > many patches he sent me to try and tweak it. > > You realize that different people get different behaviour, don't you? > Maybe not. Sure. > > People who think SD was "perfect" were simply ignoring reality. Sadly, > that seemed to include Con too, which was one of the main reasons that I > never ended entertaining the notion of merging SD for very long at all: > Con ended up arguing against people who reported problems, rather than > trying to work with them. Im not saying its perfect, not at all, neither am i saying CFS is bad, surely CFS is much better than the old one, and i agree with what that university test you mentioned on kerneltrap says, that CFS and SD is basically impossible to feel difference in, EXCEPT for 3d under load, where CFS simply can not compete with SD, theres no but, this is how it has acted on every system ive tested, and YES, others reported it too, whether you choose to see it or not. and others people who run games on linux tells me the exact same thing, and i have had quite a few people try this. > > Andrew also reported an oops in the scheduler when SD was merged into -mm, > so there were other issues. And whats the point here? If you are trying to pull the old "Con just runs away", forget it, its a certainty that he would have put the required time into fixing whatever issues arise. > > > As far as im concerned, i may be forced to unofficially maintain SD for > > my own systems(allthough lots in the gaming community is bound to be > > interrested, as it does make games lots better) > > You know what? You can do whatever you want to. That's kind of the point > of open source. Keep people honest by having alternatives. True that > > But the the thing is, if you want to do a good job of doing that, here's a > big hint: instead of keeping to your isolated world, instead of just > talking about your own machine and ignoring other peoples machines and First off, i've personally run tests on many more machines than my own, i've had lots of people try on their machines, and i've seen totally unrelated posts to lkml, plus i've seen the experiences people are writing about on IRC. Frankly, im not just thinking of myself. > issues and instead of just denying that problems may exist, and instead of > attacking people who report problems, how about working with them? As i recall, there was only 1 persons reports that were attacked, and that was because the person repeatedly reported the EXPECTED behavior as broken, simply because it was FAIRLY allocating the cpu time, and this did not meet with the dudes expectations. And it was after multiple mails he was "attacked" > > That was where the SD patches fell down. They didn't have a maintainer > that I could trust to actually care about any other issues than his own. You may not have been able to trust Con, but thats because you havent taken the time to actually really see whats been going on, if you just read the threads for SD you'd realize that he was more than willing to maintain it, after all, why do you think he wrote and submitted it? you think he just wrote it to piss you off by having it merged and leave? > > So here's a hint: if you think that your particular graphics card setup is > the only one that matters, it's not going to be very interesting for > anybody else. as explained earlier, its not just my particular setup, but actually that of alot of people, with lots of different hardware. > > > [ I realize that this comes as a shock to some of the SD people, but I'm > told that there was a university group that did some double-blind > testing of the different schedulers - old, SD and CFS - and that > everybody agreed that both SD and CFS were better than the old, but that > there was no significant difference between SD and CFS. You can try > asking Thomas Gleixner for more details. ] > > I'm happy that SD was perfect for you. It wasn't for others, and it had > nobody who was even interested in trying to solve those issues. > > As a long-term maintainer, trust me, I know what matters. And a person who > can actually be bothered to follow up on problem reports is a *hell* of a > lot more important than one who just argues with reporters. Okay, i wasnt going to ask, but ill do it anyway, did you even read the threads about SD? Con was extremely polite to everyone, and he did work with a multitude of people, you seem to be totally deadlocked into the ONE incident with a person that was unhappy with SD, simply for being a fair scheduler.
From: Linus Torvalds [email blocked] Subject: Re: Linus 2.6.23-rc1 Date: Sat, 28 Jul 2007 10:50:48 -0700 (PDT) On Sat, 28 Jul 2007, Kasper Sandberg wrote: > > First off, i've personally run tests on many more machines than my own, > i've had lots of people try on their machines, and i've seen totally > unrelated posts to lkml, plus i've seen the experiences people are > writing about on IRC. Frankly, im not just thinking of myself. Ok, good. Has anybody tried to figure out why 3D games seem to be such a special case? I know Ingo looked at it, and seemed to think that he found and fixed something. But it sounds like it's worth a lot more discussion. > Okay, i wasnt going to ask, but ill do it anyway, did you even read the > threads about SD? I don't _ever_ go on specialty mailing lists. I don't read -mm, and I don't read the -fs mailing lists. I don't think they are interesting. And I tried to explain why: people who concentrate on one thing tend to become this self-selecting group that never looks at anything else, and then rejects outside input from people who hadn't become part of the "mind meld". That's what I think I saw - I saw the reactions from where external people were talking and cc'ing me. And yes, it's quite possible that I also got a very one-sided picture of it. I'm not disputing that. Con was also ill for a rather critical period, which was certainly not helping it all. > Con was extremely polite to everyone, and he did work > with a multitude of people, you seem to be totally deadlocked into the > ONE incident with a person that was unhappy with SD, simply for being a > fair scheduler. Hey, maybe that one incident just ended up being a rather big portion of what I saw. Too bad. That said, the end result (Con's public gripes about other kernel developers) mostly reinforced my opinion that I did the right choice. But maybe you can show a better side of it all. I don't think _any_ scheduler is perfect, and almost all of the time, the RightAnswer(tm) ends up being not "one or the other", but "somewhere in between". It's not like we've come to the end of the road: the baseline has just improved. If you guys can show that SD actually is better at some loads, without penalizing others, we can (and will) revisit this issue. So what you should take away from this is that: from what I saw over the last couple of months, it really wasn't much of a decision. The difference in how Ingo and Con reacted to peoples reports was pretty stark. And no, I haven't followed the ck mailing list, and so yes, I obviously did get just a part of the picture, but the part I got was pretty damn unambiguous. But at the same time, no technical decision is ever written in stone. It's all a balancing act. I've replaced the scheduler before, I'm 100% sure we'll replace it again. Schedulers are actually not at all that important in the end: they are a very very small detail in the kernel. Linus
Related Links:
- Archive of above thread [16]
- KernelTrap interview with Ingo Molnar [17]