"Looking at these graphs (and the fixed one from your second email), it sure looks a lot like CFS is doing at *least* as well as the old scheduler in every single test, and doing much better in most of them (in addition it's much more consistent between runs)," Kyle Moffett noted regarding recent benchmarks run against the Completely Fair Scheduler by Rob Hussey. Kyle continued:
"This seems to jive with all the other benchmarks and overall empirical testing that everyone has been doing. Overall I have to say a job well done for Ingo, Peter, Con, and all the other major contributors to this impressive endeavor."
Rob's benchmarks:
lat_ctx -s 0 2
hackbench 50
pipe-test
Bound to single core:
lat_ctx -s 0 2
hackbench 50
pipe-test
From: Rob Hussey [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Thu, 13 Sep 2007 04:42:22 -0400
On 9/11/07, Rob Hussey [email blocked] wrote:
> Hi Ingo,
>
> When compiling, I get:
Yeah, this was my fault :(
I've had a chance to test this now, and everything feels great. I did
some benchmarks for 2.6.23-rc1, 2.6.23-rc6-cfs, and
2.6.23-rc6-cfs-devel:
lat_ctx -s 0 2:
2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel
5.15 4.91 5.05
5.23 5.18 4.85
5.19 4.89 5.17
5.36 5.23 4.86
5.35 5.00 5.13
5.34 5.05 5.12
5.26 4.99 5.06
5.11 5.04 4.96
5.29 5.19 5.18
5.40 4.93 5.07
hackbench 50:
2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel
6.301 5.963 5.837
6.417 5.961 5.814
6.468 5.965 5.757
6.525 5.926 5.840
6.320 5.929 5.751
6.457 5.909 5.825
pipe-test (http://redhat.com/~mingo/cfs-scheduler/tools/pipe-test.c):
2.6.23-rc1 2.6.23-rc6-cfs 2.6.23-rc6-cfs-devel
14.29 14.03 13.89
14.31 14.01 14.10
14.27 13.99 14.15
14.31 14.02 14.16
14.53 14.02 14.14
14.53 14.27 14.16
14.51 14.36 14.12
14.48 14.33 14.16
14.52 14.36 14.17
14.47 14.36 14.15
I turned the results into graphs as well. I'll attach them, but they're also at:
http://www.healthcarelinen.com/misc/lat_ctx_benchmark.png
http://www.healthcarelinen.com/misc/hackbench_benchmark.png
http://www.healthcarelinen.com/misc/pipe-test_benchmark.png
The hackbench and pipe-test numbers are very encouraging. The avg
between the 2.6.23-rc6-cfs and 2.6.23-rc6-cfs-devel lat_ctx numbers
are nearly identical (5.041 and 5.045 respectively).
From: Ingo Molnar [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Thu, 13 Sep 2007 11:06:55 +0200
* Rob Hussey [email blocked] wrote:
> On 9/11/07, Rob Hussey [email blocked] wrote:
> > Hi Ingo,
> >
> > When compiling, I get:
>
> Yeah, this was my fault :(
>
> I've had a chance to test this now, and everything feels great. I did
> some benchmarks for 2.6.23-rc1, 2.6.23-rc6-cfs, and
> 2.6.23-rc6-cfs-devel:
thanks for the numbers! Could you please also post the .config you used?
Thx,
Ingo
From: Ingo Molnar [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Thu, 13 Sep 2007 13:48:28 +0200
* Rob Hussey [email blocked] wrote:
> Well, I was going over my config myself after you asked for me to post
> it, and I thought to do the same thing. Except, disabling sched_debug
> caused the same error as before:
> In file included from kernel/sched.c:794:
> kernel/sched_fair.c: In function 'task_new_fair':
> kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first'
> undeclared (first use in this function)
> kernel/sched_fair.c:857: error: (Each undeclared identifier is
> reported only once
> kernel/sched_fair.c:857: error: for each function it appears in.)
> make[1]: *** [kernel/sched.o] Error 1
> make: *** [kernel] Error 2
>
> It only happens with sched_debug=y. I take it back, it wasn't my fault :)
>
> As for everything else, I'd be happy to.
are you sure this is happening with the latest iteration of the patch
too? (with the combo-3.patch?) You can pick it up from here:
http://people.redhat.com/mingo/cfs-scheduler/devel/sched-cfs-v2.6.23-rc6-v21-combo-3.patch
I tried your config and it builds fine here.
Ingo
From: "Rob Hussey" [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Thu, 13 Sep 2007 21:47:25 -0400
On 9/13/07, Ingo Molnar [email blocked] wrote:
>
> * Rob Hussey [email blocked] wrote:
>
> > Well, I was going over my config myself after you asked for me to post
> > it, and I thought to do the same thing. Except, disabling sched_debug
> > caused the same error as before:
> > In file included from kernel/sched.c:794:
> > kernel/sched_fair.c: In function 'task_new_fair':
> > kernel/sched_fair.c:857: error: 'sysctl_sched_child_runs_first'
> > undeclared (first use in this function)
> > kernel/sched_fair.c:857: error: (Each undeclared identifier is
> > reported only once
> > kernel/sched_fair.c:857: error: for each function it appears in.)
> > make[1]: *** [kernel/sched.o] Error 1
> > make: *** [kernel] Error 2
> >
> > It only happens with sched_debug=y. I take it back, it wasn't my fault :)
> >
> > As for everything else, I'd be happy to.
>
> are you sure this is happening with the latest iteration of the patch
> too? (with the combo-3.patch?) You can pick it up from here:
>
> http://people.redhat.com/mingo/cfs-scheduler/devel/sched-cfs-v2.6.23-rc6-v21-combo-3.patch
>
> I tried your config and it builds fine here.
>
> Ingo
>
I managed to work it all out (it was my fault after all), and I've now
made the changes you suggested to my .configs for 2.6.23-rc1 and
2.6.23-rc6. I've done the benchmarks all over, including tests with
the task bound to a single core. Without further ado, the numbers I
promised:
lat_ctx -s 0 2
# rc1 rc6 cfs-devel
1 4.58 4.39 4.42
2 4.76 4.42 4.41
3 4.74 4.52 4.67
4 4.74 4.44 4.76
5 4.79 4.74 4.59
6 4.80 4.65 4.76
7 4.52 4.54 4.50
8 4.72 4.57 4.62
9 4.87 4.67 4.80
10 4.69 4.47 4.65
hackbench 50
# rc1 rc6 cfs-devel
1 6.634 5.969 5.894
2 6.342 5.974 5.903
3 6.219 5.913 5.941
4 6.702 5.980 5.916
5 6.287 6.007 5.943
6 6.239 6.022 5.899
7 6.434 5.946 5.904
8 6.229 6.007 5.941
9 6.387 5.947 5.880
10 6.383 5.946 5.933
pipe-test
# rc1 rc6 cfs-devel
1 13.39 13.16 13.20
2 13.37 13.12 13.22
3 13.19 13.17 13.26
4 13.17 13.16 13.18
5 13.16 13.22 13.23
6 13.15 13.19 13.18
7 13.18 13.42 13.21
8 13.45 13.39 13.26
9 13.40 13.40 13.28
10 13.39 13.44 13.24
Bound to single core:
lat_ctx -s 0 2
# rc1 rc6 cfs-devel
1 3.20 2.61 2.37
2 3.20 2.60 2.40
3 3.21 2.67 2.38
4 3.19 2.66 2.34
5 3.19 2.64 2.37
6 3.22 2.67 2.36
7 3.21 3.29 2.36
8 3.22 2.61 2.44
9 3.23 2.68 2.36
10 3.22 2.60 2.37
hackbench 50
# rc1 rc6 cfs-devel
1 7.528 7.950 7.538
2 7.649 8.026 7.548
3 7.613 8.160 7.580
4 7.550 8.054 7.558
5 7.563 8.373 7.559
6 7.617 8.152 7.550
7 7.593 7.831 7.562
8 7.602 8.311 7.588
9 7.589 8.010 7.552
10 7.682 8.059 7.556
pipe-test
# rc1 rc6 cfs-devel
1 10.29 9.27 8.54
2 10.30 9.29 8.54
3 10.31 9.28 8.54
4 10.29 9.27 8.54
5 10.28 9.28 8.53
6 10.30 9.28 8.53
7 10.30 9.28 8.54
I've made graphs like last time:
http://www.healthcarelinen.com/misc/lat_ctx_benchmark.png
http://www.healthcarelinen.com/misc/hackbench_benchmark.png
http://www.healthcarelinen.com/misc/pipe-test_benchmark.png
http://www.healthcarelinen.com/misc/BOUND_lat_ctx_benchmark.png
http://www.healthcarelinen.com/misc/BOUND_hackbench_benchmark.png
http://www.healthcarelinen.com/misc/BOUND_pipe-test_benchmark.png
I'll also attach the graphs, as well as my .configs.
As for the data, I really can't read much into it. I think the graphs
tell the story better (if only I could understand the story). Overall
though, a definite improvement over 2.6.23-rc1.
From: "Rob Hussey" [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Thu, 13 Sep 2007 22:26:33 -0400
On 9/13/07, Rob Hussey [email blocked] wrote:
> Bound to single core:
...
> hackbench 50
> # rc1 rc6 cfs-devel
> 1 7.528 7.950 7.538
> 2 7.649 8.026 7.548
> 3 7.613 8.160 7.580
> 4 7.550 8.054 7.558
> 5 7.563 8.373 7.559
> 6 7.617 8.152 7.550
> 7 7.593 7.831 7.562
> 8 7.602 8.311 7.588
> 9 7.589 8.010 7.552
> 10 7.682 8.059 7.556
>
I knew there was no way I'd post all these numbers and not screw
something up. Switch rc6 and rc1 for hackbench 50 (bound to single
core). Updated graph:
http://www.healthcarelinen.com/misc/BOUND_hackbench_benchmark_fixed.png
Also attached.
From: Kyle Moffett [email blocked]
Subject: Re: [announce] CFS-devel, performance improvements
Date: Fri, 14 Sep 2007 02:59:58 -0400
On Sep 13, 2007, at 21:47:25, Rob Hussey wrote:
> On 9/13/07, Ingo Molnar [email blocked] wrote:
>> are you sure this is happening with the latest iteration of the
>> patch too? (with the combo-3.patch?) You can pick it up from here:
>>
>> http://people.redhat.com/mingo/cfs-scheduler/devel/sched-cfs-
>> v2.6.23-rc6-v21-combo-3.patch
>
> I managed to work it all out (it was my fault after all), and I've now
> made the changes you suggested to my .configs for 2.6.23-rc1 and
> 2.6.23-rc6. I've done the benchmarks all over, including tests with
> the task bound to a single core. Without further ado, the numbers I
> promised:
>
> [...]
>
> I've made graphs like last time:
> http://www.healthcarelinen.com/misc/lat_ctx_benchmark.png
> http://www.healthcarelinen.com/misc/hackbench_benchmark.png
> http://www.healthcarelinen.com/misc/pipe-test_benchmark.png
> http://www.healthcarelinen.com/misc/BOUND_lat_ctx_benchmark.png
> http://www.healthcarelinen.com/misc/BOUND_hackbench_benchmark.png
> http://www.healthcarelinen.com/misc/BOUND_pipe-test_benchmark.png
Well looking at these graphs (and the fixed one from your second
email), it sure looks a lot like CFS is doing at *least* as well as
the old scheduler in every single test, and doing much better in most
of them (in addition it's much more consistent between runs). This
seems to jive with all the other benchmarks and overall empirical
testing that everyone has been doing. Overall I have to say a job
well done for Ingo, Peter, Con, and all the other major contributors
to this impressive endeavor.
Cheers,
Kyle Moffett
When I first looked at the
When I first looked at the graphs I thought there is a big difference between the different versions. I was fooled by the y-scale that does not begin at 0. An y-scale starting from 0 would have illustrated that the differences are marginal.
y scale
The y scale doesn't have to start at 0 to be significant. The graphs say to me that there may be optimizations that can still be done on SMP machines, but the other results _are_ significant. hackbench-50 and pipe-test show much more consistent (aka predictable) behavior, and the single core lat_ctx and pipe_test graphs also show improvements of about 25% and 18%. Just because the numbers are small doesn't mean those differences aren't big...
An y-scale starting from 0
An y-scale starting from 0 would have been rediculous, hiding what each scheduler does better... There's a reason math programs like matlab zoom the graph, you know? This is not a commercial, it's input to/from a technical list.
And no, they are definately not "marginal".
Would have been interesting
It would have been even more interesting to see linux-2.6.22-ck1 (Con's latest scheduler) ran along side the ones shown for comparison..
Scheduler, linux-2.6.22-ck1 vs 2.6.23-rc6
Here is a recent report:
http://lkml.org/lkml/2007/9/16/73
Doesn't mean anything, it's just one report about desktop behaviour, but It's worth mentioning..
Something tells me it won't
Something tells me it won't happen. ^^
It already did
Actually, new benchmarks do exactly this:
http://kerneltrap.org/mailarchive/linux-kernel/2007/9/17/261647
2.6.21 (old scheduler) versus 2.6.22-ck1 (SD scheduler) versus 2.6.23-rc6-cfs-devel (CFS).