Re: 2.6.27-rc5 OLTP performance regression

Previous thread: [git pull] x86 fixes by H. Peter Anvin on Monday, September 8, 2008 - 1:52 pm. (29 messages)

Next thread: [git pull] x86 fixes by H. Peter Anvin on Monday, September 8, 2008 - 3:32 pm. (15 messages)
To: Ma, Chinang <chinang.ma@...>
Cc: Ingo Molnar <mingo@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Monday, September 8, 2008 - 2:00 pm

Because the .26 group scheduler wasn't SMP aware. The extra cost comes
from the fact that .27 is.

--

To: Ma, Chinang <chinang.ma@...>
Cc: Ingo Molnar <mingo@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Monday, September 8, 2008 - 2:04 pm

What you can do it increase the /proc/sys/kernel/sched_shares_ratelimit
value and thereby decrease the accuracy of the SMP fairness of the group
scheduler.

--

To: Peter Zijlstra <peterz@...>
Cc: Ingo Molnar <mingo@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Tuesday, September 9, 2008 - 5:39 pm

Peter,
We increased sched_shares_ratelimit and were able to recover the 2% performance regression.
Thanks,

-----Original Message-----
From: Peter Zijlstra [mailto:peterz@infradead.org]
Sent: Monday, September 08, 2008 11:04 AM
To: Ma, Chinang
Cc: Ingo Molnar; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression

What you can do it increase the /proc/sys/kernel/sched_shares_ratelimit
value and thereby decrease the accuracy of the SMP fairness of the group
scheduler.

--

To: Ma, Chinang <chinang.ma@...>
Cc: Peter Zijlstra <peterz@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Wednesday, September 10, 2008 - 8:26 am

great. Could you please post a patch for that?

Ingo
--

To: Ingo Molnar <mingo@...>
Cc: Peter Zijlstra <peterz@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Wednesday, September 10, 2008 - 12:14 pm

I increased the /proc/sys/kernel/sched_shares_ratelimit from 1 millisecond to one second to reduce the frequency of shares update. That was okay for OLTP workload. As Peter mentioned, a larger value will reduce the accuracy of SMP fairness for group scheduler and the ratelimit and using a larger default value may not be good for general case.

Thanks,
Chinang
-----Original Message-----
From: Ingo Molnar [mailto:mingo@elte.hu]
Sent: Wednesday, September 10, 2008 5:26 AM
To: Ma, Chinang
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression

great. Could you please post a patch for that?

Ingo
--

To: Ma, Chinang <chinang.ma@...>
Cc: Peter Zijlstra <peterz@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Wednesday, September 10, 2008 - 1:34 pm

does 10 msecs already get you most of the 2%? 1000 millisecond sounds
very (and probably unacceptable) coarse.

Ingo
--

To: Ingo Molnar <mingo@...>
Cc: Peter Zijlstra <peterz@...>, Srivatsa Vaddagiri <vatsa@...>, Mike Galbraith <efault@...>, Gregory Haskins <ghaskins@...>, Steven Rostedt <rostedt@...>, Nick Piggin <nickpiggin@...>, Siddha, Suresh B <suresh.b.siddha@...>, Wilcox, Matthew R <matthew.r.wilcox@...>, Tripathi, Sharad C <sharad.c.tripathi@...>, Chilukuri, Harita <harita.chilukuri@...>, linux-kernel@vger.kernel.org <linux-kernel@...>
Date: Wednesday, September 10, 2008 - 8:07 pm

When set to 10 ms the regression is about 0.4%. We get most of the 2%.

Chinang
-----Original Message-----
From: Ingo Molnar [mailto:mingo@elte.hu]
Sent: Wednesday, September 10, 2008 10:35 AM
To: Ma, Chinang
Cc: Peter Zijlstra; Srivatsa Vaddagiri; Mike Galbraith; Gregory Haskins; Steven Rostedt; Nick Piggin; Siddha, Suresh B; Wilcox, Matthew R; Tripathi, Sharad C; Chilukuri, Harita; linux-kernel@vger.kernel.org
Subject: Re: 2.6.27-rc5 OLTP performance regression

does 10 msecs already get you most of the 2%? 1000 millisecond sounds
very (and probably unacceptable) coarse.

Ingo
--

Previous thread: [git pull] x86 fixes by H. Peter Anvin on Monday, September 8, 2008 - 1:52 pm. (29 messages)

Next thread: [git pull] x86 fixes by H. Peter Anvin on Monday, September 8, 2008 - 3:32 pm. (15 messages)