Hi,

An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of
timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too
heavy. So, this patch implements a lightweight timer migration framework.

Also, in a multi-core, multi-package system, it is always desirable to have all the timers firing on the cpus present in the same package. This would enable us to place the idle package in deep sleep
state. So, migration of timers is required for this.

A per_cpu sysfs hook is created at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to cpuX/timer_migration, all regular and hrtimers from cpuX are migrated to cpu0. The fact that all
timers are migrated from cpuX to cpu0 is demonstrated by making a note of /proc/interrupts.

$:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
$:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts > file2
$:diff file1 file2 | grep LOC

As we can observe, first we are enabling timer migration in cpu3. As a result, the number of local timer interrupts on cpu3 over the next 30 seconds are zero. cpu3  has been totally rid of all regular
and hrtimers.

Please refer to the following paper for details:

http://ols.fedoraproject.org/OLS/Reprints-2008/srinivasan1-reprint.pdf


thanks,
Arun
--

[ message continues ]

This seems like an exceedingly dumb idea - cpu 0 might be a long long
way from x and by pushing all timers to cpu0 you might actually overload
cpu 0.

Initially you talked about packages an moving timers to busy cpus -

If there's anything worth reading in there it should have been mentioned
in this email and preferably in some comment in the code and Kconfig
help as well.

--

[ message continues ]

Here, I've chosen cpu0 just to demonstrate migration of timers. I'm
working on the algorithm to choose the best cpu or the best set of cpus
to migrate the timers to. This is just the first iteration on which i
intend to build upon. What could be a possible way is that, a package
can be chosen as the destination for the migrated timers and any one or
many cpus in that package can be the target(s). So, this  will prevent
Thanks for the hint. I'll do that next time.

--

[ message continues ]

Who would trigger such migrations?

I'm not sure it's a good idea to make it fully manually triggered. It would be nicer 
if it was bound to the scheduler's power saving heuristic. So if the scheduler consolidates


I suspect moving all to cpu 0 is too inflexible as Peter already mentioned.

(haven't looked at the actual code yet sorry)

-Andi
--

[ message continues ]

On Tue, 16 Sep 2008 14:42:31 +0530

while I absolutely like reducing power consumption... I'm not sure this
helps or is the right approach.

First of all, it's of course absolutely better to fix timers and apps
that cause them (and with PowerTOP we fixed basically all the bad
stuff).

Second, in terms of power; a wake up is a wake up, it doesn't really
matter where it happens. 

Now we could do some consolidation (which realistically needs the range
timer feature that's aimed for 2.6.28), but I would much rather do that
in a different way: rather than actively moving stuff, I would instead
suggest sharing the timer queues between logical cpus that share the
same cache. (Now its an admin choice if he wants this on a "shared L1",
"shared L2" or "shared L3" basis). 

Or if you want to forcefully migrate timers, don't move existing ones,
just on queue/requeue put the on not-the-local-cpu. Sure they'll fire
once on the "wrong" cpu, but that's a very short term problem!

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
--

[ message continues ]

This is a very valid point. The algorithm which decides which cpu to
migrate the timers to should be intelligent enough in the sense that it
migrates *only* when there is some benefit out of it. That is, the cpu
to which the timers are being migrated, is sufficiently busy and it is
not unnecessarily brought out of its idle state. Else, as you pointed


While migrating range timers, I would migrate only if the time range of
the source and destination cpus match. Else, there is no point in
migrating. I would rather look at timer migration as an optimization
strategy to increase the cpu idle time and not as a pro-active method to
rid the cpu of timers, no matter what. I would like to migrate timers

--

[ message continues ]