On 08/03/2008, Gregory Haskins <ghaskins@novell.com> wrote:
quoted text
> >>> On Sat, Mar 8, 2008 at  3:41 AM, in message <20080308084118.GA24552@elte.hu>,
>  Ingo Molnar <mingo@elte.hu> wrote:
>
>  > * Gregory Haskins <ghaskins@novell.com> wrote:
>  >
>  >>    Unfortunately, this patch will introduce its own set of bugs.
>  >> However, your analysis was spot-on.  I think I see the problem now.
>  >> It was introduced when I put a hack in to "fix" s2ram problems in -mm
>  >> as a result of the new root-domain logic.  I think the following patch
>  >> will fix both issues:
>  >>
>  >> (I verified that I could take a cpu offline/online, but I don't have
>  >> an s2ram compatible machine handy.  Andrew, I believe you could
>  >> reproduce the s2ram problem a few months ago when that issue popped
>  >> up.  So if you could, please verify that s2ram also works with this
>  >> patch applied, in addition to the hotplug problem.
>  >
>  > thanks Gregory, i've queued up your fix. If it passes all tests over the
>  > day i'll send it to Linus in about 12 hours.
>  >
>
>
>
> After thinking about it some more, I am not sure if I got this fix quite right. The first two hunks are technically fine and should ultimately go in.  The last hunk is questionable.
>
>  Ultimately, I believe the root cause of these reported issues is that cpu_online_map and rd->online can get out of sync.  I fear that while the current patch may fix the hotplug/s2ram case, it potentially breaks the disjoint cpuset topology change case.  I will run a few cpuset tests ASAP to confirm, though I may not have a chance until Monday.
>
>  What I probably need is to tie the code that sets/clears cpu_online_map to the root-domain rd->online map somehow.  However, a cursory cscope inspection has failed to reveal the location that this cpu_online_map manipulation occurs.  If anyone knows where cpu_online_map is actually updated, please let me know.

I guess, it's in arch-specific code. e.g. for x86-64,

down:

kernel/cpu.c :: _cpu_down -> take_cpu_down() -> __cpu_disable() [
arch/x86/kernel/smpboot_64.c ] -> cpu_clear(cpu, cpu_online_map)

up:

kernel/cpu.c :: _cpu_up() -> __cpu_up [ arch/x86/kernel/smpboot_64.c ]
-> do_boot_cpu() -> start_secondary() [ runs on a to-be-online cpu ]
-> cpu_set(smp_processor_id(), cpu_online_map)

there are cpu-notification events available before a cpu gets removed
from cpu_online_map or after it gets added, so I guess (a first guess.
I'll also look at the code) they should be a sync. point.

I have a small patch to record some events during the cpu_down/up. I
have a trace but will already analyze it tomorrow.
[ the patch is enclosed (err.. it also includes Gregory's fix... I
just forgot to commit it previously :-/) ]

e.g. it should print a message if a task is placed on the 'offline'
cpu. Note, the 'cpu' is removed from the cpu_online_map before
migrate_live_tasks() runs. So if any task remains on the offline cpu,
it has been likely placed there already after migrate_live_tasks().


quoted text
>  Regards,
>  -Greg
>

-- 
Best regards,
Dmitry Adamushko