Hi,
We are noticing a significant regression in performance with some test workloads on
x86_64 NUMA machines.  The workload is  multiple  datatest processes
running in parallel.  Each process is basically performing  writes and
reads to disk backed files.  More information about datatest can be found at:
http://home.comcast.net/~SCSIguy/SCSI_FAQ/RMiller_Tools/dt.html

To give an idea about the regression, on a plain vanilla 2.6.21 kernel,  a test suite
running a bunch of datatest processes on different files takes about 900 seconds to
complete, and with  2.6.25/26/27-rc3 the exact same workload takes 1200 seconds to complete.
Two times slower!  This is on a 24 cpu 6 node NUMA box.

Looking at schedstats, I see that there are many more remote wakeups due to
passive load balancing with 27 than in 21, which seems to be causing this
performance difference (ttwu_wake_remote). In fact, there are twice as many
ttwu_wake_remote events in the NODE domain with 27 than on 21.  If I turn off wake
balancing on the NUMA domain, the same workload runs in 300-400s!  Obviously turning off wake
balancing is not the right  solution, but, the increased
remote wakeups due to 'passive' wake balancing is definitely a regression!

I am attaching the full schedstat snapshots with 2.6.21 and 2.6.27 vanilla, before starting
the workload and at the end of the workload.  This is on a 6 node, 24 core, 12 socket
woodcrest machine.

I was wondering if this regression has been noted elsewhere.

Thanks,
Kiran