(copying lkml and some scheduler folk)


What was the performance hit?  What was your I/O setup (image format, 


Most likely it also hits non-virtualized loads as well.  If the 
scheduler pulls two long-running threads to the same cpu, performance 
will take a hit.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--

[ message continues ]

Since the hit only happens when physical cpus are saturated, and sheduling non-virtualized multiple threads of one process to same processor can benefit the performance due to cache share or other affinities, but you know it hurts performance a lot once schedule two vcpu theads to a same processor due to mutual spin-lock in guests. 
Xiantao
--

[ message continues ]

Is this on a machine that does pause-loop exits?  The current handing of 
PLE is very suboptimal.  With proper directed yield we should be much 
better there.



Spin loops need to be addressed first, they are known to kill 
performance in overcommit situations.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--

[ message continues ]

Even in overcommit case, if vcpu threads of one qemu are not scheduled or pulled to the same logical processor, the performance drop is tolerant like Xen's case today. But for KVM, it has to suffer from additional performance loss, since host's scheduler actively pulls these vcpu threads together. 
Xiantao 

--

[ message continues ]

Can you quantify this loss?  Give examples of what happens?


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

--

[ message continues ]

For example, one machine is configured with 2 pCPUs and there are two Windows guests running on the machine, and each guest is cconfigured with 2 vcpus and one webbench server runs in it. 
If use host's default scheduler, webbench's performance is very bad, but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can see 5-10X performance improvement with same CPU utilization. 
In addition, we also see kvm's perf scalability is also impacted in large systems, for some performance experiments, kvm's perf begins to drop when vCPU is overcommitted and pCPU are saturated, but once the wake_up_affine feature is switched off in scheduler, kvm's perf can keep rising in this case.   
Xiantao

--

[ message continues ]

Ok.  This is probably due to spinlock contention.

When vcpus are pinned to pcpus, there is a 50% chance that a guest's 
vcpus will be co-scheduled and spinlocks will perform will.

When vcpus are not pinned, but affine wakeups are disabled, there is a 
33% chance that vcpus will be co-scheduled.

When vcpus are not pinned and affine wakeups are enabled there is a 0% 
chance that vcpus will be co-scheduled.

Keeping both vcpus on the same core actually makes sense since they can 
communicate through the local cache faster than across cores.  What we 
need is to make sure that they don't spin.

Windows 2008 can report spinlock spinning through a hypercall.  Can you 
hook to that interface and see if it happens regularly?  Altenatively 
use a PLE capable host and trace the kvm_vcpu_on_spin() function.

-- 
error compiling committee.c: too many arguments to function

--

[ message continues ]

We only tried windows 2003 for the experiments, and have no data related to windows 2008. 
But maybe we can have  a try later.  Anyway, the key point is we have to enhance the scheduler to let it 
Know which threads are vcpu threads to avoid perf loss in this case.
Xiantao
--

[ message continues ]

I have two worries about this approach:

1.  Affine wakeups were introduced for a reason; if we disable them 
(even just for vcpus), we lost something.  Maybe we can tune the 
mechanism not to fail, instead of disabling it.

2.  Affine wakeups are a scheduler internal detail.  How do we explain 
what it does?  the scheduler may not have affine wakeups in a few years, 
yet we'll have an ABI to disable them.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

--

[ message continues ]

It's a priority transfer (in CFS terms, vruntime) (we don't know who 

The idea is to increase the chances to the target vcpu to run, and to 

Certainly that has even greater potential for Linux guests.  Note that 
we spin on mutexes now, so we need to prevent preemption while the lock 
owner is running.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--

[ message continues ]

either that, or disable spinning on (para) virt kernels. Para virt
kernels could possibly extend the thing by also checking to see if the
owner's vcpu is running.

--

[ message continues ]

I suspect we will need a combination of both approaches, given that we will not
be able to avoid preempting guests in their critical section always (too long
critical sections or real-time tasks wanting to preempt). Other idea is to
gang-schedule VCPUs of the same guest as much as possible?

- vatsa
--

[ message continues ]

Except gang scheduling is a scalability nightmare waiting to happen. I
much prefer this hint thing.
--

[ message continues ]

Gang-scheduling maybe the ideal solution to solve the issue, and has to change host's scheduler a lot to implement it, and it maybe hard to be upstream.  So can we figure out an easy way(maybe not best) for this ? 
Xiantao
--

[ message continues ]

What would you do instead?


Certainly that's worth doing.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

--

[ message continues ]