A new feature that will first be availble in the upcoming 2.6.20 kernel is KVM, a Kernel-based Virtual Machine. The project's webpage describes KVM as, "a full virtualization solution for Linux on x86 hardware. It consists of a loadable kernel module (kvm.ko) and a userspace component. Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc." The project's FAQ explains that the functionality requires "an x86 machine running a recent Linux kernel on an Intel processor with VT (virtualization technology) extensions, or an AMD processor with SVM extensions (also called AMD-V)." The userland aspect of KVM is a slighlty modified version of qemu, used to instantiate the virtual machine.

Ingo Molnar [interview] announced a new patch introducing paravirtualization support for KVM, outdating the KVM FAQ which in comparing KVM to Xen notes, "Xen supports both full virtualization and a technique called paravirtualization, which allows better performance for modified guests. kvm does not at present support paravirtualization." In describing his patch which is against the 2.6.20-rc3 + KVM trunk kernel, Ingo said it, "includes support for the hardware cr3-cache feature of Intel-VMX CPUs. (which speeds up context switches and TLB flushes)". He went on to add, "some aspects of the code are still a bit ad-hoc and incomplete, but the code is stable enough in my testing and i'd like to have some feedback." In a series of benchmarks, he found 2-task context switch performance to be improved by a factor of four, while "hackbench 1" showed twice as good performance, and "hackbench 5" showed a 30% improvement. His email goes on to detail how the paravirtualization works.

From: Ingo Molnar [email blocked]
To: kvm-devel [email blocked]
Subject: [announce] [patch] KVM paravirtualization for Linux
Date:	Fri, 5 Jan 2007 22:52:23 +0100


i'm pleased to announce the first release of paravirtualized KVM (Linux 
under Linux), which includes support for the hardware cr3-cache feature 
of Intel-VMX CPUs. (which speeds up context switches and TLB flushes)

the patch is against 2.6.20-rc3 + KVM trunk and can be found at:

   http://redhat.com/~mingo/kvm-paravirt-patches/

Some aspects of the code are still a bit ad-hoc and incomplete, but the 
code is stable enough in my testing and i'd like to have some feedback. 

Firstly, here are some numbers:

2-task context-switch performance (in microseconds, lower is better):

 native:                       1.11
 ----------------------------------
 Qemu:                        61.18
 KVM upstream:                53.01
 KVM trunk:                    6.36
 KVM trunk+paravirt/cr3:       1.60

i.e. 2-task context-switch performance is faster by a factor of 4, and 
is now quite close to native speed!

"hackbench 1" (utilizes 40 tasks, numbers in seconds, lower is better):

 native:                       0.25
 ----------------------------------
 Qemu:                         7.8
 KVM upstream:                 2.8
 KVM trunk:                    0.55
 KVM paravirt/cr3:             0.36

almost twice as fast.

"hackbench 5" (utilizes 200 tasks, numbers in seconds, lower is better):

 native:                       0.9
 ----------------------------------
 Qemu:                        35.2
 KVM upstream:                 9.4
 KVM trunk:                    2.8
 KVM paravirt/cr3:             2.2

still a 30% improvement - which isnt too bad considering that 200 tasks 
are context-switching in this workload and the cr3 cache in current CPUs 
is only 4 entries.

the patchset does the following:

- it provides an ad-hoc paravirtualization hypercall API between a Linux 
  guest and a Linux host. (this will be replaced with a proper
  hypercall later on.)

- using the hypercall API it utilizes the "cr3 target cache" feature in 
  Intel VMX CPUs, and extends KVM to make use of that cache. This 
  feature allows the avoidance of expensive VM exits into hypervisor 
  context. (The guest needs to be 'aware' and the cache has to be
  shared between the guest and the hypervisor. So fully emulated OSs
  wont benefit from this feature.)

- a few simpler paravirtualization changes are done for Linux guests: IO 
  port delays do not cause a VM exit anymore, the i8259A IRQ controller 
  code got simplified (this will be replaced with a proper, hypercall
  based and host-maintained IRQ controller implementation) and TLB 
  flushes are more efficient, because no cr3 reads happen which would 
  otherwise cause a VM exit. These changes have a visible effect
  already: they reduce qemu's CPU usage when a guest idles in HLT, by 
  about 25%. (from ~20% CPU usage to 14% CPU usage if an -rt guest has 
  HZ=1000)

Paravirtualization is triggered via the kvm_paravirt=1 boot option (for 
now, this too is ad-hoc) - if that is passed then the KVM guest will 
probe for paravirtualization availability on the hypervisor side - and 
will use it if found. (If the guest does not find KVM-paravirt support 
on the hypervisor side then it will continue as a fully emulated guest.)

Issues: i only tested this on 32-bit VMX. (64-bit should work with not 
too many changes, the paravirt.c bits can be carried over to 64-bit 
almost as-is. But i didnt want to spread the code too wide.)

Comments, suggestions are welcome!

	Ingo



From: Zachary Amsden [email blocked]
Subject: Re: [announce] [patch] KVM paravirtualization for Linux
Date:	Fri, 05 Jan 2007 14:15:19 -0800

Ingo Molnar wrote:
> i'm pleased to announce the first release of paravirtualized KVM (Linux 
> under Linux), which includes support for the hardware cr3-cache feature 
> of Intel-VMX CPUs. (which speeds up context switches and TLB flushes)
>
> the patch is against 2.6.20-rc3 + KVM trunk and can be found at:
>
>    http://redhat.com/~mingo/kvm-paravirt-patches/
>
> Some aspects of the code are still a bit ad-hoc and incomplete, but the 
> code is stable enough in my testing and i'd like to have some feedback. 
>   

Your code looks generally good.  I have some comments.

You can't do this, even though you want to:

-EXPORT_SYMBOL(paravirt_ops);
+EXPORT_SYMBOL_GPL(paravirt_ops);


The problem is it makes all modules GPL - or at least all modules that 
use any kind of locking, pull in the basic definitions to enable and 
disable interrupts, thus the paravirt_ops symbol, so basically all modules.

What you really want is more like EXPORT_SYMBOL_READABLE_GPL(paravirt_ops);

But I'm not sure that is technically feasible yet.

The kvm code should probably go in kvm.c instead of paravirt.c.


Index: linux/drivers/serial/8250.c
===================================================================
--- linux.orig/drivers/serial/8250.c
+++ linux/drivers/serial/8250.c
@@ -1371,7 +1371,7 @@ static irqreturn_t serial8250_interrupt(
 
 		l = l->next;
 
-		if (l == i->head && pass_counter++ > PASS_LIMIT) {
+		if (!kvm_paravirt 



Is this a bug that might happen under other virtualizations as well, not 
just kvm? Perhaps it deserves a disable feature instead of a kvm 
specific check.

Which also gets rid of the need for this unusually placed extern:

Index: linux/include/linux/sched.h
===================================================================
--- linux.orig/include/linux/sched.h
+++ linux/include/linux/sched.h
@@ -1911,6 +1911,11 @@ static inline void set_task_cpu(struct t
 
 #endif /* CONFIG_SMP */
 
+/*
+ * Is paravirtualization active?
+ */
+extern int kvm_paravirt;

+



From: Ingo Molnar [email blocked]
Subject: Re: [announce] [patch] KVM paravirtualization for Linux
Date:	Fri, 5 Jan 2007 23:30:09 +0100


* Zachary Amsden [email blocked] wrote:

> What you really want is more like 
> EXPORT_SYMBOL_READABLE_GPL(paravirt_ops);

yep. Not a big issue - what is important is to put the paravirt ops into 
the read-only section so that it's somewhat harder for rootkits to 
modify. (Also, it needs to be made clear that this is fundamental, 
lowlevel system functionality written by people under the GPLv2, so that 
if you utilize it beyond its original purpose, using its internals, you 
likely create a work derived from the kernel. Something simple as irq 
disabling probably doesnt qualify, and that we exported to modules for a 
long time, but lots of other details do. So the existence of 
paravirt_ops isnt a free-for all.)

> But I'm not sure that is technically feasible yet.
> 
> The kvm code should probably go in kvm.c instead of paravirt.c.

no. This is fundamental architecture boot code, not module code. kvm.c 
should eventually go into kernel/ and arch/*/kernel, not the other way 
around.

> Index: linux/drivers/serial/8250.c
> ===================================================================
> --- linux.orig/drivers/serial/8250.c
> +++ linux/drivers/serial/8250.c
> @@ -1371,7 +1371,7 @@ static irqreturn_t serial8250_interrupt(
> 
> 		l = l->next;
> 
> -		if (l == i->head && pass_counter++ > PASS_LIMIT) {
> +		if (!kvm_paravirt 
> 
> Is this a bug that might happen under other virtualizations as well, 
> not just kvm? Perhaps it deserves a disable feature instead of a kvm 
> specific check.

yes - this limit is easily triggered via the KVM/Qemu virtual serial 
drivers. You can think of "kvm_paravirt" as "Linux paravirt", it's just 
a flag.

	Ingo



From: Anthony Liguori [email blocked]
Subject: Re: [kvm-devel] [announce] [patch] KVM paravirtualization for Linux
Date:	Fri, 05 Jan 2007 17:02:28 -0600

This is pretty cool.  I've read the VT spec a number of times and never 
understood why they included the CR3 caching :-)  I suspect that you may 
even be faster than Xen for context switches because of the hardware 
assistance here.  Any chance you can run your benchmarks against Xen?

You may already know this, but there isn't a CR3 cache in SVM AFAIK so a 
fair bit of the code will probably have to be enabled conditionally.  
Otherwise, I don't think SVM support would be that hard.  The only other 
odd bit I noticed was:

> +	magic_val = KVM_API_MAGIC;
> +	para_state->guest_version = KVM_PARA_API_VERSION;
> +	para_state->host_version = -1;
> +	para_state->size = sizeof(*para_state);
> +
> +	asm volatile ("movl %0, %%cr3"
> +			: "=&r" (ret)
> +			: "a" (para_state),
> +			  "0" (magic_val)
> +	);

If I read this correctly, you're using a CR3 write as a hypercall (and 
relying on registers being set/returned that aren't part of the actual 
CR3 move)?  Any reason for not just using vmcall?  It seems a bit less 
awkward.  Even a PIO operation would be a bit cleaner IMHO.  PIO exits 
tend to be fast in VT/SVM so that's an added benefit.

Regards,

Anthony Liguori

• Archive of above thread
• KernelTrap interview with Ingo Molnar