Home » Mailing list archives » linux-kvm » 2008 » December » 11

Re: How to use PCI-passthrough with kvm

Previous thread: How to use PCI-passthrough with kvm by w1ndoz on Wednesday, December 10, 2008 - 1:07 am. (1 message)

Next thread: hello by w1ndoz on Wednesday, December 10, 2008 - 12:55 am. (1 message)
[view original message]
From: w1ndoz <w1ndoz@...>
To: <kvm@...>
Subject: How to use PCI-passthrough with kvm
Date: Wednesday, December 10, 2008 - 1:17 am

Hi


I'm interested in PCI passthrough support.


I downloaded Linux Kernel 2.6.28-rc7 and build and ran KVM (which

is KVM-79) on this kernel.

I started KVM with the following command-lines:

$ qemu-system-x86_64 -boot c -m 512 -hda test.qcow -localtime -k

en-us -pcidevice host=00:19.0

(00:19.0 is Intel(R) PRO/1000 Network Connection Device)


However, KVM caused the following errors:

Could not notify kernel about assigned device "00:19.0"

register_real_device: Device or resource busy

Segmantation_fault


dmesg messages were as follows:

$ dmesg

[  151.483167] [drm] Initialized drm 1.1.0 20060810

[  151.525100] pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low)

-> IRQ 16

[  151.525105] pci 0000:00:02.0: setting latency timer to 64

[  151.525193] pci 0000:00:02.0: irq 505 for MSI/MSI-X

[  151.525309] [drm] Initialized i915 1.6.0 20080730 on minor 0

[  187.854117] set status page addr 0x00033000

[  823.246792] kvm: guest NX capability removed

[  823.311659] kvm: guest NX capability removed

[  823.487849] qemu-system-x86[5940]: segfault at 1dc ip 0807c446

sp 96420110 error 4 in qemu-system-x86_64[8048000+194000]

[  837.010304] kvm: guest NX capability removed

[  837.019163] e1000e 0000:00:19.0: BAR 0: can't reserve mem

region [0xe0400000-0xe041ffff]

[  837.019163] kvm_vm_ioctl_assign_device: Could not get access to

device regions

[  837.019194] kvm: guest NX capability removed

[  837.219560] qemu-system-x86[5944]: segfault at 1dc ip 0807c446

sp 9636b110 error 4 in qemu-system-x86_64[8048000+194000]


I think that my cpu and my chipset maybe support VT-x and VT-d.

http://www.intel.co.jp/products/desktop/motherboards/dq35jo/dq35jo-overv...

CPU: Intel(R) Core(TM) 2 Duo CPU E6

chipset: Intel(R) Q35 Express Chipset


Any advive or tips will help ?


Thanks,

Kazushi

--
[view original message]
From: Han, Weidong <weidong.han@...>
To: 'w1ndoz' <w1ndoz@...>, 'kvm@vger.kernel.org' <kvm@...>
Subject: RE: How to use PCI-passthrough with kvm
Date: Wednesday, December 10, 2008 - 5:35 am

Hi Kazushi,


Make sure unload the driver of the device before assign it. 


Regards,

Weidong


--
[view original message]
From: w1ndoz <w1ndoz@...>
To: <kvm@...>
Subject: Re: How to use PCI-passthrough with kvm
Date: Wednesday, December 10, 2008 - 10:39 pm

Hi Weidong,


Thank you for your advice

Finally, I could recognize a Intel NIC from the guest OS.


But, I had a new problem.


A guest os have been able to recognize the NIC with

pci-passthrough. However the NIC could not get a IP address on a

guest os


The NIC causes the following errors and stops:

0000:00:05.0: eth1: Detected Tx Unit Hang:


I collected information on webs and I knew that it was a bug of

e1000e drivers, and used a driver of the newest version (e1000e

0.5.8.2.tar.gz), but the problem did not solve it.


Will this error be caused by PCI-passthrough on KVM ?

Of course the NIC works on a host OS normally.


dmesg is as follows:

[  361.424319] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow

Control: RX/TX

[  387.711336] ACPI: PCI interrupt for device 0000:00:05.0 disabled

[  414.760362] e1000e: Intel(R) PRO/1000 Network Driver - 0.5.8.2-NAPI

[  414.760366] e1000e: Copyright (c) 1999-2008 Intel Corporation.

[  414.760853] ACPI: PCI Interrupt 0000:00:05.0[A] -> Link [LNKA] ->

GSI 10 (level, high) -> IRQ 10

[  414.760901] PCI: Setting latency timer of device 0000:00:05.0 to 64

[  414.851311] 0000:00:05.0: : Failed to initialize MSI interrupts.

Falling back to legacy interrupts.

[  415.235355] 0000:00:05.0: eth1: (PCI Express:2.5GB/s:Width x1)

00:19:d1:a3:fa:b3

[  415.235359] 0000:00:05.0: eth1: Intel(R) PRO/1000 Network Connection

[  415.235381] 0000:00:05.0: eth1: MAC: 7, PHY: 6, PBA No: ffffff-0ff

[  415.395924] ADDRCONF(NETDEV_UP): eth1: link is not ready

[  416.887888] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow

Control: RX/TX

[  416.887893] 0000:00:05.0: eth1: 10/100 speed: disabling TSO

[  416.888981] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[  419.124848] 0000:00:05.0: eth1: Detected Tx Unit Hang:

[  419.124850]   TDH                  <0>

[  419.124851]   TDT                  <3>

[  419.124858]   next_to_use          <3>

[  419.124858]   next_to_clean        <0>

[  419.124859] buffer_info[next_to...
[ message continues ]
[view original message]
From: Han, Weidong <weidong.han@...>
To: 'w1ndoz' <w1ndoz@...>, 'kvm@vger.kernel.org' <kvm@...>
Subject: RE: How to use PCI-passthrough with kvm
Date: Thursday, December 11, 2008 - 3:32 am

Hi,


This onboard NIC assignment works for me. I used RHEL5u1 guest and e1000e 0.2.9.5 driver. So I don't think this issue is caused by PCI passthrough on KVM. Is there any error messages of assigning device?


Pls note that the device that shares IRQ with other devices in host cannot work by PCI-passthrough currently. 


Regards,

Weidong


--
[view original message]
From: w1ndoz <w1ndoz@...>
To: <unlisted-recipients@...>, <@...>
Cc: <kvm@...>
Subject: Re: How to use PCI-passthrough with kvm
Date: Thursday, December 11, 2008 - 6:11 am

Hi Weidong,


Thank you for your advice.


The other messages which I found are the following messages which

KVM outputs:

BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1)

BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1)


and, IRQ status is as follows:


/proc/interrupt of the host OS is as follows:

cat /proc/interrupts

           CPU0       CPU1

  0:         45          0   IO-APIC-edge      timer

  1:          1          1   IO-APIC-edge      i8042

  4:          1          1   IO-APIC-edge

  9:          0          0   IO-APIC-fasteoi   acpi

 12:          2          2   IO-APIC-edge      i8042

 14:       8675       8589   IO-APIC-edge      ata_piix

 15:          0          0   IO-APIC-edge      ata_piix

 17:          0          1   IO-APIC-fasteoi   uhci_hcd:usb3, ehci_hcd:usb7

 18:       2201       2200   IO-APIC-fasteoi   uhci_hcd:usb1,

uhci_hcd:usb6, pata_marvell

 19:         43         45   IO-APIC-fasteoi   uhci_hcd:usb5, ohci1394

 21:       1091       1060   IO-APIC-fasteoi   uhci_hcd:usb2, ata_piix, eth1

 23:          2          2   IO-APIC-fasteoi   uhci_hcd:usb4, ehci_hcd:usb8

505:       7198       7310   PCI-MSI-edge      i915@pci:0000:00:02.0

506:         50         53   PCI-MSI-edge      kvm_assigned_msi_device

NMI:          0          0   Non-maskable interrupts

LOC:      87300      87267   Local timer interrupts

RES:      54424      49773   Rescheduling interrupts

CAL:        494        446   Function call interrupts

TLB:        651        625   TLB shootdowns

SPU:          0          0   Spurious interrupts

ERR:          0

MIS:          0


and GuestOS is as follows:

$ cat /proc/interrupts

           CPU0

  0:      42779   IO-APIC-edge      timer

  1:         76   IO-APIC-edge      i8042

  2:          0    XT-PIC-XT        cascade

  4:          1   IO-APIC-edge

  8:          3   IO-APIC-edge      rtc

 10:         68   IO-APIC-edge      eth1

 11:         45   IO-APIC-edge      0, eth0

 12:        228   IO-APIC-edge      i8042

 14:       9982...
[ message continues ]
[view original message]
From: Han, Weidong <weidong.han@...>
To: 'w1ndoz' <w1ndoz@...>
Cc: 'kvm@vger.kernel.org' <kvm@...>
Subject: RE: How to use PCI-passthrough with kvm
Date: Thursday, December 11, 2008 - 6:26 am

It's not related to shared irq. MSI is already supported in KVM, so if the device has MSI capability, there is no sharing irq issue.


Can you try latest kvm.git and kvm-userspace.git? At least you are not using latest kvm-userspace because there is output "BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1)". In addition, can you try out other device, such as add-on PCIe NIC or USB? 


Regards,

Weidong


--
[view original message]
From: w1ndoz <w1ndoz@...>
To: <weidong.han@...>
Cc: <kvm@...>
Subject: Re: How to use PCI-passthrough with kvm
Date: Sunday, December 14, 2008 - 10:27 pm

Hi Weidong,

Thank you for your answer.


I tried to test latest Avi's kvm.git and kvm-userspace.git.

but, kvm-userspace fails to compile.


I used the following git repositories:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm-userspace.git

vtd


$ ./configure

$ make

 make -C libkvm

make[1]: Entering directory `/home/ddk/vtd/libkvm'

gcc -m32 -D__i386__ -MMD -MF ./.libkvm.d -g -fomit-frame-pointer -Wall

 -fno-stack-protector   -I /home/ddk/vtd/kernel/include   -c -o

libkvm.o libkvm.c

In file included from libkvm.c:38:

libkvm.h:630: warning: 'struct kvm_pit_state' declared inside parameter list

libkvm.h:630: warning: its scope is only this definition or

declaration, which is probably not what you want

libkvm.h:641: warning: 'struct kvm_pit_state' declared inside parameter list

gcc -m32 -D__i386__ -MMD -MF ./.libkvm-x86.d -g -fomit-frame-pointer

-Wall  -fno-stack-protector   -I /home/ddk/vtd/kernel/include   -c -o

libkvm-x86.o libkvm-x86.c

In file included from libkvm-x86.c:1:

libkvm.h:630: warning: 'struct kvm_pit_state' declared inside parameter list

libkvm.h:630: warning: its scope is only this definition or

declaration, which is probably not what you want

libkvm.h:641: warning: 'struct kvm_pit_state' declared inside parameter list

libkvm-x86.c:246: warning: 'struct kvm_pit_state' declared inside parameter list

libkvm-x86.c:247: error: conflicting types for 'kvm_get_pit'

libkvm.h:630: error: previous declaration of 'kvm_get_pit' was here

libkvm-x86.c: In function 'kvm_get_pit':

libkvm-x86.c:251: error: invalid application of 'sizeof' to incomplete

type 'struct kvm_pit_state'

libkvm-x86.c:251: error: array type has incomplete element type

libkvm-x86.c:251: error: invalid application of 'sizeof' to incomplete

type 'struct kvm_pit_state'

libkvm-x86.c:251: error: invalid application of 'sizeof' to incomplete

type 'struct kvm_pit_state'

libkvm-x86.c: At top level:

libkvm-x86.c:259: warning: 'struct kvm_pit_state' declared inside parameter list

libkvm-x86.c:260: e...
[ message continues ]
[view original message]
From: Han, Weidong <weidong.han@...>
To: 'w1ndoz' <w1ndoz@...>
Cc: 'kvm@vger.kernel.org' <kvm@...>
Subject: RE: How to use PCI-passthrough with kvm
Date: Sunday, December 14, 2008 - 10:37 pm

I think you need sync with kernel code:


$ ./configure

$ cd kernel

$ make sync LINUX=../../kvm  -- kvm kernel directory

$ cd ..

$ make

$ make install


BTW, you should also need following patch to pass compilation.


---

 kernel/x86/external-module-compat.h |    2 +-

 1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/kernel/x86/external-module-compat.h b/kernel/x86/external-module-compat.h

index b5e11e2..2f16ca7 100644

--- a/kernel/x86/external-module-compat.h

+++ b/kernel/x86/external-module-compat.h

@@ -335,7 +335,7 @@ struct kvm_desc_ptr {

 #define FEATURE_CONTROL_VMXON_ENABLED	(1<<2)

 #endif


-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29)

+#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,28)


 struct mtrr_var_range {

 	u32 base_lo;

--

1.5.1


--
Previous thread: How to use PCI-passthrough with kvm by w1ndoz on Wednesday, December 10, 2008 - 1:07 am. (1 message)

Next thread: hello by w1ndoz on Wednesday, December 10, 2008 - 12:55 am. (1 message)