Just updated my machine from 2.6.26.1 to 2.6.26.5, and it does not
wan to boot anymore.  It says "Initializing USB Mass Storage driver.."
(when trying to load usb-storage module) and sits here forever, every
2 minutes showing warnings about stuck processes (traces below).
With 2.6.26.1 or any previous version I never experienced this
problem.  Both 32- and 64-bits kernels behave the same way.
While waiting for usb-storage to complete, no other USB devices
works (incl. mouse and keyboard).

The machine is an AMD X2-64 (BE-2400) processor with an nVidia MCP51
chipset (lspci is below).  Motherboard is Asus M2NPV-VM, latest BIOS
(1401, dated 08/07/2008)

I'm rebuilding 2.6.26.3 for now -- trying to bisect the issue...

Thanks.

/mjt

Example of dmesg output:

Sep 13 16:04:51 gandalf kernel: INFO: task insmod:2662 blocked for more than 120 seconds.
Sep 13 16:04:51 gandalf kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:04:51 gandalf kernel: insmod        D c01d7662     0  2662   2580
Sep 13 16:04:51 gandalf kernel:        f7241340 00000082 00000000 c01d7662 00000000 f7228d38 c03dddac f7c0b6c0 
Sep 13 16:04:51 gandalf kernel:        00000001 f7f07dc0 c0399080 f7fe4584 00000008 00001f00 f77925d0 f7792734 
Sep 13 16:04:51 gandalf kernel:        00000001 f7228d70 c03dddac f7792734 c205e7ac c01c24e0 c205e7ac f7c28000 
Sep 13 16:04:51 gandalf kernel: Call Trace:
Sep 13 16:04:51 gandalf kernel:  [<c01d7662>] idr_get_empty_slot+0xe2/0x260
Sep 13 16:04:51 gandalf kernel:  [<c01c24e0>] sysfs_ilookup_test+0x0/0x10
Sep 13 16:04:51 gandalf kernel:  [<c02c642d>] schedule_timeout+0x7d/0xb0
Sep 13 16:04:51 gandalf kernel:  [<c01c27b1>] sysfs_find_dirent+0x21/0x30
Sep 13 16:04:51 gandalf kernel:  [<c02c6da5>] __down+0x55/0x90
Sep 13 16:04:51 gandalf kernel:  [<c0140b2a>] down+0x3a/0x40
Sep 13 16:04:51 gandalf kernel:  [<c0233863>] __driver_attach+0x23/0x80
Sep 13 16:04:51 gandalf kernel:  [<c0233064>] bus_for_each_dev+0x44/0x70
Sep 13 16:04:51 gandalf ...
[ message continues ]

[...]

And it's more: even without usb-storage, other USB devices does
not work, waiting in the same usb_start_wait_urb.

Ok, it turned out to be pretty.. interesting.

It's not a regression within -stable kernel series.  The problem
happens when BOTH are true:

   - using latest BIOS for this mobo (1401), AND
   - enabling KVM_GUEST in kernel.

The only difference in my kernel config between .1 and .5 was
to enable KVM_GUEST and KVM_CLOCK, and also I2C_HELPER_AUTO
which was added by kernel update.

Going with previous BIOS (1201) OR disabling PARAVIRT_GUEST
fixes the problem.  I'll try to figure out which config
--

[ message continues ]

Michael Tokarev wrote:

[usb-storage hangs on asus M2NPV-VM (nVidia GeForce 6150/nForce 430)

After seeing similar situation on another motherboard, I
become curious.

Here's the result:  M3A78-EM motherboard (also from Asus),
AMD780G/SB700 etc.  Updating to one of the latest bios
makes the system unbootable.  Kernel hangs on boot right
after displaying

  SLUB: Genslabs=12, HWalign=64, Order=0-1, MinObjects=4, CPUs=2, Nodes=1

Reverting to the older BIOS or disabling PARAVIRT_GUEST in kernel
fixes this (turning off KVM_GUEST isn't enough), and kernel happily
boots further, with next line being

  hpet clockevent registered
  ...

Reportedly, another Asus motherboard (M2N-SLI DELUXE) also hangs
with new BIOS and KVM_GUEST optimizations turned on, but not on
older BIOS.

So it's quite.. funny.  At least SOME latest BIOSes from Asustec,
together with PARAVIRT_GUEST, -- a problem.  Don't do either of
those, and it's all ok.

The question is if Asus introduced the same bug in quite.. some
of their recent BIOSes, or PARAVIRT_GUEST code is somehow buggy
and the bug is exposed by new BIOS code.

Any hints on this? ;)

Thanks!

/mjt
--

[ message continues ]

PARAVIRT_GUEST doesn't enable anything by itself; it just makes other
config options appear.  What other config options are you enabling?

That is, what's the diff between a working and a non-working .config?

    J
--

[ message continues ]

Jeremy Fitzhardinge wrote:

I stand corrected.  I meant to say CONFIG_PARAVIRT, not CONFIG_PARAVIRT_GUEST:

# diff -U0 config.working config.bad
--- config.working	2008-09-16 22:09:14.844324435 +0400
+++ .config.bad	2008-09-16 22:10:04.914320258 +0400
@@ -4 +4 @@
-# Tue Sep 16 21:47:31 2008
+# Tue Sep 16 22:10:04 2008
@@ -181 +181 @@
-# CONFIG_PARAVIRT is not set
+CONFIG_PARAVIRT=y

/mjt
--

[ message continues ]

Two more Asus motherboards expose exactly the same problem.

M3A-H/HDMI - AMD 780G/SB700, and
M2N-VM - nVidia MCP67 /  GeForce 7050

And from the above, 3 more:

M2NPV-VM - nVidia GeForce 6150/nForce 430
M3A78-EM - AMD780G/SB700
M2N-SLI DELUXE - nVidia MCP55 (unconfirmed)

With new BIOSes and with CONFIG_PARAVIRT enabled in kernel,
system does not boot.  It seems the stalls happens at more
or less random places, but the place is always the same on
the same motherboard and different kernels -- f.e. my
M3A78-EM always stops after "SLUB: " line as shown above,

--

[ message continues ]

Hm, mysterious.  CONFIG_PARAVIRT should have no functional effect on a
native boot.

Could you post the complete dmesg output (with "initcall_debug" on the
kernel command line), with both CONFIG_PARAVIRT on and off, and ideally,
comparing working vs non-working BIOS?

As a separate experiment (which is really grasping at straws), does

Thanks,
    J
--

[ message continues ]

It turned out to be more difficult than I thought.  There were other
problems with the motherboard I experimented with (my new home machine),
unrelated.  Also, it wasn't possible to re-flash bios back as it said
that the old version is incompatible with currently running bios so it
can't re-flash it.

Also, initcall_debug does not make any difference for my mobo.  All the
debugging shown by this option are AFTER the place where it hangs.
And for netconsole I'd need another machine at home, which I don't
have currently.

Instead, while debugging other issues, I tried 2.6.27-rc7 kernel -- this
one works on 2 motherboards with new BIOSes, no matter which PARAVIRT
options I turn on.  I'll try this kernel on other mobos where possible,
and if no issue will be found, I'd declare the issue to be solved.

If not, I'll try to find some time to debug it later.  After all, this
very issue is already worked around -- I enabled those options in order
to add (optimized) support for KVM guests, but for those I just built
another kernel flavour with only that config change.

Oh well.

And thank you for your support!

/mjt
--

[ message continues ]