Re: Linux 2.6.25-rc4

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Ingo Molnar
Date: Thursday, March 6, 2008 - 2:00 am

* Linus Torvalds <torvalds@linux-foundation.org> wrote:


hm, tonight's randconfig bootrun produced a failing (soft-hung) kernel 
after about 120 iterations - and the log i captured _seems_ to indicate 
some block IO (or libata) completion weirdness.

unfortunately, it's not readily reproducible, and i triggered it with 
about 100 sched.git and 300 x86.git patches applied. BUT, virtually the 
same 100+300 patches queue produced a successful 1000+ randconfig 
testrun over the last weekend so i'm reasonably sure the regression is 
new and came in via upstream. Also, the config is UP (and it's a rather 
simple config in other aspects as well), so this must be something 
rather fundamental, not an SMP race.

I just spent about an hour trying to figure out a pattern but the bug 
just doesnt reproduce after 20 bootup attempts with the same bzImage. 
When it hung then it hung for hours, so the condition is permanent.

I've attached the bootup log which includes the SysRq-T output and the 
config. The hang seems to occur because an rc.sysinit task is not coming 
back from io_schedule():

rc.sysinit    D f75bcc24     0  1922   1893
       f761c810 00000086 f75bcd38 f75bcc24 1954bff5 00000015 f7746000 f761c974 
       f761c974 f7c17698 c180e7a8 f7747cc4 00000000 f7747ccc c180e7a8 c097bff7 
       c01a3acb c097c27d c01a3aa0 f7872a90 00000002 c01a3aa0 f7747e48 c097c2fc 
Call Trace:
 [<c097bff7>] io_schedule+0x37/0x70
 [<c01a3acb>] sync_buffer+0x2b/0x30
 [<c097c27d>] __wait_on_bit+0x4d/0x80
 [<c01a3aa0>] sync_buffer+0x0/0x30
 [<c01a3aa0>] sync_buffer+0x0/0x30
 [<c097c2fc>] out_of_line_wait_on_bit+0x4c/0x60
 [<c0142340>] wake_bit_function+0x0/0x40
 [<c01a3a51>] __wait_on_buffer+0x21/0x30
 [<c0209915>] ext3_bread+0x55/0x70
 [<c020cff8>] ext3_find_entry+0x258/0x660
 [<c03a0026>] avc_has_perm+0x46/0x50
 [<c03a0d14>] inode_has_perm+0x44/0x80
 [<c020de69>] ext3_lookup+0x29/0xa0
 [<c0189f90>] do_lookup+0x130/0x180
 [<c018b540>] __link_path_walk+0x340/0xd50
 [<c03a0d14>] inode_has_perm+0x44/0x80
 [<c018bf8a>] link_path_walk+0x3a/0xa0
 [<c016feb4>] __do_fault+0x1a4/0x3d0
 [<c018c1b7>] do_path_lookup+0x77/0x210
 [<c018cb57>] __user_walk_fd+0x27/0x40
 [<c01860d5>] vfs_stat_fd+0x15/0x40
 [<c016feb4>] __do_fault+0x1a4/0x3d0
 [<c01861ef>] sys_stat64+0xf/0x30
 [<c0125a5d>] do_page_fault+0x2ad/0x670
 [<c03db6cc>] trace_hardirqs_on_thunk+0xc/0x10
 [<c0115a5f>] sysenter_past_esp+0x5f/0x90
 =======================

So the last known-good kernel would be last Friday's -git:

  commit d395991c117d43bfca97101a931a41d062a93852
  Merge: b73384f... b445c56...
  Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
  Date:   Fri Feb 29 16:54:33 2008 -0800

but ... "git-log d395991c117d4.. block/" does not show anything 
particularly exciting. Note that the IO scheduler in question is:

  CONFIG_DEFAULT_IOSCHED="anticipatory"

so it's not the usual CFQ - that's due to randconfig.

	Ingo
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 4, 10:03 pm)
Re: Linux 2.6.25-rc4, FUJITA Tomonori, (Wed Mar 5, 1:09 am)
Re: Linux 2.6.25-rc4, Grant Grundler, (Wed Mar 5, 9:46 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 2:00 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 5:59 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 6:06 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 6:12 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 6:33 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Thu Mar 6, 6:38 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 6:55 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Thu Mar 6, 7:06 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Thu Mar 6, 2:17 pm)
Re: Linux 2.6.25-rc4, Jens Axboe, (Fri Mar 7, 1:48 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Fri Mar 7, 1:53 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Fri Mar 7, 1:57 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Fri Mar 7, 2:02 am)
Re: Linux 2.6.25-rc4, Paul Mackerras, (Fri Mar 7, 2:59 am)
Re: Linux 2.6.25-rc4, Valdis.Kletnieks, (Fri Mar 7, 8:20 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Fri Mar 7, 3:04 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sat Mar 8, 1:22 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sat Mar 8, 2:05 pm)
Re: Linux 2.6.25-rc4, Pavel Machek, (Sat Mar 8, 4:36 pm)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Sun Mar 9, 4:59 am)
Re: Linux 2.6.25-rc4, Andi Kleen, (Sun Mar 9, 5:55 am)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Jiri Slaby, (Sun Mar 9, 6:41 am)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Rafael J. Wysocki, (Sun Mar 9, 3:49 pm)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Jiri Slaby, (Sun Mar 9, 4:04 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 10, 1:55 am)
Re: Linux 2.6.25-rc4, Pavel Machek, (Mon Mar 10, 3:10 am)
Re: Linux 2.6.25-rc4, Andi Kleen, (Mon Mar 10, 4:52 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Mon Mar 10, 5:36 am)
Re: Linux 2.6.25-rc4, Rafael J. Wysocki, (Mon Mar 10, 6:10 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 10, 6:19 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Mon Mar 10, 6:56 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Mon Mar 10, 7:04 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 7:01 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 7:29 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 7:29 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 8:14 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 9:56 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 10:13 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 11:07 am)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 11:13 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 11:18 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 11:23 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 11:25 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 11:26 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 11:36 am)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 11:44 am)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 11:56 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 12:08 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 12:39 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Sun Mar 16, 12:54 pm)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 1:31 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 2:06 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 3:59 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 4:27 pm)
Re: Linux 2.6.25-rc4, Jens Axboe, (Mon Mar 17, 12:23 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 17, 2:09 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Mon Mar 17, 3:52 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 17, 5:18 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Tue Mar 18, 6:03 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 6:32 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Tue Mar 18, 7:48 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 8:10 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Tue Mar 18, 8:41 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 9:30 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Tue Mar 18, 9:47 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 2:02 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Tue Mar 18, 6:21 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 6:28 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Tue Mar 18, 8:24 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 8:28 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 8:56 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Tue Mar 18, 9:03 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 9:48 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierki ..., (Wed Mar 19, 4:14 am)
Re: Linux 2.6.25-rc4, Alan Cox, (Fri Mar 21, 7:49 am)
Re: Linux 2.6.25-rc4, Mark Lord, (Fri Mar 21, 8:03 am)