Re: Linux 2.6.25-rc4

!MAILaRCHIVE_VOTE_RePLACE
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Linus Torvalds <torvalds@...>
Cc: Linux Kernel Mailing List <linux-kernel@...>, Rafael J. Wysocki <rjw@...>, Jens Axboe <jens.axboe@...>
Date: Thursday, March 6, 2008 - 5:00 am

* Linus Torvalds <torvalds@linux-foundation.org> wrote:


hm, tonight's randconfig bootrun produced a failing (soft-hung) kernel 
after about 120 iterations - and the log i captured _seems_ to indicate 
some block IO (or libata) completion weirdness.

unfortunately, it's not readily reproducible, and i triggered it with 
about 100 sched.git and 300 x86.git patches applied. BUT, virtually the 
same 100+300 patches queue produced a successful 1000+ randconfig 
testrun over the last weekend so i'm reasonably sure the regression is 
new and came in via upstream. Also, the config is UP (and it's a rather 
simple config in other aspects as well), so this must be something 
rather fundamental, not an SMP race.

I just spent about an hour trying to figure out a pattern but the bug 
just doesnt reproduce after 20 bootup attempts with the same bzImage. 
When it hung then it hung for hours, so the condition is permanent.

I've attached the bootup log which includes the SysRq-T output and the 
config. The hang seems to occur because an rc.sysinit task is not coming 
back from io_schedule():

rc.sysinit    D f75bcc24     0  1922   1893
       f761c810 00000086 f75bcd38 f75bcc24 1954bff5 00000015 f7746000 f761c974 
       f761c974 f7c17698 c180e7a8 f7747cc4 00000000 f7747ccc c180e7a8 c097bff7 
       c01a3acb c097c27d c01a3aa0 f7872a90 00000002 c01a3aa0 f7747e48 c097c2fc 
Call Trace:
 [<c097bff7>] io_schedule+0x37/0x70
 [<c01a3acb>] sync_buffer+0x2b/0x30
 [<c097c27d>] __wait_on_bit+0x4d/0x80
 [<c01a3aa0>] sync_buffer+0x0/0x30
 [<c01a3aa0>] sync_buffer+0x0/0x30
 [<c097c2fc>] out_of_line_wait_on_bit+0x4c/0x60
 [<c0142340>] wake_bit_function+0x0/0x40
 [<c01a3a51>] __wait_on_buffer+0x21/0x30
 [<c0209915>] ext3_bread+0x55/0x70
 [<c020cff8>] ext3_find_entry+0x258/0x660
 [<c03a0026>] avc_has_perm+0x46/0x50
 [<c03a0d14>] inode_has_perm+0x44/0x80
 [<c020de69>] ext3_lookup+0x29/0xa0
 [<c0189f90>] do_lookup+0x130/0x180
 [<c018b540>] __link_path_walk+0x340/0xd50
 [<c03a0d14>] inode_has_perm+0x44/0x80
 [<c018bf8a>] link_path_walk+0x3a/0xa0
 [<c016feb4>] __do_fault+0x1a4/0x3d0
 [<c018c1b7>] do_path_lookup+0x77/0x210
 [<c018cb57>] __user_walk_fd+0x27/0x40
 [<c01860d5>] vfs_stat_fd+0x15/0x40
 [<c016feb4>] __do_fault+0x1a4/0x3d0
 [<c01861ef>] sys_stat64+0xf/0x30
 [<c0125a5d>] do_page_fault+0x2ad/0x670
 [<c03db6cc>] trace_hardirqs_on_thunk+0xc/0x10
 [<c0115a5f>] sysenter_past_esp+0x5f/0x90
 =======================

So the last known-good kernel would be last Friday's -git:

  commit d395991c117d43bfca97101a931a41d062a93852
  Merge: b73384f... b445c56...
  Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
  Date:   Fri Feb 29 16:54:33 2008 -0800

but ... "git-log d395991c117d4.. block/" does not show anything 
particularly exciting. Note that the IO scheduler in question is:

  CONFIG_DEFAULT_IOSCHED="anticipatory"

so it's not the usual CFQ - that's due to randconfig.

	Ingo
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Linux 2.6.25-rc4, Linus Torvalds, (Wed Mar 5, 1:03 am)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Jiri Slaby, (Sun Mar 9, 9:41 am)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Rafael J. Wysocki, (Sun Mar 9, 6:49 pm)
Re: [patch] drivers/char/esp.c: fix bootup lockup, Jiri Slaby, (Sun Mar 9, 7:04 pm)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 5:00 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 8:59 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Thu Mar 6, 9:38 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 9:33 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Thu Mar 6, 10:06 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 9:55 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Thu Mar 6, 5:17 pm)
Re: Linux 2.6.25-rc4, Jens Axboe, (Fri Mar 7, 4:48 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Fri Mar 7, 6:04 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sat Mar 8, 4:22 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sat Mar 8, 5:05 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 10, 4:55 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Mon Mar 10, 8:36 am)
Re: Linux 2.6.25-rc4, Rafael J. Wysocki, (Mon Mar 10, 9:10 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Mon Mar 10, 10:04 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 10:01 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 12:56 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 1:13 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 2:18 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 2:07 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 2:13 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 3:54 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 6:59 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Sun Mar 16, 7:27 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 17, 5:09 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Mon Mar 17, 6:52 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 17, 8:18 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Tue Mar 18, 9:03 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 9:32 am)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Tue Mar 18, 11:41 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Tue Mar 18, 9:21 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 9:28 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Tue Mar 18, 11:24 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 11:28 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Tue Mar 18, 11:56 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Wed Mar 19, 12:03 am)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Wed Mar 19, 12:48 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Wed Mar 19, 7:14 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 12:30 pm)
Re: Linux 2.6.25-rc4 , Linus Torvalds, (Tue Mar 18, 12:47 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 5:02 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Tue Mar 18, 10:48 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Tue Mar 18, 11:10 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 2:36 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 3:08 pm)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 2:56 pm)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 3:39 pm)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 4:31 pm)
Re: Linux 2.6.25-rc4, Mark Lord, (Fri Mar 21, 11:03 am)
Re: Linux 2.6.25-rc4, Alan Cox, (Fri Mar 21, 10:49 am)
Re: Linux 2.6.25-rc4, Linus Torvalds, (Sun Mar 16, 5:06 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 2:26 pm)
Re: Linux 2.6.25-rc4, Jens Axboe, (Mon Mar 17, 3:23 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 2:25 pm)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 2:23 pm)
Re: Linux 2.6.25-rc4, Alan Cox, (Sun Mar 16, 2:44 pm)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 10:29 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Sun Mar 16, 10:29 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Sun Mar 16, 11:14 am)
Re: Linux 2.6.25-rc4 , Anders Eriksson, (Mon Mar 10, 9:19 am)
Re: Linux 2.6.25-rc4, Bartlomiej Zolnierkiewicz..., (Mon Mar 10, 9:56 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Thu Mar 6, 9:06 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Fri Mar 7, 4:53 am)
Re: Linux 2.6.25-rc4, Pavel Machek, (Sat Mar 8, 7:36 pm)
Re: Linux 2.6.25-rc4, Andi Kleen, (Sun Mar 9, 8:55 am)
Re: Linux 2.6.25-rc4, Pavel Machek, (Mon Mar 10, 6:10 am)
Re: Linux 2.6.25-rc4, Andi Kleen, (Mon Mar 10, 7:52 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Sun Mar 9, 7:59 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Fri Mar 7, 4:57 am)
Re: Linux 2.6.25-rc4, , (Fri Mar 7, 11:20 am)
Re: Linux 2.6.25-rc4, Ingo Molnar, (Fri Mar 7, 5:02 am)
Re: Linux 2.6.25-rc4, Paul Mackerras, (Fri Mar 7, 5:59 am)
Re: Linux 2.6.25-rc4, Jens Axboe, (Thu Mar 6, 9:12 am)
Re: Linux 2.6.25-rc4, FUJITA Tomonori, (Wed Mar 5, 4:09 am)
Re: Linux 2.6.25-rc4, Grant Grundler, (Wed Mar 5, 12:46 pm)