login
Header Space

 
 

Re: 2.6.23-mm1 - list_add corruption in cgroup

Score:
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: Andrew Morton <akpm@...>
Cc: <linux-kernel@...>, Paul Menage <menage@...>
Date: Wednesday, October 17, 2007 - 11:54 am

Hello !

While polling the contents of a cgroup task file, I caught the 
following corruption. Is there a known race (and a fix) or should 
I start digging ? 

the program running in the cgroup is fork/exec intensive:

 while (1) {
        int i, s;    
    
        for (i = 0; i < count; i++)
            if (fork() == 0) 
                execlp("/bin/true", "true", 0);
        
        for (i = 0; i < count; i++) 
            wait(&s);
    }

Thanks for any insights, 

C. 



 list_add corruption. next->prev should be prev (ffffffff80a3f338), but was 0000000000200200. (next=ffff810103dcbe90).
------------[ cut here ]------------
kernel BUG at /home/legoater/linux/2.6.23-mm1/lib/list_debug.c:27!
invalid opcode: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:1e.0/0000:01:01.0/local_cpus
CPU 3 
Modules linked in: ipt_REJECT iptable_filter autofs4 nfs lockd sunrpc tg3 sg joydev ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 2441, comm: bash Not tainted 2.6.23-mm1 #4
RIP: 0010:[<ffffffff80308cda>]  [<ffffffff80308cda>] __list_add+0x27/0x5b
RSP: 0018:ffff810103d87dd8  EFLAGS: 00010296
RAX: 0000000000000079 RBX: ffff810105033040 RCX: 0000000000000079
RDX: ffff810103d960c0 RSI: 0000000000000001 RDI: 0000000000000096
RBP: ffff810103d87dd8 R08: 0000000000000002 R09: ffff810008123780
R10: 0000000000000000 R11: ffff810103d87a98 R12: 0000000000000000
R13: ffff810105033040 R14: ffff810104c11ac0 R15: 0000000000000000
FS:  00007f4e273556f0(0000) GS:ffff81010011a840(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006ca2f8 CR3: 0000000103d82000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 2441, threadinfo ffff810103d86000, task ffff810103d960c0)
last branch before last exception/interrupt
 from  [<ffffffff80235885>] printk+0x68/0x69
 to  [<ffffffff80308cda>] __list_add+0x27/0x5b
Stack:  ffff810103d87de8 ffffffff80308d1a ffff810103d87e08 ffffffff802606bf
 ffff810103d87e08 0000000000000000 ffff810103d87ea8 ffffffff80233dca
 ffff810103ddf340 00007f4e27355780 0000000000000000 ffff810103d87f58
Call Trace:
 [<ffffffff80308d1a>] list_add+0xc/0xe
 [<ffffffff802606bf>] cgroup_post_fork+0x41/0x52
 [<ffffffff80233dca>] copy_process+0x12d0/0x143a
 [<ffffffff8020b9b5>] tracesys+0xdc/0xe1
 [<ffffffff80234095>] do_fork+0x76/0x203
 [<ffffffff802679cc>] audit_syscall_entry+0x148/0x17e
 [<ffffffff8020b9b5>] tracesys+0xdc/0xe1
 [<ffffffff80209dd5>] sys_clone+0x23/0x25
 [<ffffffff8020bb67>] ptregscall_common+0x67/0xb0

INFO: lockdep is turned off.

Code: 0f 0b eb fe 4c 8b 00 49 39 f0 74 18 48 89 c1 4c 89 c2 48 c7 
RIP  [<ffffffff80308cda>] __list_add+0x27/0x5b
 RSP <ffff810103d87dd8>
BUG: soft lockup - CPU#1 stuck for 11s! [true:2030]
CPU 1:
Modules linked in: ipt_REJECT iptable_filter autofs4 nfs lockd sunrpc tg3 sg joydev ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 2030, comm: true Tainted: G      D 2.6.23-mm1 #4
RIP: 0010:[<ffffffff80306baf>]  [<ffffffff80306baf>] __write_lock_failed+0xf/0x20
RSP: 0018:ffff81010513be80  EFLAGS: 00000287
RAX: 0000000000000001 RBX: ffff81010513be98 RCX: ffffffff807d8d60
RDX: 0000000000000037 RSI: 0000000000000037 RDI: ffffffff805beac0
RBP: ffff81010289e040 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff8026072c R11: ffff81010513be08 R12: ffff81000812c300
R13: ffff81010289e040 R14: ffff81010513a000 R15: ffff810087acb000
FS:  0000000000000000(0000) GS:ffff8101000560c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8171b028b0 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff80308a1d>] _raw_write_lock+0x6c/0x8b
 [<ffffffff8026072c>] cgroup_exit+0x5c/0xc3
 [<ffffffff80474803>] _write_lock+0x2d/0x31
 [<ffffffff8026072c>] cgroup_exit+0x5c/0xc3
 [<ffffffff802383c1>] do_exit+0x2a0/0x7a5
 [<ffffffff80238955>] sys_exit_group+0x0/0x14
 [<ffffffff80238967>] sys_exit_group+0x12/0x14
 [<ffffffff8020b9b5>] tracesys+0xdc/0xe1
-
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
2.6.23-mm1, Andrew Morton, (Fri Oct 12, 12:31 am)
kernel panic when running tcpdump, Mariusz Kozlowski, (Mon Oct 22, 2:40 pm)
Re: kernel panic when running tcpdump, Andrew Morton, (Mon Oct 22, 3:03 pm)
Re: kernel panic when running tcpdump, Mariusz Kozlowski, (Mon Oct 22, 5:16 pm)
mysqld prevents s2ram [Re: 2.6.23-mm1], Mattia Dongili, (Sun Oct 21, 1:58 am)
Re: mysqld prevents s2ram [Re: 2.6.23-mm1], Pavel Machek, (Sun Oct 21, 5:58 am)
Re: mysqld prevents s2ram [Re: 2.6.23-mm1], Rafael J. Wysocki, (Sun Oct 21, 7:53 am)
Re: mysqld prevents s2ram [Re: 2.6.23-mm1], Mattia Dongili, (Sun Oct 21, 2:28 am)
Re: 2.6.23-mm1 - list_add corruption in cgroup, Cedric Le Goater, (Wed Oct 17, 11:54 am)
Re: 2.6.23-mm1 - list_add corruption in cgroup, Paul Menage, (Fri Oct 19, 6:11 pm)
Re: 2.6.23-mm1 - list_add corruption in cgroup, Paul Menage, (Thu Oct 18, 11:56 am)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Laurent Riffard, (Sun Oct 14, 6:34 pm)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Christoph Hellwig, (Mon Oct 15, 4:40 am)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Laurent Riffard, (Mon Oct 15, 3:51 pm)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Jeff Mahoney, (Mon Oct 15, 2:31 pm)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Laurent Riffard, (Mon Oct 15, 4:06 pm)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Jeff Mahoney, (Mon Oct 15, 4:23 pm)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Christoph Hellwig, (Wed Oct 17, 4:59 am)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Christoph Hellwig, (Wed Oct 17, 4:58 am)
Re: 2.6.23-mm1: BUG in reiserfs_delete_xattrs, Jeff Mahoney, (Wed Oct 17, 10:55 am)
[2.6.23-mm1] CONFIG_LOCALVERSION handling broken, Tilman Schmidt, (Sat Oct 13, 6:11 pm)
Re: [2.6.23-mm1] CONFIG_LOCALVERSION handling broken, Tilman Schmidt, (Wed Oct 17, 7:06 pm)
Re: [2.6.23-mm1] CONFIG_LOCALVERSION handling broken, Tilman Schmidt, (Sat Oct 27, 11:19 am)
Re: [2.6.23-mm1] CONFIG_LOCALVERSION handling broken, Sam Ravnborg, (Sat Oct 27, 11:28 am)
Re: [2.6.23-mm1] CONFIG_LOCALVERSION handling broken, Sam Ravnborg, (Wed Oct 17, 4:27 pm)
Re: 2.6.23-mm1, Gabriel C, (Sat Oct 13, 1:12 pm)
Re: 2.6.23-mm1, Andrew Morton, (Sat Oct 13, 2:01 pm)
Re: 2.6.23-mm1, Gabriel C, (Sat Oct 13, 2:08 pm)
Re: 2.6.23-mm1, Dave Hansen, (Mon Oct 15, 12:28 pm)
Re: 2.6.23-mm1 - build failure with advansys, Kamalesh Babulal, (Sat Oct 13, 12:44 am)
Re: 2.6.23-mm1 - build failure with advansys, Andrew Morton, (Sat Oct 13, 2:52 am)
Re: 2.6.23-mm1 - build failure with advansys, Paul Mackerras, (Wed Oct 17, 8:07 pm)
Re: 2.6.23-mm1 - build failure with advansys, Matthew Wilcox, (Wed Oct 17, 9:48 pm)
Re: 2.6.23-mm1 - Build failure on rgmii, Kamalesh Babulal, (Sat Oct 13, 12:35 am)
Re: 2.6.23-mm1, Laurent Riffard, (Fri Oct 12, 4:38 pm)
Re: 2.6.23-mm1, Andrew Morton, (Fri Oct 12, 5:00 pm)
Re: 2.6.23-mm1, Zan Lynx, (Mon Oct 15, 12:13 pm)
Re: 2.6.23-mm1 - build failure on axonram, Kamalesh Babulal, (Fri Oct 12, 3:44 am)
oops in lbmIODone, fails to boot [Re: 2.6.23-mm1], Mattia Dongili, (Sat Oct 20, 12:57 am)
Re: oops in lbmIODone, fails to boot [Re: 2.6.23-mm1], Andrew Morton, (Sat Oct 20, 1:34 am)
Re: oops in lbmIODone, fails to boot [Re: 2.6.23-mm1], Dave Kleikamp, (Sat Oct 20, 8:18 am)
Re: oops in lbmIODone, fails to boot [Re: 2.6.23-mm1], Mattia Dongili, (Sun Oct 21, 1:44 am)
Re: 2.6.23-mm1 - autofs broken, Rik van Riel, (Sat Oct 20, 1:13 am)
Re: 2.6.23-mm1 - autofs broken, Ian Kent, (Sun Oct 21, 11:45 pm)
Re: 2.6.23-mm1 - autofs broken, Rik van Riel, (Mon Oct 22, 12:46 pm)
Re: 2.6.23-mm1 - autofs broken, Andrew Morton, (Sat Oct 20, 1:39 am)
Re: 2.6.23-mm1 - autofs broken, Rik van Riel, (Sat Oct 20, 1:54 am)
Re: 2.6.23-mm1 - autofs broken, Rik van Riel, (Sat Oct 20, 1:54 am)
Re: 2.6.23-mm1 - autofs broken, Rik van Riel, (Sat Oct 20, 10:56 am)
Re: 2.6.23-mm1 - autofs broken, Dave Hansen, (Mon Oct 22, 6:03 pm)
Re: 2.6.23-mm1, KAMEZAWA Hiroyuki, (Wed Oct 17, 3:01 am)
Re: 2.6.23-mm1, Jiri Kosina, (Wed Oct 17, 5:10 am)
Re: 2.6.23-mm1, KAMEZAWA Hiroyuki, (Wed Oct 17, 5:36 am)
Re: 2.6.23-mm1, Jiri Kosina, (Fri Oct 19, 5:54 pm)
Re: 2.6.23-mm1, Jiri Kosina, (Wed Oct 17, 7:42 am)
Re: 2.6.23-mm1, KAMEZAWA Hiroyuki, (Wed Oct 17, 8:33 am)
PIE randomization (was Re: 2.6.23-mm1), Jiri Kosina, (Fri Oct 19, 5:07 am)
Re: 2.6.23-mm1, Andrew Morton, (Wed Oct 17, 5:02 am)
Suspend Broken (Re: 2.6.23-mm1), Dhaval Giani, (Sat Oct 13, 1:58 pm)
Re: Suspend Broken (Re: 2.6.23-mm1), Rafael J. Wysocki, (Sat Oct 13, 2:33 pm)
Re: Suspend Broken (Re: 2.6.23-mm1), Dhaval Giani, (Sun Oct 14, 12:26 am)
Re: Suspend Broken (Re: 2.6.23-mm1), Rafael J. Wysocki, (Sun Oct 14, 10:19 am)
Re: 2.6.23-mm1 pm_prepare() and _finish() w/ args vs. without, Rafael J. Wysocki, (Sat Oct 13, 1:22 pm)
Re: 2.6.23-mm1 pm_prepare() and _finish() w/ args vs. without, Rafael J. Wysocki, (Sat Oct 13, 3:13 pm)
Re: 2.6.23-mm1 pm_prepare() and _finish() w/ args vs. without, Rafael J. Wysocki, (Sun Oct 14, 4:20 pm)
Re: 2.6.23-mm1 pm_prepare() and _finish() w/ args vs. without, Rafael J. Wysocki, (Mon Oct 15, 4:55 pm)
Re: 2.6.23-mm1, Rafael J. Wysocki, (Fri Oct 12, 5:32 pm)
Re: 2.6.23-mm1, Mark Gross, (Mon Oct 15, 12:09 pm)
Re: 2.6.23-mm1, Rafael J. Wysocki, (Mon Oct 15, 4:40 pm)
Re: 2.6.23-mm1, Mark Gross, (Tue Oct 16, 3:58 pm)
Re: 2.6.23-mm1, Rafael J. Wysocki, (Tue Oct 16, 4:28 pm)
Re: 2.6.23-mm1, Mark Gross, (Tue Oct 16, 7:31 pm)
Re: 2.6.23-mm1, KAMEZAWA Hiroyuki, (Fri Oct 12, 1:03 am)
Re: 2.6.23-mm1, Cedric Le Goater, (Fri Oct 12, 2:48 am)
Re: 2.6.23-mm1, Andrew Morton, (Fri Oct 12, 2:42 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Fri Oct 12, 4:31 am)
Re: 2.6.23-mm1, Andrew Morton, (Fri Oct 12, 4:37 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 4:01 am)
Re: 2.6.23-mm1, Jeff Garzik, (Sat Oct 13, 6:55 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 8:03 am)
Re: 2.6.23-mm1, Jeff Garzik, (Sat Oct 13, 8:19 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 10:32 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 10:40 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 11:13 am)
Re: 2.6.23-mm1, Jeff Garzik, (Sat Oct 13, 1:48 pm)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 2:05 pm)
Re: 2.6.23-mm1, Jeff Garzik, (Sat Oct 13, 2:41 pm)
Re: 2.6.23-mm1, Andrew Morton, (Sat Oct 13, 2:18 pm)
Re: 2.6.23-mm1, Torsten Kaiser, (Sun Oct 14, 7:54 am)
Re: 2.6.23-mm1, Andrew Morton, (Sun Oct 14, 2:39 pm)
Re: 2.6.23-mm1, Torsten Kaiser, (Sun Oct 14, 3:12 pm)
Re: 2.6.23-mm1, Andrew Morton, (Sun Oct 14, 3:26 pm)
Re: 2.6.23-mm1, Torsten Kaiser, (Sun Oct 14, 3:40 pm)
Re: 2.6.23-mm1, Milan Broz, (Sun Oct 14, 6:03 pm)
Re: 2.6.23-mm1, Jens Axboe, (Mon Oct 15, 2:50 am)
Re: 2.6.23-mm1, Torsten Kaiser, (Sat Oct 13, 2:35 pm)
Re: 2.6.23-mm1, Torsten Kaiser, (Fri Oct 12, 8:46 am)
Re: 2.6.23-mm1, KAMEZAWA Hiroyuki, (Fri Oct 12, 3:25 am)
Re: 2.6.23-mm1, Sam Ravnborg, (Fri Oct 12, 4:36 am)
Re: 2.6.23-mm1, Al Viro, (Fri Oct 12, 2:46 am)
Re: 2.6.23-mm1, Andrew Morton, (Fri Oct 12, 3:13 am)
Re: 2.6.23-mm1 thread exit_group issue, Mathieu Desnoyers, (Fri Oct 12, 3:47 pm)
Re: 2.6.23-mm1 thread exit_group issue, Andrew Morton, (Fri Oct 12, 9:03 pm)
Re: 2.6.23-mm1 thread exit_group issue, Oleg Nesterov, (Sat Oct 13, 7:48 am)
Re: 2.6.23-mm1 thread exit_group issue, Mathieu Desnoyers, (Sun Oct 14, 12:04 am)
Re: 2.6.23-mm1 thread exit_group issue, Oleg Nesterov, (Sat Oct 13, 8:02 am)
Re: 2.6.23-mm1 thread exit_group issue, Andrew Morton, (Sat Oct 13, 1:49 pm)
Re: 2.6.23-mm1 thread exit_group issue, Andrew Morton, (Fri Oct 12, 4:01 pm)
Re: 2.6.23-mm1, Al Viro, (Fri Oct 12, 3:04 pm)
[PATCH net-2.6] uml: hard_header fix, Stephen Hemminger, (Fri Oct 12, 2:06 pm)
Build Failure (Was Re: 2.6.23-mm1), Dhaval Giani, (Fri Oct 12, 5:42 am)
[PATCH] add missing parenthesis in cfe_writeblk() macro, Mariusz Kozlowski, (Fri Oct 12, 2:51 am)
nfs mmap adventure (was: 2.6.23-mm1), Peter Zijlstra, (Mon Oct 15, 8:28 am)
Re: nfs mmap adventure (was: 2.6.23-mm1), David Howells, (Mon Oct 15, 10:06 am)
Re: nfs mmap adventure (was: 2.6.23-mm1), Nick Piggin, (Mon Oct 15, 9:46 pm)
Re: nfs mmap adventure (was: 2.6.23-mm1), David Howells, (Mon Oct 15, 7:27 pm)
Re: nfs mmap adventure (was: 2.6.23-mm1), Trond Myklebust, (Mon Oct 15, 11:51 am)
Re: nfs mmap adventure (was: 2.6.23-mm1), Peter Zijlstra, (Mon Oct 15, 12:38 pm)
Re: nfs mmap adventure (was: 2.6.23-mm1), Trond Myklebust, (Mon Oct 15, 11:43 am)
speck-geostationary