login
Search
2.6.33.2-rt13 worked fine, however on 2.6.33.3-rt16, when I do reboot, it oops:
# reboot
# Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT 83xx Sys
Modules linked in:
NIP: c00efc68 LR: c00efc38 CTR: 00000000
REGS: ce6e3dc0 TRAP: 0300 Not tainted (2.6.33.3-rt16)
MSR: 00009032 <EE,ME,IR,DR> CR: 24000448 XER: 00000000
DAR: 00000038, DSISR: 20000000
TASK = cd89ccc0[1613] 'umount' THREAD: ce6e2000
GPR00: 00000000 ce6e3e70 cd89ccc0 ce6e3ddc 22222222 00000000 ce6e3e24 ce6e3e04
GPR08: 00008000 00000010 cdfa2130 cdfa26e0 44000442 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 00000001 ce6e3eb8 00000021
GPR24: 00000060 00000000 00000000 ceb94c40 00000000 ceb94cc0 c065781c ce6e3e70
NIP [c00efc68] fs_may_remount_ro+0x6c/0xd8
LR [c00efc38] fs_may_remount_ro+0x3c/0xd8
Call Trace:
[ce6e3e70] [c00efc38] fs_may_remount_ro+0x3c/0xd8 (unreliable)
[ce6e3e90] [c00f1198] do_remount_sb+0x11c/0x164
[ce6e3eb0] [c0113a3c] do_mount+0x538/0x86c
[ce6e3f10] [c0113e30] sys_mount+0xc0/0x120
[ce6e3f40] [c00178d8] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfe5f8c4
LR = 0x10051b88
Instruction dump:
38000000 817d00c0 3bbd00c0 60088000 814b0000 2f8a0000 419e0008 7c00522c
7f8be800 419e004c 812b000c 81290040 <80090028> 2f800000 419e0028 a009006e
---[ end trace 17c711f9d369c3a3 ]---
------------[ cut here ]------------
Kernel BUG at c045eeac [verbose debug info unavailable]
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT 83xx Sys
Modules linked in:
NIP: c045eeac LR: c045ee84 CTR: 00000000
REGS: ce6e3a80 TRAP: 0700 Tainted: G D (2.6.33.3-rt16)
MSR: 00021032 <ME,CE,IR,DR> CR: 44004428 XER: 00000000
TASK = cd89ccc0[1613] 'umount' THREAD: ce6e2000
GPR00: 00000001 ce6e3b30 cd89ccc0 c068f6b4 c045fc68 00000000 ce6e3b84 ce6e3b64
GPR08: ce6e3b5c c0690000 cd89ccc0 ce6e3b30 24004422 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 c0657824 ce6e3eb8 ce6e3b3c
GPR24: cf028ea0 cec84d1c c065781c cec86a60 00009032 c065781c ...On Tue, 27 Apr 2010, Xianghua Xiao wrote: --
Hey Xianghua, What filesystem was this on? And what architecture? thanks -john --
Also a .config would be helpful. thanks -john --
John, it's ext2 and powerpc 834x. config.gz is attached. the same config is used on 2.6.33.2-rt13 which did not show this umount oops. Thanks! Xianghua
Hi. I tried 2.6.33.3-rt16 patch, and it did not work, first compile failure, and then after applying the inode fix, it will not reach the login screen. It does run some scripts before though. Peace Be With You, Uwaysi Bin Kareem. --
So I've not been able to reproduce the issue, but I have found a few
problems in hunting down the issue Luis reported, and one of them may be
affecting you here.
Could you try the patch below and let me know if it resolves it for you?
thanks
-john
Fix 3 logic bugs in the vfs-scalability patches.
1) Typo that could cause a deadlock in do_umount
2) Improve MNT_MOUNT handling on cloned rootfs
3) Fix might_sleep in atomic in put_mnt_ns
These may not be totally correct, as I still am chasing down some
namespace issues triggered by unshare().
Signed-off-by: John Stultz <johnstul@us.ibm.com>
diff --git a/fs/namespace.c b/fs/namespace.c
index 5459a05..8c5d60b 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1233,7 +1233,7 @@ static int do_umount(struct vfsmount *mnt, int flags)
*/
vfsmount_write_lock();
if (count_mnt_count(mnt) != 2) {
- vfsmount_write_lock();
+ vfsmount_write_unlock();
return -EBUSY;
}
vfsmount_write_unlock();
@@ -1376,6 +1376,12 @@ struct vfsmount *copy_tree(struct vfsmount *mnt, struct dentry *dentry,
if (!q)
goto Enomem;
q->mnt_mountpoint = mnt->mnt_mountpoint;
+ /*
+ * We don't call attach_mnt on rootfs, so set
+ * it as mounted here.
+ */
+ WARN_ON(q->mnt_flags & MNT_MOUNTED);
+ q->mnt_flags |= MNT_MOUNTED;
p = mnt;
list_for_each_entry(r, &mnt->mnt_mounts, mnt_child) {
@@ -2513,17 +2519,15 @@ void put_mnt_ns(struct mnt_namespace *ns)
{
struct vfsmount *root;
LIST_HEAD(umount_list);
- spinlock_t *lock;
- lock = &get_cpu_var(vfsmount_lock);
- if (!atomic_dec_and_lock(&ns->count, lock)) {
- put_cpu_var(vfsmount_lock);
+ vfsmount_write_lock();
+ if (!atomic_dec_and_test(&ns->count)){
+ vfsmount_write_unlock();
return;
}
root = ns->root;
ns->root = NULL;
- spin_unlock(lock);
- put_cpu_var(vfsmount_lock);
+ vfsmount_write_unlock();
down_write(&namespace_sem);
vfsmount_write_lock();
--
John,
Just tried the patch, still got umount hang, please see below.
Thanks!
Xianghua
# umount hda2
# reboot
# Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT 834x SYS
Modules linked in:
NIP: c009ddd8 LR: c009dda8 CTR: 00000000
REGS: ce0f1dd0 TRAP: 0300 Not tainted (2.6.33.3-rt16)
MSR: 00009032 <EE,ME,IR,DR> CR: 24000444 XER: 00000000
DAR: 00000028, DSISR: 20000000
TASK = ceb65ab0[973] 'umount' THREAD: ce0f0000
GPR00: 00000000 ce0f1e80 ceb65ab0 ce0f1dfc 22222222 00000000 ce0f1e44
ce0f1e24
GPR08: 00008000 00000000 cf17cc50 cf17c978 44000442 100bbc1c 0fffd000
ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a8 00000000
ce0f1ec8
GPR24: 00000021 00000060 cebaec40 00000000 00000021 cebaecc0 00000001
c051221c
NIP [c009ddd8] fs_may_remount_ro+0x58/0xd0
LR [c009dda8] fs_may_remount_ro+0x28/0xd0
Call Trace:
[ce0f1e80] [c009dda8] fs_may_remount_ro+0x28/0xd0 (unreliable)
[ce0f1ea0] [c009ef1c] do_remount_sb+0x138/0x178
[ce0f1ec0] [c00bdbe8] do_mount+0x54c/0x840
[ce0f1f10] [c00bdfac] sys_mount+0xd0/0xfc
[ce0f1f40] [c0014208] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfe5f8c4
LR = 0x10051b44
Instruction dump:
38000000 817d00c0 60088000 3bbd00c0 814b0000 2f8a0000 419e0008
7c00522c
7f8be800 419e0060 812b000c 81290040 <80090028> 2f800000 419e0028
a009006e
---[ end trace faefbff1ebfe68f9 ]---
------------[ cut here ]------------
Kernel BUG at c03ae294 [verbose debug info unavailable]
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT 834x SYS
Modules linked in:
NIP: c03ae294 LR: c03ae26c CTR: 00000000
REGS: ce0f1af0 TRAP: 0700 Tainted: G D (2.6.33.3-rt16)
MSR: 00021032 <ME,CE,IR,DR> CR: 24004428 XER: 00000000
TASK = ceb65ab0[973] 'umount' THREAD: ce0f0000
GPR00: 00000001 ce0f1ba0 ceb65ab0 00000001 11111111 00000000 ce0f1bf4
ce0f1bd4
GPR08: ce0f1bcc 00000000 ceb65ab0 ce0f0000 24004422 100bbc1c 0fffd000
ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a8 c0512224
ce0f1ec8
GPR24: ce0f1bac cf0281a0 ...Can you please apply the patch below and provide the debug output ?
Thanks,
tglx
---
fs/file_table.c | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
Index: linux-2.6-tip/fs/file_table.c
===================================================================
--- linux-2.6-tip.orig/fs/file_table.c
+++ linux-2.6-tip/fs/file_table.c
@@ -410,7 +410,27 @@ int fs_may_remount_ro(struct super_block
list = &sb->s_files;
#endif
list_for_each_entry(file, list, f_u.fu_list) {
- struct inode *inode = file->f_path.dentry->d_inode;
+ struct inode *inode;
+
+ if (!file->f_path) {
+ printk(KERN_ERR "file %p fpath == NULL\n",
+ file);
+ continue;
+ }
+
+ if (!file->f_path.dentry) {
+ printk(KERN_ERR "file %p dentry == NULL\n",
+ file);
+ continue;
+ }
+
+ if (!file->f_path.dentry->d_inode) {
+ printk(KERN_ERR "file %p d_inode == NULL\n",
+ file);
+ continue;
+ }
+
+ inode = file->f_path.dentry->d_inode;
/* File with pending delete? */
if (inode->i_nlink == 0)
--
Thomas,
I patched it and re-run it however did not find any condition from
your patch had a hit.
In your patch I changed :
if (!file->f_path) {
to
if(!(&(file->f_path))){
Otherwise it won't compile as f_path is a not a pointer.
Thanks,
Xianghua
# reboot
# Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT 834x SYS
Modules linked in:
NIP: c009d5e0 LR: c009d69c CTR: 00000001
REGS: cde87dd0 TRAP: 0300 Not tainted (2.6.33.3-rt16)
MSR: 00009032 <EE,ME,IR,DR> CR: 24000424 XER: 20000000
DAR: 2e657490, DSISR: 20000000
TASK = ce99e9f0[1404] 'umount' THREAD: cde86000
GPR00: 00007000 cde87e80 ce99e9f0 00000024 00003da7 ffffffff c0542548 00020000
GPR08: c054292c 2e657468 0001ffff cde12b58 24000422 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a0 00000000 cde87ec8
GPR24: 00000021 00000060 c045b5a8 c045b5c4 c050cd6c ce953488 00008000 cde12940
NIP [c009d5e0] fs_may_remount_ro+0x88/0x150
LR [c009d69c] fs_may_remount_ro+0x144/0x150
Call Trace:
[cde87e80] [c009d69c] fs_may_remount_ro+0x144/0x150 (unreliable)
[cde87ea0] [c009e5dc] do_remount_sb+0x138/0x178
[cde87ec0] [c00bd25c] do_mount+0x54c/0x840
[cde87f10] [c00bd620] sys_mount+0xd0/0xfc
[cde87f40] [c0014208] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfe5f8c4
LR = 0x10051b88
Instruction dump:
817f0000 2f8b0000 419e0008 7c005a2c 7f9fe800 419e0080 813f000c 2f890000
419e00a8 81290024 2f890000 419e00b4 <80090028> 2f800000 419e0028 a009006e
---[ end trace 3fba518eec56e584 ]---
------------[ cut here ]------------
Kernel BUG at c03ad89c [verbose debug info unavailable]
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT 834x SYS
Modules linked in:
NIP: c03ad89c LR: c03ad874 CTR: c0121220
REGS: cde87b00 TRAP: 0700 Tainted: G D (2.6.33.3-rt16)
MSR: 00021032 <ME,CE,IR,DR> CR: 84004428 XER: 00000000
TASK = ce99e9f0[1404] 'umount' THREAD: cde86000
GPR00: 00000001 cde87bb0 ce99e9f0 00000001 000002ac 000002ac 00008000 00000000
GPR08: 00000000 00000000 ce99e9f0 ...Ok. Can you please enable CONFIG_DEBUG_LIST ? Thanks, tglx --
I turned on that, could not find any difference from the oops log.
If I try to remount it rw then ro, the remount ro will cause similar oops
Thanks,
Xianghua
# reboot
# Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT 834x SYS
Modules linked in:
NIP: c009ca1c LR: c009c9cc CTR: 00000000
REGS: cde43dd0 TRAP: 0300 Not tainted (2.6.33.3-rt16)
MSR: 00009032 <EE,ME,IR,DR> CR: 24000444 XER: 20000000
DAR: 31c554a2, DSISR: 20000000
TASK = ce9219d0[1396] 'umount' THREAD: cde42000
GPR00: 0000001d cde43e80 ce9219d0 c0454910 000002ac 000002ac 00008000 00000000
GPR08: 00007fff 31c5547a c0454910 cea82b78 44000442 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a0 00000000 cde43ec8
GPR24: 00000021 00000060 c045869c c04586b8 c050bd6c ce951488 00008000 cea82960
NIP [c009ca1c] fs_may_remount_ro+0x88/0x150
LR [c009c9cc] fs_may_remount_ro+0x38/0x150
Call Trace:
[cde43e80] [c009c9cc] fs_may_remount_ro+0x38/0x150 (unreliable)
[cde43ea0] [c009da10] do_remount_sb+0x138/0x178
[cde43ec0] [c00bc420] do_mount+0x54c/0x840
[cde43f10] [c00bc7e4] sys_mount+0xd0/0xfc
[cde43f40] [c00141e8] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfe5f8c4
LR = 0x10051b88
Instruction dump:
817f0000 2f8b0000 419e0008 7c005a2c 7f9fe800 419e0080 813f000c 2f890000
419e00a8 81290024 2f890000 419e00b4 <80090028> 2f800000 419e0028 a009006e
---[ end trace cd3eb2ed5361fbce ]---
------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:808!
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT 834x SYS
Modules linked in:
NIP: c03aa79c LR: c03aa774 CTR: c011fbfc
REGS: cde43b00 TRAP: 0700 Tainted: G D (2.6.33.3-rt16)
MSR: 00021032 <ME,CE,IR,DR> CR: 82004428 XER: 00000000
TASK = ce9219d0[1396] 'umount' THREAD: cde42000
GPR00: 00000001 cde43bb0 ce9219d0 00000001 000002ac 000002ac 00008000 00000000
GPR08: 00000000 00000000 ce9219d0 cde42000 22004422 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a0 00000000 ...Can you please decode the code lines with # addr2line -e vmlinux 0xc009ca1c 0xc009c9cc You need to enable CONFIG_DEBUG_INFO to get real line numbers. Thanks, tglx
Here it is, thanks!
Xianghua
# reboot
# Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT 834x SYS
Modules linked in:
NIP: c009ded8 LR: c009de88 CTR: 00000000
REGS: cde51dd0 TRAP: 0300 Not tainted (2.6.33.3-rt16)
MSR: 00009032 <EE,ME,IR,DR> CR: 24000444 XER: 00000000
DAR: 00000030, DSISR: 20000000
TASK = ce99d580[1404] 'umount' THREAD: cde50000
GPR00: 0000001d cde51e80 ce99d580 cde51dfc 22222222 00000000 cde51e44 cde51e24
GPR08: cde51e1c 00000008 ce99d580 cdf77c90 44000442 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a0 00000000 cde51ec8
GPR24: 00000021 00000060 c045a09c c045a0b8 c051321c cdf084c0 00008000 cdf779b8
NIP [c009ded8] fs_may_remount_ro+0x88/0x150
LR [c009de88] fs_may_remount_ro+0x38/0x150
Call Trace:
[cde51e80] [c009de88] fs_may_remount_ro+0x38/0x150 (unreliable)
[cde51ea0] [c009ef50] do_remount_sb+0x138/0x178
[cde51ec0] [c00bd9c0] do_mount+0x54c/0x840
[cde51f10] [c00bdd84] sys_mount+0xd0/0xfc
[cde51f40] [c00141e8] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xfe5f8c4
LR = 0x10051b88
Instruction dump:
817f0000 2f8b0000 419e0008 7c005a2c 7f9fe800 419e0080 813f000c 2f890000
419e00a8 81290040 2f890000 419e00b4 <80090028> 2f800000 419e0028 a009006e
---[ end trace 8efa68ffffb3f0d2 ]---
------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:808!
Oops: Exception in kernel mode, sig: 5 [#2]
PREEMPT 834x SYS
Modules linked in:
NIP: c03ac1fc LR: c03ac1d4 CTR: 00000000
REGS: cde51af0 TRAP: 0700 Tainted: G D (2.6.33.3-rt16)
MSR: 00021032 <ME,CE,IR,DR> CR: 24004428 XER: 00000000
TASK = ce99d580[1404] 'umount' THREAD: cde50000
GPR00: 00000001 cde51ba0 ce99d580 00000001 11111111 00000000 cde51bf4 cde51bd4
GPR08: cde51bcc 00000000 ce99d580 cde50000 24004422 100bbc1c 0fffd000 ffffffff
GPR16: 00000001 00000000 007fff00 00000000 00000000 0fffa1a0 c0513224 cde51ec8
GPR24: cde51bac cf0281a0 cec21e84 c051321c cec22a60 00009032 cde0e060 cde0e060
NIP [c03ac1fc] ...Again. Can you please decode the code lines with # addr2line -e vmlinux 0xc009ded8 0xc009de88 Please run the above shell command in the directory where your kernel compile output resides. If you compiled with O=BUILD_DIR then cd to $BUILD_DIR otherwise you will find vmlinux in the root of your kernel source tree. Please provide the output. Thanks, tglx --
here it is, just in case I also attached the source file related: addr2line -e vmlinux c009ded8 c009de88 /home/xxiao/xxiao/linux-2.6.33.3/fs/file_table.c:436 /home/xxiao/xxiao/linux-2.6.33.3/fs/file_table.c:440 thanks,
The code line in all traces is
if (inode->i_nlink == 0)
and interestingly enough the inode pointer is not NULL, in the various
traces there are random values: 0x00000008, 0x00000010, 0x2e657468,
0x31c5547a
Can you please verify whether 2.6.33.3-rt15 works ?
Thanks,
tglx
--
On Thu, Apr 29, 2010 at 10:15 AM, Phil Carmody Thomas, I confirm that 2.6.33.3-rt15 worked fine, reboot/umount will not oops. Thanks, Xianghua --
Just pushed out 2.6.33.3-rt17 which has the problem fixed. Can you please verify that it works for you as well ? Thanks, tglx --
Yes it works fine now. Thanks a lot for the efforts! Xianghua --
