On loaded proxies running under 2.6.25-rc7-git one, and 2.6.25-rc8 i have at night outages.
The filesystem is crashing without any serious reason (there is no power outage or even reboot). Even i did xfs_repair, after while it is crashing again,
so probably or there is my mistake or regression kind of very serious.
On some proxies i did kexec while upgrading kernel, and probably filesystem was not unmounted properly. But it crashes again after xfs_repair!
I have absolutely same configuration proxy with 2.6.24.x, it is rock solid.
Here is dmesg i have on crash
uname
Linux Proxy-Karam114 2.6.25-rc7-git5-build-0026 #10 SMP Mon Mar 31 04:41:05 EEST 2008 i686 unknown
mount options
/bin/mount -o noatime -L CACHE1 /cache1
/bin/mount -o noatime -L CACHE2 /cache2
Proxy-Karam114 /proc/sys/fs/xfs # grep "" *
age_buffer_centisecs:1500
error_level:3
filestream_centisecs:3000
inherit_noatime:1
inherit_nodefrag:1
inherit_nodump:1
inherit_nosymlinks:0
inherit_sync:1
irix_sgid_inherit:0
irix_symlink_mode:0
panic_mask:0
restrict_chown:1
rotorstep:1
stats_clear:0
xfsbufd_centisecs:100
xfssyncd_centisecs:3000
Apr 2 09:08:52 194.146.153.114 kernel: [194090.355098] xfs_force_shutdown(sdd1,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xf8b25dd6
Crashed, probably FS was unclean, i did xfs_repair
Here FS probably unclean
Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_inotobp: xfs_imap() returned an error 22 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_iunlink_remove: xfs_inotobp() returned an error 22 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_inactive: xfs_ifree() returned an error = 22 on sdd1
Apr 4 03:03:10 194.146.153.114 kernel: [350703.385931] xfs_force_shutdown(sdd1,0x1) called from line 1737 of file fs/xfs/xfs_vnodeops.c. Return address = 0xf8b257db
Apr 4 03:03:10 194.146.153.114 kernel: [379153.774514] xfs_difree: xfs_inobt_lookup_le returned() an error 5 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [379153.838789] xfs_iunlink_remove: xfs_itobp() returned an error 5 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [379153.838843] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [379153.838865] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [379153.867218] xfs_difree: xfs_inobt_lookup_le returned() an error 5 on sdd1. Returning error.
Apr 4 03:03:10 194.146.153.114 kernel: [379153.867243] xfs_iunlink_remove: xfs_trans_read_buf() returned an error 5 on sdd1. Returning error.
First errors appears in squid, nothing came to dmesg from kernel
Apr 4 03:46:38 194.146.153.28 squid[1793]: cache_dir /cache3/squid: (5) Input/output error
Apr 4 03:46:38 194.146.153.28 ERROR: squid stopped? status 134
Apr 4 03:46:41 194.146.153.28 ERROR: squid started
Apr 4 03:46:41 194.146.153.28 squid[2073]: cache_dir /cache3/squid: (5) Input/output error
Apr 4 03:46:41 194.146.153.28 ERROR: squid stopped? status 134
Apr 4 03:46:43 194.146.153.28 ERROR: squid started
After while
Apr 4 06:44:26 194.146.153.28 kernel: [285436.482594] xfs_force_shutdown(sde1,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xf8b73dd6
--
------
Technical Manager
Virtual ISP S.A.L.
Lebanon
--
