> Sorry, it looks like I sent two replies directly to NeilBrown instead
> of the mailing list. Here they are:
>
> First reply:
>
> Before I checked this email, I upgraded to the newest version of mdadm
> on advice I got from #linux on freenode. The array is no longer
> segfaults when it's assembled. Instead it picks up exactly where it
> left off trying to grow the array but not actually progressing. In the
> new version of mdadm, however, after a short while the drives on the
> mv_sas controller are dropped and mdstat reports that a resync is
> pending.
>
> This looks like an improvement to me. I am going to try to test out
> the controller and drives to see if I can find out what's going on.
>
> ----------------------
>
> Second reply:
>
> I hope I haven't screwed up my disks beyond saving. I was using this
> earlier question on the mailing list as a reference
>
http://www.mail-archive.com/linux-raid@vger.kernel.org/msg08907.html .
> If I could re-assemble the original array, I would feel a lot more
> comfortable proceeding. In the above referenced thread, Greg Nicholson
> mentions that order matters when it comes to assembling arrays, so I
> wrote a perl script (full of terrible hacks) that took the list of the
> 6 original devices and iterated over the permutations of the order of
> those devices, assembled the array with --assume-clean and attempt to
> mount the file system.
>
> This failed for all 720 permutations. It was probably a stupid idea, anyway...
>
> I also tried plugging in a known working disk into the mv_sas
> controller that may have had an issue and it looked like everything
> was working OK.
>
> Now I'm really not sure what to do. I'm completely lost.
>
> Thanks again for your help.
>
> On Fri, Mar 19, 2010 at 12:02 AM, Neil Brown <neilb@suse.de> wrote:
>> On Thu, 18 Mar 2010 21:52:54 -0400
>> Stephan Stachurski <ses1984@gmail.com> wrote:
>>
>>> I have had a RAID5 up for quite a while with 6 disks. I recently added
>>> 4 and attempted to grow the array to span all 10 devices.
>>>
>>> For an hour after starting the grow command, the speed of the
>>> operation was 0K/s the entire time. I thought something must be wrong,
>>> and the best course of action would be to reboot and start over from a
>>> clean boot. I'm not a linux expert so I thought that if I rebooted
>>> everything would try to exit gracefully.
>>>
>>> After one hour, the system still had not finished shutting down. I
>>> then did alt-sysreq RSEISUB waiting over one minute between each
>>> command. I've included the syslog of what happened up until the next
>>> start up, but put it last because it's by far the longest.
>>
>> I think you will be able to get your data back. It won't be trivial, but it
>> should be possible.
>>
>> I looks like the driver for the mv_sas controller has issues. When md/raid5
>> started writing data on to them to reshape the array something went wrong and
>> the writes didn't complete, so nothing else happened.
>>
>> I don't know why mdadm is getting a segmentation fault. Possibly this is
>> fixed in a newer version of mdadm. However it is possibly good that it
>> didn't manage to restart the array fully as it would probably has just failed
>> again and might have made more of a mess.
>>
>> To get you data back we need to understand exactly what happened.
>> What should happen when you run "mdadm --grow ... " is that it sets
>> up for a reshape but doesn't let it progress.
>> Then it prints:
>>
>> mdadm: Need to backup 1152K of critical section..
>>
>> It then copies the first 1152K (in the case of 6->10 with 256K chunk)
>> from the start of the array to near the end of each of the 'spares'.
>> Then it allows the reshape to proceed.
>> Once the reshape has progressed past that 1152K it removes the
>> copy that it made (erases some metadata for it) and prints
>>
>> mdadm: ... critical section passed.
>>
>>
>> I presume that it didn't successfully pass the critical section, else
>> the Reshape Position would be greater than 0.
>>
>> It is possible that the reshape didn't start at all and your data is exactly
>> where you left it, but we cannot be sure without looking...
>>
>> What I suggest you do is:
>> 1/ find the backup of the first 1152K
>> 2/ re-create the array as the original 6-drive raid5
>> 3/ Check if the backup needs to be restored and possibly restore it
>> 4/ Don't use the new drives until you are really sure they will work.
>>
>> 1 is the hardest. I have a vague plan of giving mdadm the ability to do this
>> but I haven't yet. I could possibly do it next week some time if you can
>> wait.
>> You would need to look at the code in Grow.c to see where it is written. I
>> think there is a block of metadata near the end of the device - just before
>> the md metadata - which records what has been backed up where. Once you find
>> and decode that from one of the spares you can easily use 'dd' to extract the
>> backup.
>>
>> 2 is quite easy:
>>
>> mdadm -C /dev/md0 -e 0.90 -l 5 -n 6 -c 256 \
>> --assume-clean /dev/sdb /dev/sdg /dev/sdh /dev/sdd /dev/sda /dev/sdc
>>
>> Make sure you have the devices if the right order. If you aren't sure, then
>> mdadm -E list..of..devices | grep this
>> should give you an ascending series in columns 2 and 5.
>>
>> 3 is simply a 'cmp' between /dev/md0 and the backup that you restored. or
>> maybe just 'fsck' of /dev/md0.
>> If you decide to restore (be sure before you do), just dd the backup to the
>> start of /dev/md0
>>
>> I don't know how you can make yourself sure that the drives really do work.
>> Lots of testing of the new devices by themselves in an array ???
>>
>> Good luck.
>>
>> NeilBrown
>>
>>>
>>> When I rebooted, the array seemed to be up, but mounting it resulted
>>> in a bad FS type error, even when I tried to specify it (ext4). After
>>> stopping the inactive array, and trying to reassemble it, mdadm
>>> crashed to segmentation fault.
>>> Is it possible to recover the data? We have backups, but they're
>>> spread out over 1500 DVDs.
>>>
>>> When I examine the drives, the output looks pretty much like this for
>>> each drive (6 drives say active and 4 say clean, corresponding to the
>>> 6 original and 4 added drives):
>>> $ mdadm --examine /dev/sda
>>> /dev/sda:
>>> Magic : a92b4efc
>>> Version : 00.91.00
>>> UUID : 56c16545:07db76d6:e368bf24:bd0fce41
>>> Creation Time : Tue Feb 2 09:58:58 2010
>>> Raid Level : raid5
>>> Used Dev Size : 976762368 (931.51 GiB 1000.20 GB)
>>> Array Size : 8790861312 (8383.62 GiB 9001.84 GB)
>>> Raid Devices : 10
>>> Total Devices : 10
>>> Preferred Minor : 0
>>> Reshape pos'n : 0
>>> Delta Devices : 4 (6->10)
>>> Update Time : Thu Mar 18 23:33:40 2010
>>> State : active
>>> Active Devices : 10
>>> Working Devices : 10
>>> Failed Devices : 0
>>> Spare Devices : 0
>>> Checksum : 79904299 - correct
>>> Events : 270611
>>> Layout : left-symmetric
>>> Chunk Size : 256K
>>> Number Major Minor RaidDevice State
>>> this 4 8 0 4 active sync /dev/sda
>>> 0 0 8 16 0 active sync /dev/sdb
>>> 1 1 8 96 1 active sync /dev/sdg
>>> 2 2 8 112 2 active sync /dev/sdh
>>> 3 3 8 48 3 active sync /dev/sdd
>>> 4 4 8 0 4 active sync /dev/sda
>>> 5 5 8 32 5 active sync /dev/sdc
>>> 6 6 8 160 6 active sync /dev/sdk
>>> 7 7 8 144 7 active sync /dev/sdj
>>> 8 8 8 128 8 active sync /dev/sdi
>>> 9 9 8 80 9 active sync /dev/sdf
>>>
>>> #syslog showing how mdadm detects the bad 10-drive array on boot
>>> Mar 18 23:47:35 raidserver kernel: [ 2.178393] md: bind<sdc>
>>> Mar 18 23:47:35 raidserver kernel: [ 2.193132] md: bind<sdd>
>>> Mar 18 23:47:35 raidserver kernel: [ 2.211906] md: bind<sdb>
>>> Mar 18 23:47:35 raidserver kernel: [ 2.220062] md: bind<sda>
>>> Mar 18 23:47:35 raidserver kernel: [ 2.230062] ohci1394: fw-host0:
>>> OHCI-1394 1.1 (PCI): IRQ=[22] MMIO=[fd6ff000-fd6ff7ff] Max
>>> Packet=[2048] IR/IT contexts=[4/8]
>>> Mar 18 23:47:35 raidserver kernel: [ 3.551483] ieee1394: Host
>>> added: ID:BUS[0-00:1023] GUID[003635c7006cf049]
>>> Mar 18 23:47:35 raidserver kernel: [ 6.579349]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 0
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 6.579353]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 0
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 6.780038]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 1
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 6.780041]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 1
>>> attach sas addr is 1
>>> Mar 18 23:47:35 raidserver kernel: [ 6.990054]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 2
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 6.990057]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 2
>>> attach sas addr is 2
>>> Mar 18 23:47:35 raidserver kernel: [ 7.200052]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 3
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.200055]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 3
>>> attach sas addr is 3
>>> Mar 18 23:47:35 raidserver kernel: [ 7.310035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 4
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.310038]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 4
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.420035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 5
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.420038]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 5
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.630052]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 6
>>> attach dev info is 2000000
>>> Mar 18 23:47:35 raidserver kernel: [ 7.630055]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 6
>>> attach sas addr is 6
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840053]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 7
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840056]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 7
>>> attach sas addr is 7
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840062] scsi8 : mvsas
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840605]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 0 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840610]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 1 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840614]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 2 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840617]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 3 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840621]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 6 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840624]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 380:phy 7 byte
>>> dmaded.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840871] mvsas 0000:03:00.0:
>>> mvsas: driver version 0.8.2
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840885] mvsas 0000:03:00.0:
>>> PCI INT A -> GSI 16 (level, low) -> IRQ 16
>>> Mar 18 23:47:35 raidserver kernel: [ 7.840891] mvsas 0000:03:00.0:
>>> setting latency timer to 64
>>> Mar 18 23:47:35 raidserver kernel: [ 7.841965]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[0:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.843312] ata9.00: ATA-8: WDC
>>> WD10EARS-00Y5B1, 80.00A80, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.843316] ata9.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.843480] mvsas 0000:03:00.0:
>>> mvsas: PCI-E x4, Bandwidth Usage: 2.5 Gbps
>>> Mar 18 23:47:35 raidserver kernel: [ 7.845117] ata9.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.845182] scsi 8:0:0:0:
>>> Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 7.845801]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[1:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.846588] ata10.00: ATA-8: WDC
>>> WD10EADS-00L5B1, 01.01A01, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.846591] ata10.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.847419] ata10.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.847455] scsi 8:0:1:0:
>>> Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 7.848069]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[2:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.848894] ata11.00: ATA-8: WDC
>>> WD10EACS-00D6B1, 01.01A01, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.848897] ata11.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.849713] ata11.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.849751] scsi 8:0:2:0:
>>> Direct-Access ATA WDC WD10EACS-00D 01.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 7.850909]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[3:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.852188] ata12.00: ATA-8: WDC
>>> WD10EARS-00Y5B1, 80.00A80, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.852192] ata12.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.853488] ata12.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.853524] scsi 8:0:3:0:
>>> Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 7.854665]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[4:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.855955] ata13.00: ATA-8: WDC
>>> WD10EARS-00Y5B1, 80.00A80, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.855959] ata13.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.857258] ata13.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.857293] scsi 8:0:4:0:
>>> Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 7.858437]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1365:found
>>> dev[5:5] is gone.
>>> Mar 18 23:47:35 raidserver kernel: [ 7.859713] ata14.00: ATA-8: WDC
>>> WD10EARS-00Y5B1, 80.00A80, max UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.859716] ata14.00: 1953525168
>>> sectors, multi 0: LBA48 NCQ (depth 31/32)
>>> Mar 18 23:47:35 raidserver kernel: [ 7.861014] ata14.00: configured
>>> for UDMA/133
>>> Mar 18 23:47:35 raidserver kernel: [ 7.861051] scsi 8:0:5:0:
>>> Direct-Access ATA WDC WD10EARS-00Y 80.0 PQ: 0 ANSI: 5
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831403] sd 8:0:0:0: Attached
>>> scsi generic sg5 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831507] sd 8:0:1:0: Attached
>>> scsi generic sg6 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831610] sd 8:0:2:0: Attached
>>> scsi generic sg7 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831713] sd 8:0:3:0: Attached
>>> scsi generic sg8 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831823] sd 8:0:4:0: Attached
>>> scsi generic sg9 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.831927] sd 8:0:5:0: Attached
>>> scsi generic sg10 type 0
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832481] sd 8:0:0:0: [sdf]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832485] sd 8:0:1:0: [sdg]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832547] sd 8:0:1:0: [sdg]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832551] sd 8:0:0:0: [sdf]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832555] sd 8:0:0:0: [sdf]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832559] sd 8:0:1:0: [sdg]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832586] sd 8:0:0:0: [sdf]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832590] sd 8:0:1:0: [sdg]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832806] sdg:
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832853] sdf:
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832914] sd 8:0:2:0: [sdh]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832953] sd 8:0:2:0: [sdh]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832956] sd 8:0:2:0: [sdh]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.832976] sd 8:0:2:0: [sdh]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833087] sdh:
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833182] sd 8:0:3:0: [sdi]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833197] sd 8:0:4:0: [sdj]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833238] sd 8:0:3:0: [sdi]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833241] sd 8:0:3:0: [sdi]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833250] sd 8:0:4:0: [sdj]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833252] sd 8:0:4:0: [sdj]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833270] sd 8:0:3:0: [sdi]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833279] sd 8:0:4:0: [sdj]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833442] sdi:
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833467] sdj:
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833553] sd 8:0:5:0: [sdk]
>>> 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833593] sd 8:0:5:0: [sdk]
>>> Write Protect is off
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833596] sd 8:0:5:0: [sdk]
>>> Mode Sense: 00 3a 00 00
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833617] sd 8:0:5:0: [sdk]
>>> Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>> Mar 18 23:47:35 raidserver kernel: [ 8.833734] sdk: unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 8.846211] sd 8:0:2:0: [sdh]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 8.846599] unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 8.846759] sd 8:0:1:0: [sdg]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 9.316040] unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 9.316243] sd 8:0:3:0: [sdi]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 9.316249] unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 9.316404] sd 8:0:0:0: [sdf]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 9.317860] unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 9.318033] sd 8:0:4:0: [sdj]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 9.321964] unknown partition table
>>> Mar 18 23:47:35 raidserver kernel: [ 9.322127] sd 8:0:5:0: [sdk]
>>> Attached SCSI disk
>>> Mar 18 23:47:35 raidserver kernel: [ 12.220036]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 0
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.220039]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 0
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.330034]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 1
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.330037]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 1
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.440035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 2
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.440037]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 2
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.550034]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 3
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.550038]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 3
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.660035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 4
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.660038]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 4
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.770035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 5
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.770037]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 5
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.880035]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 6
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.880037]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 6
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.990034]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1214:port 7
>>> attach dev info is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.990037]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c 1216:port 7
>>> attach sas addr is 0
>>> Mar 18 23:47:35 raidserver kernel: [ 12.990043] scsi9 : mvsas
>>> Mar 18 23:47:35 raidserver kernel: [ 13.595116] md: bind<sdi>
>>> Mar 18 23:47:35 raidserver kernel: [ 13.651656] md: bind<sdf>
>>> Mar 18 23:47:35 raidserver kernel: [ 13.653928] md: bind<sdj>
>>> Mar 18 23:47:35 raidserver kernel: [ 13.854601] md: bind<sdk>
>>> Mar 18 23:47:35 raidserver kernel: [ 14.055322] md: bind<sdh>
>>> Mar 18 23:47:35 raidserver kernel: [ 14.255683] md: bind<sdg>
>>> Mar 18 23:47:35 raidserver kernel: [ 14.259239] xor: automatically
>>> using best checksumming function: generic_sse
>>> Mar 18 23:47:35 raidserver kernel: [ 14.300015] generic_sse:
>>> 6593.600 MB/sec
>>> Mar 18 23:47:35 raidserver kernel: [ 14.300018] xor: using function:
>>> generic_sse (6593.600 MB/sec)
>>> Mar 18 23:47:35 raidserver kernel: [ 14.300599] async_tx: api
>>> initialized (async)
>>> Mar 18 23:47:35 raidserver kernel: [ 14.470026] raid6: int64x1 1711 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 14.640019] raid6: int64x2 2392 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 14.810046] raid6: int64x4 1567 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 14.980047] raid6: int64x8 1540 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 15.150016] raid6: sse2x1 2931 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 15.320030] raid6: sse2x2 3916 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 15.490023] raid6: sse2x4 4088 MB/s
>>> Mar 18 23:47:35 raidserver kernel: [ 15.490025] raid6: using
>>> algorithm sse2x4 (4088 MB/s)
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493236] md: raid6
>>> personality registered for level 6
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493240] md: raid5
>>> personality registered for level 5
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493242] md: raid4
>>> personality registered for level 4
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493642] raid5: md0 is not
>>> clean -- starting background reconstruction
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493645] raid5:
>>> reshape_position too early for auto-recovery - aborting.
>>> Mar 18 23:47:35 raidserver kernel: [ 15.493647] md: pers->run() failed ...
>>> Mar 18 23:47:35 raidserver kernel: [ 15.931347] md: linear
>>> personality registered for level -1
>>> Mar 18 23:47:35 raidserver kernel: [ 15.934321] md: multipath
>>> personality registered for level -4
>>> Mar 18 23:47:35 raidserver kernel: [ 15.936772] md: raid0
>>> personality registered for level 0
>>> Mar 18 23:47:35 raidserver kernel: [ 15.940395] md: raid1
>>> personality registered for level 1
>>> Mar 18 23:47:35 raidserver kernel: [ 15.949846] md: raid10
>>> personality registered for level 10
>>> #syslog of the segfault
>>>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.028406] md: md0 stopped.
>>> Mar 18 23:41:47 raidserver kernel: [ 155.028443] md: unbind<sdg>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.051309] md: export_rdev(sdg)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.051464] md: unbind<sdk>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.091288] md: export_rdev(sdk)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.091418] md: unbind<sdj>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.131274] md: export_rdev(sdj)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.131407] md: unbind<sdi>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.161277] md: export_rdev(sdi)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.161421] md: unbind<sdl>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.191275] md: export_rdev(sdl)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.191400] md: unbind<sdh>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.221276] md: export_rdev(sdh)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.221403] md: unbind<sda>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.251276] md: export_rdev(sda)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.251385] md: unbind<sdb>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.281277] md: export_rdev(sdb)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.281379] md: unbind<sdc>
>>> Mar 18 23:41:47 raidserver kernel: [ 155.311276] md: export_rdev(sdc)
>>> Mar 18 23:41:47 raidserver kernel: [ 155.311377] md: unbind<sdd>
>>> Mar 18 23:41:47 raidserver mdadm[1738]: DeviceDisappeared event
>>> detected on md device /dev/md0
>>> Mar 18 23:41:47 raidserver kernel: [ 155.341274] md: export_rdev(sdd)
>>> Mar 18 23:43:46 raidserver kernel: [ 274.246878] md: md0 stopped.
>>> Mar 18 23:45:17 raidserver kernel: [ 365.227246] md: md0 stopped.
>>> Mar 18 23:45:18 raidserver kernel: [ 365.828455] __ratelimit: 30
>>> callbacks suppressed
>>> Mar 18 23:45:18 raidserver kernel: [ 365.828466] mdadm[2874]:
>>> segfault at 38 ip 00000000004184ff sp 00007fffc2aa6bd0 error 4 in
>>> mdadm[400000+2a000]
>>>
>>> #syslog of the grow and subsequent reboot
>>>
>>> Mar 18 23:27:41 raidserver kernel: [ 861.411806] md: bind<sdf>
>>> Mar 18 23:27:42 raidserver ata_id[2638]: HDIO_GET_IDENTITY failed for '/dev/sdi'
>>> Mar 18 23:27:42 raidserver kernel: [ 862.024028] md: bind<sdi>
>>> Mar 18 23:27:43 raidserver ata_id[2650]: HDIO_GET_IDENTITY failed for '/dev/sdj'
>>> Mar 18 23:27:43 raidserver kernel: [ 863.133531] md: bind<sdj>
>>> Mar 18 23:27:43 raidserver ata_id[2658]: HDIO_GET_IDENTITY failed for '/dev/sdk'
>>> Mar 18 23:27:43 raidserver kernel: [ 863.285276] md: bind<sdk>
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792375] RAID5 conf printout:
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792385] --- rd:10 wd:10
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792393] disk 0, o:1, dev:sdb
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792398] disk 1, o:1, dev:sdg
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792402] disk 2, o:1, dev:sdh
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792406] disk 3, o:1, dev:sdd
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792410] disk 4, o:1, dev:sda
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792415] disk 5, o:1, dev:sdc
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792419] disk 6, o:1, dev:sdk
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792443] RAID5 conf printout:
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792447] --- rd:10 wd:10
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792450] disk 0, o:1, dev:sdb
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792454] disk 1, o:1, dev:sdg
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792457] disk 2, o:1, dev:sdh
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792461] disk 3, o:1, dev:sdd
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792465] disk 4, o:1, dev:sda
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792469] disk 5, o:1, dev:sdc
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792472] disk 6, o:1, dev:sdk
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792476] disk 7, o:1, dev:sdj
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792484] RAID5 conf printout:
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792487] --- rd:10 wd:10
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792491] disk 0, o:1, dev:sdb
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792494] disk 1, o:1, dev:sdg
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792498] disk 2, o:1, dev:sdh
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792502] disk 3, o:1, dev:sdd
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792506] disk 4, o:1, dev:sda
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792509] disk 5, o:1, dev:sdc
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792513] disk 6, o:1, dev:sdk
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792517] disk 7, o:1, dev:sdj
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792520] disk 8, o:1, dev:sdi
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792528] RAID5 conf printout:
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792531] --- rd:10 wd:10
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792535] disk 0, o:1, dev:sdb
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792538] disk 1, o:1, dev:sdg
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792542] disk 2, o:1, dev:sdh
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792545] disk 3, o:1, dev:sdd
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792549] disk 4, o:1, dev:sda
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792552] disk 5, o:1, dev:sdc
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792556] disk 6, o:1, dev:sdk
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792559] disk 7, o:1, dev:sdj
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792563] disk 8, o:1, dev:sdi
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792567] disk 9, o:1, dev:sdf
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792713] md: reshape of RAID array md0
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792722] md: minimum
>>> _guaranteed_ speed: 1000 KB/sec/disk.
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792728] md: using maximum
>>> available idle IO bandwidth (but not more than 200000 KB/sec) for
>>> reshape.
>>> Mar 18 23:28:35 raidserver kernel: [ 914.792746] md: using 128k
>>> window, over a total of 976762368 blocks.
>>> Mar 18 23:28:35 raidserver mdadm[1627]: RebuildStarted event detected
>>> on md device /dev/md0
>>> Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on
>>> md device /dev/md0, component device /dev/sdk
>>> Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on
>>> md device /dev/md0, component device /dev/sdj
>>> Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on
>>> md device /dev/md0, component device /dev/sdi
>>> Mar 18 23:28:35 raidserver mdadm[1627]: SpareActive event detected on
>>> md device /dev/md0, component device /dev/sdf
>>> Mar 18 23:29:05 raidserver kernel: [ 945.010492]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:29:05 raidserver kernel: [ 945.010501]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:29:05 raidserver kernel: [ 945.010517]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:29:05 raidserver kernel: [ 945.010523]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:29:36 raidserver kernel: [ 976.010049]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:29:36 raidserver kernel: [ 976.010058]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:29:36 raidserver kernel: [ 976.010071]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:29:36 raidserver kernel: [ 976.010077]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:30:07 raidserver kernel: [ 1007.010495]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:30:07 raidserver kernel: [ 1007.010504]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:30:07 raidserver kernel: [ 1007.010518]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:30:07 raidserver kernel: [ 1007.010525]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:30:38 raidserver kernel: [ 1038.010051]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:30:38 raidserver kernel: [ 1038.010060]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:30:38 raidserver kernel: [ 1038.010075]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:30:38 raidserver kernel: [ 1038.010081]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:31:09 raidserver kernel: [ 1069.010493]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:31:09 raidserver kernel: [ 1069.010503]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:31:09 raidserver kernel: [ 1069.010517]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:31:09 raidserver kernel: [ 1069.010523]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620078] INFO: task
>>> md0_reshape:2679 blocked for more than 120 seconds.
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620086] "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620092] md0_reshape D
>>> 00000000ffffffff 0 2679 2 0x00000000
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620103] ffff8800441e1ad0
>>> 0000000000000046 ffff8800441e1a80 0000000000015880
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620113] ffff880068199a60
>>> 0000000000015880 0000000000015880 0000000000015880
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620122] 0000000000015880
>>> ffff880068199a60 0000000000015880 0000000000015880
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620131] Call Trace:
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620169]
>>> [<ffffffffa01d6ce1>] get_active_stripe+0x2a1/0x360 [raid456]
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620185]
>>> [<ffffffff81053d60>] ? default_wake_function+0x0/0x10
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620197]
>>> [<ffffffffa01d92e0>] reshape_request+0x4a0/0x980 [raid456]
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620210]
>>> [<ffffffffa01d9ada>] sync_request+0x31a/0x3a0 [raid456]
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620221]
>>> [<ffffffffa01d69ae>] ? raid5_unplug_device+0x7e/0x110 [raid456]
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620233]
>>> [<ffffffff813daa1e>] md_do_sync+0x5fe/0xba0
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620242]
>>> [<ffffffff813db774>] md_thread+0x44/0x120
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620249]
>>> [<ffffffff813db730>] ? md_thread+0x0/0x120
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620257]
>>> [<ffffffff81078746>] kthread+0xa6/0xb0
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620266]
>>> [<ffffffff810130ea>] child_rip+0xa/0x20
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620273]
>>> [<ffffffff810786a0>] ? kthread+0x0/0xb0
>>> Mar 18 23:31:20 raidserver kernel: [ 1080.620279]
>>> [<ffffffff810130e0>] ? child_rip+0x0/0x20
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010040]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010049]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010065]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010072]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010127] sd 8:0:1:0: [sdg]
>>> Unhandled error code
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010132] sd 8:0:1:0: [sdg]
>>> Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010141] end_request: I/O
>>> error, dev sdg, sector 0
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010310] sd 8:0:2:0: [sdh]
>>> Unhandled error code
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010314] sd 8:0:2:0: [sdh]
>>> Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
>>> Mar 18 23:31:40 raidserver kernel: [ 1100.010321] end_request: I/O
>>> error, dev sdh, sector 8
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010514]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010523]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010545]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010551]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010562]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010567]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010580]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010585]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010596]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010601]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010612]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:11 raidserver kernel: [ 1131.010617]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010054]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010063]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010084]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010090]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010101]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010107]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010119]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010125]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010136]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010142]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010153]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:32:42 raidserver kernel: [ 1162.010159]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010053]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010062]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010083]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010089]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010100]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010106]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010118]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010123]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010134]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010139]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010150]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:33:13 raidserver kernel: [ 1193.010156]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:33:16 raidserver ata_id[2906]: HDIO_GET_IDENTITY failed for '/dev/sdj'
>>> Mar 18 23:33:16 raidserver ata_id[2910]: HDIO_GET_IDENTITY failed for '/dev/sdk'
>>> Mar 18 23:33:16 raidserver ata_id[2911]: HDIO_GET_IDENTITY failed for '/dev/sdi'
>>> Mar 18 23:33:34 raidserver kernel: [ 1213.857434] md: md0 still in use.
>>> Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0
>>> blocks 0 reqs (0 success)
>>> Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0
>>> extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0 lost
>>> Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0
>>> generated and it took 0
>>> Mar 18 23:33:40 raidserver kernel: [ 1219.907801] EXT4-fs: mballoc: 0
>>> preallocated, 0 discarded
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010051]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010061]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010087]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010093]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010109]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010115]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010130]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010135]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010150]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010155]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010166]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:10 raidserver kernel: [ 1250.010171]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010517]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010528]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010550]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010556]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010572]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010578]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010592]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010597]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010611]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010617]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010627]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1669:mvs_abort_task:rc= 5
>>> Mar 18 23:34:41 raidserver kernel: [ 1281.010633]
>>> /build/buildd/linux-2.6.31/drivers/scsi/mvsas/mv_sas.c
>>> 1608:mvs_query_task:rc= 5
>>> Mar 18 23:35:06 raidserver kernel: Kernel logging (proc) stopped.
>>> Mar 18 23:35:06 raidserver rsyslogd: [origin software="rsyslogd"
>>> swVersion="4.2.0" x-pid="1027" x-info="http://www.rsyslog.com"]
>>> exiting on signal 15.
>>> Mar 18 23:39:31 raidserver kernel: imklog 4.2.0, log source =
>>> /var/run/rsyslog/kmsg started.
>>> Mar 18 23:39:31 raidserver rsyslogd: [origin software="rsyslogd"
>>> swVersion="4.2.0" x-pid="647" x-info="http://www.rsyslog.com"]
>>> (re)start
>>> Mar 18 23:39:31 raidserver rsyslogd: rsyslogd's groupid changed to 102
>>> Mar 18 23:39:31 raidserver rsyslogd: rsyslogd's userid changed to 101
>>>
>>> --
>>> Stephan E Stachurski
>>> 773-315-1684
>>>
ses1984@gmail.com
>>>
>>>
>>>
>>>
>>> --
>>> Stephan E Stachurski
>>> 773-315-1684
>>>
ses1984@gmail.com
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to
majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
>
> --
> Stephan E Stachurski
> 773-315-1684
>
ses1984@gmail.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to
majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
