Dear all,

I finally managed to find a *reproducible* setup and way to trigger
random corruptions using a sata sil 3114 controller connected to 4
seagate drives

port 1: ST3400832AS sda
port 2: ST3400620AS sdb
port 3: ST3750640AS sdc
port 4: ST3750640AS sdd

sda & sdb form md0 via a raid1 setup followed by an additional
devicemapper layer ( root ). sdc and sdb are separate and also have an
additional device mapper layer ( public ) and ( backups ).

Now when I write large files of zeros to root(sda&sdb) and read the file
back in it contains a few nonzero entries:

# dd if=/dev/zero of=/foo bs=1M count=2000
# hexdump /foo
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
<after >1GB random parts, within large blocks of zeroes> 

I can reliably trigger this on the md0 / devmapper-root setup when I
write about 2GB of data (note that this machine has 1.5G of memory - and
still 1GB is often enough to see this problem). Here it does not matter
where in the filesystem I do these writes.

As a test I did the same on sdc / devmapper-public and
sdd/devmapper-backups with even 30G of zeros. Nothing, no errors
everything is perfectly OK.

So I thought that this is also the the sil mod15write problem
http://home-tj.org/wiki/index.php/Sil_m15w and applied patches 1 & 2
from http://lkml.org/lkml/2007/10/11/115 (adding my two disks) and
rebooted. Now there was some MOD15 stuff in dmesg for the two disks but
still apart from the disks being even slower it was of no use - the
corruption problem was still there (I then also tried patch 3 from Bernd
but that immediately caused oopses fs/errors). So it looks like the
problem I am having is different...

Now I remembered that this machine also has two idle promise pdc20376
sata ports where I first tried the ST3400832AS (sda) and ST3400620AS
(sdb) on about a year ago
http://lists.openwall.net/linux-kernel/2006/08/27/106 . At that time I
just saw random error messages and then finally hangs - quoting Tejon
Heo:

 "I see.  your drive is ...
[ message continues ]

Helo,



Yeah, still the same.  Your drives don't like the way promise controller
speaks to them (e.g. promise generates signals which are ) but now that
sata_promise has proper EH.  It can recover from those errors.  As long
as nothing worse happens, it should be okay.

Thanks.

-- 
tejun
-

[ message continues ]

It is an asus a7v8x with a AMD Athlon(TM) XP 3000+ and admittingly
almost completely filled pci slots (4 dvb cards, 1 with the sil3114; 1
empty; in the agp slot a radeon 9200). Nevertheless I would not expect
the power supply to be the problem (it got replaced recently by a 500W

These errors only appear when I generate some stress (like with the dd).
The machine is now up 2 days 8hrs and no further such warnings in the
log.

Soeren
-

[ message continues ]

Hello,


Thats almost the same test as I'm always doing. Only I do not write only 2GB, 
but as much as it fits onto the disk. On reading back this file, the 
filesystem will report errors somewhere between 50GB and 230GB (disk size is 

All tested S2882 boards here.


Cheers,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH
-

[ message continues ]

Well when I read your mail I thought that I could be seeing exactly the
same bug... it still may be. However ``my'' problem does not go away

Wow, I really see lots of corruptions (well every 1-2 GB a couple of
bytes are corrupted). Are you getting similiarly many in the 50G - 230G

I assume all equipped with lots of memory and mostly empty pci slots?

Soeren
-

[ message continues ]

Yeah, pity it did not fix it :( I will try to port Tejuns patch 
(http://home-tj.org/wiki/index.php/Sil_m15w#Patches) to 2.6.23 today or 

Yes, all pci-slots are free and the systems to have between 4 and 16GB memory 
(ecc, monitored with edac). Well, those are cluster systems (actually tyan 
names those B2882).
Do you think the configuration is related? Here it also happens with odirect, 
we tested this to minimize memory effects.


Cheers,
Bernd


-- 
Bernd Schubert
Q-Leap Networks GmbH
-

[ message continues ]

Hmmhh, dmesg said the m15 fix was turned on (at least it appeared for
the 2 drives in question in dmesg), so I fear it is something different.
On the other hand this is a 'production' machine so I am not too eager

Mine is just a a7v8x with via KT400 chipset... really old, but several
of the pci slots are filled, so the problem may be more likely to happen
it may happen here... on the other hand I never tried writing 50-250G on
the drives I considered OK. Will do. Also what could be helpful is that
we both see patterns in the corruptions, like corruptions are always 512
bytes long or so (IIRC in my case they were only up to 64 bytes).

Soeren
-

[ message continues ]

I never tested what is corrupted. Well, a diff over 250GB would take quite a 
lot of time...

-- 
Bernd Schubert
Q-Leap Networks GmbH
-

[ message continues ]

Actually hexdump does not display duplicate lines, so if your file is
really all zeros it will only display a single line + the count, however
I think it is not so optimized...

Soeren
-

[ message continues ]