I have an existing raid10,f2 array with four drives, all running on a single
SATA controller.  I have a second controller to add to the system and I'd like
to split the existing drives between the two controllers.  I'm hoping to make
the configuration more robust against the possibility of a single controller
failure.  It would also be nice to get more performance out of the array, though
I doubt having a single controller is a bottleneck with only 4 7200RPM drives.

So with four drives sda, sdb, sdc, and sdd and two controllers C1 and C2, should
I go with

    C1: sda, sdb
    C2: sdc, sdd

or

    C1: sda, sdc
    C2: sdb, sdd

or some other configuration?

I've looked through the last six months of messages in the archives, and the
md(4) and mdadm(8) manpages, and the wiki on https://raid.wiki.kernel.org/ and
didn't see anything that quite answers this question at a level I can
understand.  If there is a reference I can consult, I'm happy to keep digging.

If it will help, the output of /proc/mdstat and "mdadm --detail" on the md
device are included below.


Mike McCallister


# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
[raid10]
md3 : active raid10 sda5[0] sdd5[3] sdc5[2] sdb5[1]
      1445318656 blocks super 1.1 256K chunks 2 far-copies [4/4] [UUUU]
      bitmap: 6/345 pages [24KB], 2048KB chunk

# mdadm --detail /dev/md3
/dev/md3:
        Version : 01.01.03
  Creation Time : Sun Nov  9 22:47:00 2008
     Raid Level : raid10
     Array Size : 1445318656 (1378.36 GiB 1480.01 GB)
  Used Dev Size : 1445318656 (689.18 GiB 740.00 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Apr 11 11:47:09 2010
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=1, far=2
     Chunk Size : 256K

           Name : ozark:3
       ...
[ message continues ]

I'd use smartctl to get the drive serial numbers, then move what are
currently shown as sdb5 and sdd5 to your new controller during a
coldboot.
--

[ message continues ]

Thanks.  I did this and was able to boot and reassemble successfully with a live
CD.  However, the question in the back of my mind is "is there any way to tell
whether this would have worked with one of the controllers down?"  In other
words, if I just pulled the plug on sdb and sdd instead of moving them to a new
controller, would the array have been able to be reassembled in a degraded mode
with only sda and sdc?  Or would it actually need a different combination of
drives (e.g., sda and sdb) to be able to reassemble?

Obviously I could try this, but if I am able to reassemble an array with two
drives missing, won't I have to resync the other drives when I return them to
the array?  I'd rather avoid experiments involving resyncs if possible.


Mike


--

[ message continues ]

Hi,
I am not sure if I understand your question, but I suppose you will hit 
the problem of udev naming of your disks, and if your disks are 
hotswap-able, then also the problem of hotswap, which is currently 
discussed here in this mailing list.

I have spent too much time trying to solve both these problems with the 
current linux SW RAID (mdadm v. 2.6) and here is the first part of my 
solution:
http://wiki.debian.org/Persistent_disk_names

The page is still not really  finished, I am still playing with my 
server so the documentation must wait.



--

[ message continues ]

Thanks for the heads up.  I did the swap while powered down and only brought the
system up using an Ubuntu Live CD, so the udev on my current root partition
didn't come into play.  During that live-CD boot, I had no apparent problems
with the changing device names.  They had changed pretty significantly: sdb
became sdd, sdc became sdb, sdd became sde, and another device not part of the
array became sdc.  In spite of the shuffle and a new SATA controller, mdadm was
able to reassemble the RAID devices on those drive partitions with no input from
me.  I believe this is because mdadm scanned the partitions looking device for
the appropriate UUIDs in the metadata, and with all the devices accounted for,
it was able to "do the right thing".  I was very impressed.

When I boot using the original root partition (running Ubuntu Hardy), maybe udev
will shuffle the device names more severely.  But so long as mdadm is configured
to scan all the available partitions, I expect it will locate all the pieces,
whatever they may be named, and successfully reassemble the arrays.


Mike


--

[ message continues ]

I cannot quite understand your problem.  As every part of the array
contains it's own metadata, it doesn't matter to md which /dev/sdX a
drive is.  It might matter a bit for boot-time assembly, but actually

But I understand yours, Michael.  You would like to put those
RAID0-pieces of your RAID10 each on a single controller.  I believe one
must find out what "NEAR" and "FAR" is.
I'd understand it as: "NEAR" is the closer disk - yeah.  Closer in the
array-tree, or it's logical base-LBA is closer?  Neil, could you please

All the best,
Stefan
--

[ message continues ]

I know how UUIDs work.
My problem with device names is not a "critical", it's about "user 
friendliness" of the physical disk management.
If a disk fails and I receive email "A Fail event had been detected on 
md device /dev/md2. It could be related to component device /dev/sdd3", 
how will I know which disk should be replaced, if the device name is not 
fixed/persistent? Is it the disk in the first bay, or the second? Aside 
of the solution documented on the wiki page, I could also use try a 
simpler one based on the idea here:
http://www.outsidaz.org/blog/2009/11/05/identifying-failed-drives-via-udev-and-mdadm/

But for the disk management purposes, I prefer having the disk names 
fixed according the the disk bay position. So disk in bay nr. 1 is 
/dev/sda, bay nr. 2 is /dev/sdb, etc.

The wiki page I created is just about this. I haven't found any document 
like this anywhere else, so if it helps someone, I'll be glad.

Tom
--

[ message continues ]

Record the serial number of the disks and  which bay they are in.
--

[ message continues ]

I am not sure it has been said, but for a 4 disk raid10,f2 array
you should place the first and the second disk on one controller, and then the 3rd
and 4th on the second controller. Then you would have a copy of all blocks even
if one controller fails.

I believe the order is defined by the order the disks are specified in
on the mdadm --create line.

best regards
keld

--

[ message continues ]

On Tue, 13 Apr 2010 11:23:54 +0200

Uhmm... no.

Disk:   1    2    3    4
blocks  a    b    c    d
        e    f    g    h
         .........
        d    a    b    c
        h    e    f    g

If you only have first and second, then block c is missing.
You want 1st and 3rd on one controller, 2nd and 4th on the other

Correct.


--

[ message continues ]