> Hello all,
>
> I have been using an in-house mod to the raid5.c driver to optimize
> for linear writes. The optimization is probably too specific for
> general kernel inclusion, but I wanted to throw out what I have been
> doing in case anyone is interested.
>
> The application involves a kernel module that can produce precisely
> aligned, long, linear writes. In the case of raid-5, the obvious plan
> is to issue writes that are complete raid stripes of
> 'optimal_io_length'.
>
> Unfortunately, optimal_io_length is often less than the advertised max
> io_buf size value and sometime less than the system max io_buf size
> value. Thus just pumping up the max value inside of raid5 is dubious.
> Even though dubious, just punching up the
> mddev->queue->limits.max_hw_sectors does seem to work, not break
> anything obvious, and does help performance out a little.
>
> In looking at long linear writes with the stock raid5 driver, I am
> seeing a small amount of reads to individual devices. The test
> application code calling the raid layer has > 100MB of locked kernel
> buffer slamming the raid5 driver, so exactly why raid5 needs to
> back-fill some reads is not very clear to me. Looking at the raid5
> code, it does not look like there is a real "scheduler" for deciding
> when to back-fill the stripe cache, but instead it just relies on
> thread round trips. In my case, I am testing on server-class systems
> with 8 or 16 3GHz threads, so availability of CPU cycles for the raid5
> code is very high.
>
> My patch ended up special casing a single inbound bio that contained a
> write for a single full raid stripe. So for 8 drives raid-5, this is
> 7 * 64K or an IO 448KB long. With 4K pages this is a bi_io_vec array
> of 112 pages. Big for kernel memory generally, but easily handled by
> server systems. With more drives, you can be talking well over 1MB in
> a single bio call.
>
> The patch takes this special case write, makes sure it is raid-5 and
> layout 2, is not degraded and is not migrating. If all of these are
> true, the code allocates a new bi_io_vec and pages for the parity
> stripe, new bios for each drive, computes parity "in thread", and then
> issues simultanious IOs to all of the devices. A single bio complete
> function catches any errors and completes the IO.
>
> My testing is all done using SSDs. I have tests for 8 drives and for
> 32 partition on the 8 drives. The drives themselves do about
> 100MB/sec per drive. With the stock code I tend to get 550 MB/sec
> with 8 drives and 375 MB/sec with 32 partitions on 8 drives. With the
> patch, both 8 and 32 yield about 670 MB/sec which is within 5% of
> theoretical bandwidth.
>
> My "fix" for linear writes is probably way to "miopic" for general
> kernel use, but it does show that properly fed, really big raid/456
> arrays should be able to crank linear bandwidth far beyond the current
> code base.
>
> What is really needed is some general technique to give the raid
> driver a "hint" that an IO stream is linear writes so that it will not
> try to back-fill too eagerly. Exactly how this can make it back up
> the bio stack is the real trick.
>
> I am happy to discuss this on-list or privately.
>
> --
> Doug Dumitru
> EasyCo LLC
>
> ps: I am also working on patches to propagate "discard" requests
> through the raid stack, but don't have any operational code yet.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to
majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
