I have been trying to optimize my raid throughput. I am trying to read hundred of files simultaneously from 6 disks raid 5 array (3ware).I found out the weirdest things:
1. xfs degrades linear read performance. just loading this filesystem degrades a linear read from the raid.
2.I have written a simple c program that reads hundred of files synchronously (file after file )from ext2 filesystem. I got a terrible performance. The minute i switched to direct io the performance improved greatly. I have no idea why. Could it be that the page cache radix degrades the performance ? I must add that when i read these files I am using 10 meg buffers and the cache has no meaning because i do not read the same block twice .
I would be happy to hear any ideas for this odd behaviour.