I have 12 enterprise-class seagate 1TiB disks on a 965 desktop board and 
it appears I have hit the limit, if I were able to get the maximum speed 
of all drives, ~70MiB/avg * 12 = 840MiB/s but it seems to stop aound 774 
MiB/s (currently running badblocks on all drives)..

I am testing some drives for someone and was curious to see how far one 
can push the disks/backplane to their theoretical limit.

dstat output:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
   1  12   0  83   3   2|   -     0 |   0     0 |   0     0 |  13k   23k
   1  11   0  84   2   2| 774M    0 |2100B 7373B|   0     0 |  13k   23k
   1  12   0  83   3   2| 774M    0 |   0     0 |   0     0 |  13k   23k
   1  11   0  82   4   2| 774M    0 |2030B 5178B|   0     0 |  13k   23k
   1  11   0  83   4   2| 774M    0 |   0     0 |   0     0 |  13k   23k
   1  11   0  83   3   2| 774M    0 |2264B 6225B|   0     0 |  13k   23k

vmstat 1 output:
~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
  0 12    124 3841772      8  13012    0    0 12595 163880  379  352  0 31 37 32
  2 12    124 3841772      8  12992    0    0 791744     8 12796 23033  1 18  0 82
  0 12    124 3841772      8  12992    0    0 792192     0 12677 22918  1 15  0 84
  0 12    124 3841772      8  12992    0    0 792960     0 12894 22929  1 15  0 84

When I was writing to all of the drives it was maxing out around ~650 
MiB/s.

I also have 2 raptors on a PCI card (as I ran out of PCI-e cards) and:
When I read from 1 of the raptors (w/ dd/example shown below) the speed 
drops:

   1  13   0  79   5   3| 764M    0 |2240B 7105B|   0     0 |  12k   21k
   1  13   0  80   5   2| 764M    0 |   0     0 |   0     0 |  12k   21k
   1  11   0  82   5   2| 764M    0 |2170B 5446B|   0     0 |  12k   21k
   1  12   0  81   5   2| 762M    0 |  ...
[ message continues ]

Small correction, they are 7200.11 Seagate Desktop Drives (ST31000340AS), 
not enterprise drives:

http://www.seagate.com/ww/v/index.jsp?vgnextoid=0732f141e7f43110VgnVCM100000f5ee0a0aRCRD
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148274

--

[ message continues ]

http://www.intel.com/products/chipsets/g965/diagram.jpg

Basically it appears I am hammering the southbridge as for this board the 
PCI-e (x1) slots also traverse through the southbridge.

  6_SATA -> G965 ICH8
3_PCI-e -> G965 ICH8

From which has to ship that data across the DMI (2GB) link to the 
northbridge.

If one utilized a 12, 16 or 24 port raid card (but used SW RAID) on the 
x16 slot on the northbridge itself, would this barrier exist as the:
GMCH<->CPU is (8.5GB/s)..?

Also on the X38 and X48 the speed increases slightly:
http://www.intel.com/products/chipsets/X38/X38_Block_Diagram.jpg (10.6GB/s)
http://www.intel.com/products/chipsets/x48/x48_block_diagram.jpg (12.8GB/s)

If one asks why would one need such speed?

Example:
LTO-4 drives can write at 120 MiB/s each:
http://en.wikipedia.org/wiki/Linear_Tape-Open

It is then therefore imperative one could sustain this rate to possibly 
multiple(!) tape drives on a single system.

Justin.
--

[ message continues ]

It looks like graphic games are pushing the technologies to their
limits, which is good for us. I have bought X38 motherboards for
10 Gbps experimentations, and this chipset is perfectly capable of
feeding two Myri10GE NICs (20 Gbps total). This is 2.5 GB/s, not
counting overhead. So I/O bandwidth is a premium requirement today.

Other chipsets I have tested (945 and 965) were very poor (about 4.7
and 6.5 Gbps respectively if my memory serves me right).

Willy

--

[ message continues ]

This testing would indeed only suggest theoretical limits.  In a
production environment, I think a person would be hard pressed to

Nope, but your post inspired me to give it a try.  My setup is as
follows:

Kernel:			linux 2.6.25.3-18 (Fedora 9)
Motherboard:		Intel SE7520BD2-DDR2
SATA Controller:	(2) 8 port 3Ware 9550SX
Disks			(12) 750GB Seagate ST3750640NS

Disks sd[a-h] are plugged into the first 3Ware controller while
sd[i-l] are plugged into the second controller.  Both 3Ware cards
are plugged onto PCIX 100 slots.  The disks are being exported as
"single disk" and write caching has been disabled.  The OS is 
loaded on sd[a-d] (small 10GB partitions mirrored). For my first 
test, I ran dd on a single disk:

dd if=/dev/sde of=/dev/null bs=1M

dstat -D sde

----total-cpu-usage---- --dsk/sde-- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   7  53  40   0   0|  78M    0 | 526B  420B|   0     0 |1263  2559 
  0   8  53  38   0   0|  79M    0 | 574B  420B|   0     0 |1262  2529 
  0   7  54  39   0   0|  78M    0 | 390B  420B|   0     0 |1262  2576 
  0   7  54  39   0   0|  76M    0 | 284B  420B|   0     0 |1216  2450 
  0   8  54  38   0   0|  76M    0 | 376B  420B|   0     0 |1236  2489 
  0   9  54  36   0   0|  79M    0 | 397B  420B|   0     0 |1265  2537 
  0   9  54  37   0   0|  77M    0 | 344B  510B|   0     0 |1262  2872 
  0   8  54  38   0   0|  75M    0 | 637B  420B|   0     0 |1214  2992 
  0   8  53  38   0   0|  78M    0 | 422B  420B|   0     0 |1279  3179 

And for a write:

dd if=/dev/zero of=/dev/sde bs=1M

dstat -D sde

----total-cpu-usage---- --dsk/sde-- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   7   2  90   0   0|   0    73M| 637B  420B|   0     0 | 614   166 
  0   7   0  93   0   0|   0    73M| 344B  420B|   0     0 | 586   105 
  0   7   0  93   0   0|   0    75M| 344B  420B|   0     0 | 629   177 
  0   ...
[ message continues ]