On Wed, Nov 11, 2009 at 02:30:25PM -0600, Marco Peereboom wrote:
quoted text
> You defeated read ahead with your RAID test and found out that kernel
> crypto is slow.  Oh and you tested softdep pretty well too because that
> is what you were trying to measure, right?
> 
> Lies, damn lies and benchmarks...

Lies, damn lies, statistics, and *&*%^* benchmarks I think is the correct
ordering.

.... Ken

quoted text
> 
> On Wed, Nov 11, 2009 at 08:09:40PM +0100, Robert wrote:
> > Warning, long post...
> >
> > This whole performance discussion made me curious, and since I had 2
> > identical SATA disks available I made some tests with softraid and
> > crypto (dmesg at the end).
> >
> > summary (MB/sec):
> > write	read	layout
> > 80	80	single disk / raw
> > 68	80	single disk / file system
> > 30	30	single disk + softraid crypto / raw
> > 28	30	single disk + softraid crypto / file system
> > 21	50	single disk + VN crypto / raw
> > 8	37	single disk + VN crypto / file system
> > 80	78	2 disk softraid RAID1 / raw
> > 70	76	2 disk softraid RAID1 / file system
> > 21	32	2 disk softraid RAID1 + VN crypto / raw
> > 20	22	2 disk softraid RAID1 + VN crypto / file system
> > 30	22	2 disk softraid RAID1 + softraid crypto / raw
> > 28	22	2 disk softraid RAID1 + softraid crypto / file system
> >
> > One thing that I don't understand: read performance with RAID1 is
> > basically the same as with a single disk. I would have expected way
> > better results, since data can be read from both disks at the same time
> > (which means in theory 200% performance)?
> >
> >
> > Now for the boring ;) numbers:
> >
> > *) CPU usage was measured with "top -S -s 0.1"
> > *) interrupts/sec was measured with "systat -s 0.1 vmstat"
> >
> >
> > 1) simple disk
> >
> >
> > 1.1) raw speed
> > # dd if=/dev/zero of=/dev/rwd1c bs=1m count=500
> > 500+0 records in
> > 500+0 records out
> > 524288000 bytes transferred in 6.470 secs (81022423 bytes/sec)
> > # dd if=/dev/rwd1c of=/dev/null bs=1m count=500
> > 500+0 records in
> > 500+0 records out
> > 524288000 bytes transferred in 6.499 secs (80667486 bytes/sec)
> >
> > # dd if=/dev/zero of=/dev/rwd2c bs=1m count=500
> > 500+0 records in
> > 500+0 records out
> > 524288000 bytes transferred in 6.360 secs (82429854 bytes/sec)
> > # dd if=/dev/rwd2c of=/dev/null bs=1m count=500
> > 500+0 records in
> > 500+0 records out
> > 524288000 bytes transferred in 6.268 secs (83632786 bytes/sec)
> > (as expected, since the disks are identical...)
> >
> > *) CPU load nearly none, interrupts ca. 1.000/sec
> >
> >
> > 1.2) simple file system
> > # fdisk -iy wd1
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" |disklabel -E wd1
> > # newfs wd1a
> > # mount -o softdep /dev/wd1a /mnt/test
> >
> > # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
> >
> >
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 15.412 secs (68036212 bytes/sec)
> > *) interrupts went up to 5.000/sec (jumping around) and CPU was aroung
> > 10% for dd
> >
> > # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 13.079 secs (80168224 bytes/sec)
> > *) interrupts at 1.500/sec and 5% CPU for dd
> >
> >
> > 2) softraid crypto on 1 disk
> >
> > # fdisk -iy wd1
> > # printf "a\n\n\n\nRAID\nw\nq\n\n" |disklabel -E wd1
> > # bioctl -c C -l /dev/wd1a softraid0
> > scsibus1 at softraid0: 1 targets
> > sd0 at scsibus1 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
> > sd0: 476937MB, 512 bytes/sec, 976767923 sec total
> > # fdisk -iy sd0
> >
> > 2.1) raw speed
> > # dd if=/dev/zero of=/dev/rsd0c bs=1m count=100
> > 100+0 records in
> > 100+0 records out
> > 104857600 bytes transferred in 3.485 secs (30086175 bytes/sec)
> > # dd if=/dev/rsd0c of=/dev/null bs=1m count=100
> > 100+0 records in
> > 100+0 records out
> > 104857600 bytes transferred in 3.452 secs (30374596 bytes/sec)
> >
> > *) crypto just decreased the speed to 37%...
> > *) interrupts were at ca. 500/second
> > *) CPU load of dd kept climbing (60%+); it seems that the CPU/encryption
> > speed is the limiting factor here (?)
> >
> >
> > 2.2) writing a file in crypto
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" |disklabel -E sd0
> > # newfs sd0a
> > # mount -o softdep /dev/sd0a /mnt/test
> > # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 37.349 secs (28074519 bytes/sec)
> >
> > *) little less than raw speed
> > *) interesting note: the interrupts (for pciide) kept jumping between
> > 500-1700 (??); CPU load for dd was aroung 20%
> >
> >
> > 2.3) reading from the same file
> > # sync
> > # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 34.888 secs (30055309 bytes/sec)
> >
> > *) CPU load was around 3%; decrypting seems to be WAY less expensive
> > *) interrupts were around 500
> >
> >
> > 3) SVN crypto on 1 disk
> >
> > # fdisk -iy wd1
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" |disklabel -E wd1
> > # newfs wd1a
> > # mount -o softdep /dev/wd1a /mnt/test
> > # dd if=/dev/arandom of=/mnt/test/crypt.file bs=1m count=2000
> > # vnconfig -ck svnd0 /mnt/test/crypt.file
> > # fdisk -iy svnd0
> >
> >
> > 3.1) raw speed
> > # dd if=/dev/zero of=/dev/rsvnd0c bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 49.077 secs (21365694 bytes/sec)
> > # dd if=/dev/rsvnd0c of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 21.019 secs (49885388 bytes/sec)
> > *) This used up all the CPU and interrupts jumped between 0-5.000; the
> > system was unusable meanwhile.
> >
> >
> > 3.2) file system speed
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" | disklabel -E svnd0
> > # newfs svnd0a
> > # mount -o softdep /dev/svnd0a /mnt/crypto
> >
> > # dd if=/dev/zero of=/mnt/crypto/test.file bs=1m count=1000
> >
> >
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 134.516 secs (7795120 bytes/sec)
> > *) interrupts at 1.000/sec and CPU for dd was 35%
> > *) I retried this; it's really just 8MB/sec...
> >
> > # dd if=/mnt/crypto/test.file of=/dev/null bs=1m count=1000
> >
> >
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 28.189 secs (37196955 bytes/sec)
> > *) this used all the CPU and interrupts were at 500/sec
> >
> >
> > 4) software RAID1
> >
> > 4.1) First I checked if the SATA controller has a problem if both drives
> > have a constant transfer at the same time, so I did the raw test from
> > above again, this time for wd1 and wd2 at the same time. The results
> > were the same.
> > Try this with a cheap SATA/IDE controller... (hint: DON'T!)
> >
> > # fdisk -iy wd1
> > # fdisk -iy wd2
> > # printf "a\n\n\n\nRAID\nw\nq\n\n" | disklabel -E wd1
> > # printf "a\n\n\n\nRAID\nw\nq\n\n" | disklabel -E wd2
> > # bioctl -c 1 -l /dev/wd1a,/dev/wd2a softraid0
> > scsibus1 at softraid0: 1 targets
> > sd0 at scsibus1 targ 0 lun 0: <OPENBSD, SR RAID 1, 003> SCSI2 0/direct fixed
> > sd0: 476937MB, 512 bytes/sec, 976767923 sec total
> >
> >
> > 4.2) raw speed
> > # dd if=/dev/zero of=/dev/rsd0c bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 13.084 secs (80140652 bytes/sec)
> > # dd if=/dev/rsd0c of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 13.449 secs (77963475 bytes/sec)
> >
> > *) basically no penalty because of RAID1, but also no read improvement
> > *) CPU load was nearly none; interrupts went to 2.200/sec (write) and
> > 1.100/sec (read)
> >
> >
> > 4.3) file system speed
> > # fdisk -iy sd0
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" | disklabel -E sd0
> > # newfs sd0a
> > # mount -o softdep /dev/sd0a /mnt/test
> >
> > # dd if=/dev/zero of=/mnt/test/test.file bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 15.084 secs (69511723 bytes/sec)
> > # sync
> > # dd if=/mnt/test/test.file of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 13.777 secs (76105178 bytes/sec)
> >
> > *) interrupts went up to 5.000/sec and CPU was around 15% during write;
> > 1.200/sec interrupts and 5% CPU during read
> >
> >
> > 5) SVN crypto on top of RAID1
> >
> > For this I take the existing RAID1 from the last example and create a
> > file (container) in it.
> > # dd if=/dev/arandom of=/mnt/test/crypto.file bs=1m count=1000
> > # vnconfig -ck svnd0 /mnt/test/crypto.file
> > # fdisk -iy svnd0
> >
> >
> > 5.1) raw speed
> > # dd if=/dev/zero of=/dev/rsvnd0c bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 50.070 secs (20942016 bytes/sec)
> > *) This used all of the CPU and interrupts jumped around between 0-5.000
> > (no useful measures). Again, the CPU seems to be the limit here. The
> > system was basically unusable during this test.
> > # dd if=/dev/rsvnd0c of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 32.732 secs (32034694 bytes/sec)
> > *) CPU was aroung 40% and interrupts at 500/sec; but the system was usable
> >
> >
> > 5.2) file system speed
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" | disklabel -E svnd0
> > # newfs svnd0a
> > # mount -o softdep /dev/svnd0a /mnt/crypto
> >
> > # dd if=/dev/zero of=/mnt/crypto/test.file bs=1m count=900
> > 900+0 records in
> > 900+0 records out
> > 943718400 bytes transferred in 46.859 secs (20139196 bytes/sec)
> > *) This used all of the CPU and interrupts jumped around between 0-5.000
> > (no useful measures). Again, the CPU seems to be the limit here. The
> > system was basically unusable during this test.
> > # dd if=/mnt/crypto/test.file of=/dev/null bs=1m count=900
> > 900+0 records in
> > 900+0 records out
> > 943718400 bytes transferred in 42.361 secs (22277956 bytes/sec)
> > *) CPU was aroung 40% and interrupts at 500/sec; but the system was usable
> >
> >
> > 6) softraid crypto on top of RAID1
> > NOTE: I have been warned that this is currently not supported (softraid
> > on softraid), but I still wanted to test it ;)
> > The existing RAID1 (sd0) from the example above is used.
> >
> > # fdisk -iy sd0
> > # printf "a\n\n\n\nRAID\nw\nq\n\n" | disklabel -E sd0
> > # bioctl -c C -l /dev/sd0a softraid0
> > scsibus2 at softraid0: 1 targets
> > sd1 at scsibus2 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
> > sd1: 476929MB, 512 bytes/sec, 976751858 sec total
> > # fdisk -iy sd1
> >
> > 6.1) raw speed
> > # dd if=/dev/zero of=/dev/rsd1c bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 35.323 secs (29684642 bytes/sec)
> > *) CPU went up to 60%+; interrupts were around 600/sec
> >
> > # dd if=/dev/rsd1c of=/dev/null bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 47.620 secs (22019455 bytes/sec)
> > *) Interrupts were around 300/sec, but there was nearly no CPU usage -
> > what's going on here? Decrypting should be expensive?? I think the
> > physical IO was so slow that the data was already decrypted meanwhile...
> >
> >
> > 6.2) file system speed
> > # printf "a\n\n\n\n4.2BSD\nw\nq\n\n" | disklabel -E sd1
> > # newfs sd1a
> > # mount -o softdep /dev/sd1a /mnt/crypto
> >
> > # dd if=/dev/zero of=/mnt/crypto/test.file bs=1m count=1000
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 37.923 secs (27649482 bytes/sec)
> > *) interrupts were around 1.500/sec and CPU load for dd was 15%
> >
> > # dd if=/mnt/crypto/test.file of=/dev/null bs=1m count=1000
> >
> >
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes transferred in 48.426 secs (21653069 bytes/sec)
> > *) interrupts at 300/sec and CPU around 2%
> >
> >
> >
> > dmesg:
> >
> > bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xf0450 (77 entries)
> > bios0: vendor Dell Inc. version "2.6.1" date 12/06/2007
> > bios0: Dell Inc. OptiPlex 745
> > acpi0 at bios0: rev 2
> > acpi0: tables DSDT FACP SSDT APIC BOOT ASF! MCFG HPET TCPA SLIC SSDT  
> > SSDT SSDT
> > acpi0: wakeup devices VBTN(S4) PCI0(S5) PCI4(S5) PCI2(S5) PCI3(S5)  
> > PCI1(S5) PCI5(S5) PCI6(S5) MOU_(S3) USB0(S3) USB1(S3) USB2(S3) USB3(S3)  
> > USB4(S3)
> > acpitimer0 at acpi0: 3579545 Hz, 24 bits
> > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz, 2128.26 MHz
> > cpu0:  
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
> > cpu0: 2MB 64b/line 8-way L2 cache
> > cpu0: apic clock running at 265MHz
> > cpu1 at mainbus0: apid 1 (application processor)
> > cpu1: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz, 2128.00 MHz
> > cpu1:  
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG
> > cpu1: 2MB 64b/line 8-way L2 cache
> > ioapic0 at mainbus0 apid 8 pa 0xfec00000, version 20, 24 pins
> > ioapic0: misconfigured as apic 0, remapped to apid 8
> > acpihpet0 at acpi0: 14318179 Hz
> > acpiprt0 at acpi0: bus 4 (PCI4)
> > acpiprt1 at acpi0: bus 2 (PCI2)
> > acpiprt2 at acpi0: bus -1 (PCI3)
> > acpiprt3 at acpi0: bus 1 (PCI1)
> > acpiprt4 at acpi0: bus 3 (PCI5)
> > acpiprt5 at acpi0: bus -1 (PCI6)
> > acpiprt6 at acpi0: bus 0 (PCI0)
> > acpicpu0 at acpi0: PSS
> > acpicpu1 at acpi0: PSS
> > acpibtn0 at acpi0: VBTN
> > cpu0: Enhanced SpeedStep 2128 MHz: speeds: 2133, 1867, 1600 MHz
> > memory map conflict 0xbfe03c00/0x1fc400
> > pci0 at mainbus0 bus 0
> > pchb0 at pci0 dev 0 function 0 "Intel 82Q965 Host" rev 0x02
> > ppb0 at pci0 dev 1 function 0 "Intel 82Q965 PCIE" rev 0x02: apic 8 int  
> > 16 (irq 11)
> > pci1 at ppb0 bus 1
> > vga1 at pci1 dev 0 function 0 "ATI Radeon HD 3650" rev 0x00
> > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> > wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> > azalia0 at pci1 dev 0 function 1 "ATI Radeon HD 3600 HD Audio" rev 0x00:  
> > apic 8 int 17 (irq 10)
> > azalia0: codecs: ATI/0xaa01
> > audio0 at azalia0
> > uhci0 at pci0 dev 26 function 0 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 16 (irq 11)
> > uhci1 at pci0 dev 26 function 1 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 17 (irq 10)
> > ehci0 at pci0 dev 26 function 7 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 22 (irq 9)
> > usb0 at ehci0: USB revision 2.0
> > uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > azalia1 at pci0 dev 27 function 0 "Intel 82801H HD Audio" rev 0x02: apic  
> > 8 int 16 (irq 11)
> > azalia1: codecs: Analog Devices AD1983
> > audio1 at azalia1
> > ppb1 at pci0 dev 28 function 0 "Intel 82801H PCIE" rev 0x02: apic 8 int  
> > 16 (irq 11)
> > pci2 at ppb1 bus 2
> > ppb2 at pci0 dev 28 function 4 "Intel 82801H PCIE" rev 0x02: apic 8 int  
> > 16 (irq 11)
> > pci3 at ppb2 bus 3
> > bge0 at pci3 dev 0 function 0 "Broadcom BCM5754" rev 0x02, BCM5754/5787  
> > A2 (0xb002): apic 8 int 16 (irq 11), address 00:1e:4f:93:35:bf
> > brgphy0 at bge0 phy 1: BCM5787 10/100/1000baseT PHY, rev. 0
> > uhci2 at pci0 dev 29 function 0 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 23 (irq 5)
> > uhci3 at pci0 dev 29 function 1 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 17 (irq 10)
> > uhci4 at pci0 dev 29 function 2 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 18 (irq 3)
> > ehci1 at pci0 dev 29 function 7 "Intel 82801H USB" rev 0x02: apic 8 int  
> > 23 (irq 5)
> > usb1 at ehci1: USB revision 2.0
> > uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xf2
> > pci4 at ppb3 bus 4
> > uhci5 at pci4 dev 0 function 0 "VIA VT83C572 USB" rev 0x62: apic 8 int  
> > 16 (irq 11)
> > uhci6 at pci4 dev 0 function 1 "VIA VT83C572 USB" rev 0x62: apic 8 int  
> > 17 (irq 10)
> > ehci2 at pci4 dev 0 function 2 "VIA VT6202 USB" rev 0x65: apic 8 int 18  
> > (irq 3)
> > usb2 at ehci2: USB revision 2.0
> > uhub2 at usb2 "VIA EHCI root hub" rev 2.00/1.00 addr 1
> > "VIA VT6306 FireWire" rev 0x46 at pci4 dev 0 function 3 not configured
> > cmpci0 at pci4 dev 2 function 0 "C-Media Electronics CMI8738/C3DX Audio"  
> > rev 0x10: apic 8 int 18 (irq 3)
> > audio2 at cmpci0
> > opl0 at cmpci0: model OPL3
> > midi0 at opl0: <CMPCI Yamaha OPL3>
> > mpu at cmpci0 not configured
> > usb3 at uhci5: USB revision 1.0
> > uhub3 at usb3 "VIA UHCI root hub" rev 1.00/1.00 addr 1
> > usb4 at uhci6: USB revision 1.0
> > uhub4 at usb4 "VIA UHCI root hub" rev 1.00/1.00 addr 1
> > pcib0 at pci0 dev 31 function 0 "Intel 82801H LPC" rev 0x02
> > pciide0 at pci0 dev 31 function 2 "Intel 82801H SATA" rev 0x02: DMA,  
> > channel 0 configured to native-PCI, channel 1 configured to native-PCI
> > pciide0: using apic 8 int 20 (irq 9) for native-PCI interrupt
> > wd0 at pciide0 channel 0 drive 0: <WDC WD800JD-75MSA3>
> > wd0: 16-sector PIO, LBA48, 76293MB, 156250000 sectors
> > wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
> > atapiscsi0 at pciide0 channel 1 drive 0
> > scsibus0 at atapiscsi0: 2 targets
> > cd0 at scsibus0 targ 0 lun 0: <Optiarc, DVD RW AD-7191S, 1.01> ATAPI  
> > 5/cdrom removable
> > cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5
> > ichiic0 at pci0 dev 31 function 3 "Intel 82801H SMBus" rev 0x02: apic 8  
> > int 20 (irq 9)
> > iic0 at ichiic0
> > spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-5300CL5
> > spdmem1 at iic0 addr 0x51: 2GB DDR2 SDRAM non-parity PC2-5300CL5
> > spdmem2 at iic0 addr 0x52: 2GB DDR2 SDRAM non-parity PC2-5300CL5
> > spdmem3 at iic0 addr 0x53: 2GB DDR2 SDRAM non-parity PC2-5300CL5
> > pciide1 at pci0 dev 31 function 5 "Intel 82801H SATA" rev 0x02: DMA,  
> > channel 0 wired to native-PCI, channel 1 wired to native-PCI
> > pciide1: using apic 8 int 20 (irq 9) for native-PCI interrupt
> > wd1 at pciide1 channel 0 drive 0: <WDC WD5000AAKS-65YGA0>
> > wd1: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
> > wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
> > wd2 at pciide1 channel 1 drive 0: <WDC WD5000AAKS-65YGA0>
> > wd2: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
> > wd2(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
> > usb5 at uhci0: USB revision 1.0
> > uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> > usb6 at uhci1: USB revision 1.0
> > uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> > usb7 at uhci2: USB revision 1.0
> > uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> > usb8 at uhci3: USB revision 1.0
> > uhub8 at usb8 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> > usb9 at uhci4: USB revision 1.0
> > uhub9 at usb9 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> > isa0 at pcib0
> > isadma0 at isa0
> > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > pckbc0 at isa0 port 0x60/5
> > pckbd0 at pckbc0 (kbd slot)
> > pckbc0: using irq 1 for kbd slot
> > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > pcppi0 at isa0 port 0x61
> > midi1 at pcppi0: <PC speaker>
> > spkr0 at pcppi0
> > lpt0 at isa0 port 0x378/4 irq 7
> > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> > fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> > mtrr: Pentium Pro MTRR support
> > uhidev0 at uhub9 port 1 configuration 1 interface 0 "Primax Electronics  
> > USB Optical Mouse" rev 2.00/2.00 addr 2
> > uhidev0: iclass 3/1
> > ums0 at uhidev0: 3 buttons, Z dir
> > wsmouse0 at ums0 mux 0
> > uhidev1 at uhub5 port 1 configuration 1 interface 0 "Belkin Components  
> > USB-PS2 Adapter" rev 1.10/1.20 addr 2
> > uhidev1: iclass 3/1
> > ukbd0 at uhidev1: 8 modifier keys, 6 key codes
> > wskbd1 at ukbd0 mux 1
> > wskbd1: connecting to wsdisplay0
> > uhidev2 at uhub5 port 1 configuration 1 interface 1 "Belkin Components  
> > USB-PS2 Adapter" rev 1.10/1.20 addr 2
> > uhidev2: iclass 3/1
> > ums1 at uhidev2: 5 buttons, Z dir
> > wsmouse1 at ums1 mux 0
> > softraid0 at root
> > root on wd0a swap on wd0b dump on wd0b