Hello,

I just had the opportunity to test some Fluke network equipment, notably
one which is able to throughput test gigabit networks.

I installed a Nexcom NSA1086 with OpenBSD 4.1-stable and did some tests.
The NSA1086 units are equipped with a Pentium IV 3.2Ghz (hyperthreading
disabled), and 1Gb ram. They have 4 sk gigabit interfaces, and 4 msk
gigabit interfaces. Here are the tests:

**********
ifconfig sk0 up
ifconfig sk1 up
ifconfig bridge0 create
ifconfig bridge0 up
brconfig bridge0 add sk0
brconfig bridge0 add sk1
sysctl kern.maxclusters=256000

Then I connected the fluke analyzers to both sk ports. connections were
correctly seen at 1000 base T full Duplex. then I started a throughput
test on 1Gbps, and I was extremely surprised to see how performance was
very poor. The throughput was only about 77Mbps.

***************

Seeing that I decided to try on msk interfaces and got about the same
"performance".

***************

Then I configured routing between two interfaces.
ifconfig sk2 inet 192.168.0.1 255.255.255.0
ifconfig sk3 inet 192.168.1.1 255.255.255.0
sysctl net.inet.ip.forwarding=1
sysctl kern.maxclusters=256000

I did the throughput test, and got about 500Mbps both on sk and msk,
with the CPU keeping quiet.

**************

So the weak performance doesn't seem related to the bus, the CPU, or the
sk/msk drivers. Has someone an explanation on why I get this kind of
behavior?

you forgot to enable sysctl net.inet.ip.forwarding=1 on the first test.  
Turn that on, then re-measure the speeds of the bridge.



--
Anthony's Law of Force:
	Don't force it; get a larger hammer.

bridging is layer 2 and has nothing to do with net.inet.ip.forwarding=1.
The problem is more somewhere deep inside the bridging code. One possible
problem could be a to small interface queue but I could be totaly wrong.
If nobody beats me I will have a look at this somewhen in the next month

I've had net.inet.ip.forwarding help on a bridge before.  Not saying 
there isn't a problem, but that I've gotten a performance increase from 
enabling it.


--
You can do this in a number of ways.  IBM chose to do all of them.
Why do you find that funny?
		-- D. Taylor, Computer Science 350, University of Washington

It can cause problems, though, too.  For example, if you have a bridge
in front of machines that do multicast-based IP load balancing
(specifically, Microsoft's NLB) then the packet duplication with
net.inet.ip.forwarding craps on your network.  Turning off
net.inet.ip.forwarding reduces the packet duplication and shit works
again.

-Mark

Possibly interrupt issues.  Where them dmesg'?

Here is the dmesg:
OpenBSD 4.1-stable (GENERIC) #0: Wed May  9 14:47:40 CEST 2007
    root@pufftest.lcl:/var/log/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR
real mem  = 1072197632 (1047068K)
avail mem = 970940416 (948184K)
using 4278 buffers containing 53735424 bytes (52476K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+ BIOS, date 08/29/05, BIOS32 rev. 0 @ 0xf9680,
SMBIOS rev. 2.2 @ 0xf0800 (39 entries)
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
apm0: flags 70102 dobusy 1 doidle 1
pcibios0 at bios0: rev 3.0 @ 0xf0000/0xbde4
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfbca0/272 (15 entries)
pcibios0: PCI Exclusive IRQs: 5 9 10 11 12
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801FB LPC" rev 0x00)
pcibios0: PCI bus #5 is the last bus
bios0: ROM list: 0xc0000/0xa800! 0xcc000/0x8000! 0xef000/0x1000!
acpi at mainbus0 not configured
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82915G/P/GV Host" rev 0x0e
vga1 at pci0 dev 2 function 0 "Intel 82915G/P/GV Video" rev 0x0e:
aperture at 0xd0500000, size 0x10000000
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 28 function 0 "Intel 82801FB PCIE" rev 0x04
pci1 at ppb0 bus 1
mskc0 at pci1 dev 0 function 0 "Marvell Yukon 88E8053" rev 0x15, Yukon-2
EC rev. A3 (0x2): irq 5
msk0 at mskc0 port A, address 00:10:f3:0e:cb:0d
eephy0 at msk0 phy 0: Marvell 88E1111 Gigabit PHY, rev. 2
ppb1 at pci0 dev 28 function 1 "Intel 82801FB PCIE" rev 0x04
pci2 at ppb1 bus 2
mskc1 at pci2 dev 0 function 0 "Marvell Yukon 88E8053" rev 0x15, Yukon-2
EC rev. A3 (0x2): irq 10
msk1 at mskc1 port A, address 00:10:f3:0e:cb:0e
eephy1 at msk1 phy 0: Mar...
[ message continues ]

No nothing showed in the logs. And I don't understand why such a
performance difference between routing and bridging.

I believe Marco requested the dmesg info and indicated it might be an 
interrupt issue.

# vmstat -i

# systat -w 1 vmstat

Well, I didn't understand it that way, but I will post the dmesg
tomorrow. Unfortunately, I cannot reproduce the 1Gbps tests as I don't
have the 1Gbps fluke analyzer anymore. I can only have access to one
that produces 155Mbps max throughput.