Hello,

This got even more interesting. After reading your email I had the idea to
start turning off the various carp interfaces to see what would be the
effect.
I have two onboard "Broadcom BCM5704C" and a "Intel PRO/1000MT QP (82546GB)"
quad nic.
One carp is configured for one onboard nic and two other for the quad nic.
I removed the two carps for the quad nic at backup node and rebooted it a
few times. There are no failures in iperf test (I used a long time to make
sure it was always running between all the tests) which is the same as your
tests and normal expected result.
Removing the onboard carp and activating both or one of the quad nic carps
gives the failures I reported previously. Without pfsync active in the
master node, I get a  small failure in iperf tests while the backup node is
coming back. If I activate pfsync, I get the same small failure plus
sometimes a total mess up of iperf connection states.
So it seems the problem is happening with the quad nic. I don't see any
performance problems with the quad nic because I left iperf running for 2
days without any problem. CPU usage in interrupts is around 15% and load
0.20 while doing tests. The firewall is still not in production, so only
traffic is only my test and internet junk being dropped.
Kernel is GENERIC 4.2 without any patches (I don't see any of them relevant
to this problem). I doubt about any hardware problems because the same
happens if I exchange their roles as master and backup.

I can't understand how the backup node can generate these results with a
reboot. While writing this I remembered to do another test. I destroyed the
quad nic carps (with ifconfig carpX destroy) and then brought them back with
sh /etc/netstart. Iperf keeps running smoothly this time... Master node
receives the bulk update requests without any problems. Did this a few times
and nothing happened.
Even more weird now !!! Something is being done while those interfaces got
up for the first time after the reboot!
Any ideas ?

Thanks,
John

On 10/04/2008, Calomel <avertege@calomel.org> wrote:
quoted text
>
> John,
>
> I ran a test using iperf on an external openbsd system (client) through a
> carp
> firewall to an internal openbsd system (server). All systems are running
> OpenBSD v4.2 with the latest patches.
>
>       external               ---> CARP --->  internal
> (iperf -i 1 -t 600 -c carp0)                (iperf -s)
>
> I did _not_ see any slow down through the MASTER when I rebooted the
> BACKUP
> server. For example, I started the reboot of the BACKUP at 5 seconds and
> the BACKUP finished rebooting at 102 seconds:
>
> [  3]  1.0- 2.0 sec  81.2 MBytes    681 Mbits/sec
> [  3]  2.0- 3.0 sec  82.3 MBytes    690 Mbits/sec
> [  3]  3.0- 4.0 sec  83.8 MBytes    703 Mbits/sec
> [  3]  4.0- 5.0 sec  86.6 MBytes    727 Mbits/sec -- start reboot
> [  3]  5.0- 6.0 sec  86.8 MBytes    728 Mbits/sec
> [  3]  6.0- 7.0 sec  86.3 MBytes    724 Mbits/sec
> [  3]  7.0- 8.0 sec  82.8 MBytes    695 Mbits/sec
> [  3]  8.0- 9.0 sec  86.7 MBytes    728 Mbits/sec
> [  3]  9.0-10.0 sec  85.8 MBytes    720 Mbits/sec
> [  3] 10.0-11.0 sec  86.1 MBytes    722 Mbits/sec
>
> ....cut....
>
> [  3] 96.0-97.0 sec  83.4 MBytes    699 Mbits/sec
> [  3] 97.0-98.0 sec  82.4 MBytes    692 Mbits/sec
> [  3] 98.0-99.0 sec  81.9 MBytes    687 Mbits/sec
> [  3] 99.0-100.0 sec  84.7 MBytes    710 Mbits/sec
> [  3] 100.0-101.0 sec  83.3 MBytes    699 Mbits/sec
> [  3] 101.0-102.0 sec  83.7 MBytes    702 Mbits/sec -- finished reboot
> [  3] 102.0-103.0 sec  83.3 MBytes    699 Mbits/sec
> [  3] 103.0-104.0 sec  83.6 MBytes    701 Mbits/sec
> [  3] 104.0-105.0 sec  85.3 MBytes    716 Mbits/sec
> [  3] 105.0-106.0 sec  83.4 MBytes    699 Mbits/sec
>
> I also did not see any errors in the logs of either system running ipref
> or on the firewalls. The load on the MASTER firewall was around 0.30.
>
> Are the firewalls kernel patched? Are their any hardware failures to
> report? Are the firewalls overloaded?
>
> You are welcome to check out some of the "how to's" I have at
> http://calomel.org if you need to.
>
>
> --
>   Calomel @ http://calomel.org
>   Open Source Research and Reference