Hi,

this should have gone into the thread 'spurious "need to frag"
messages'. Sorry for opening a new thread.

On Wed, 31.03.2010 at 13:36:48 +0200, Toni Mueller <openbsd-misc@oeko.net> wrote:
quoted text
> recently, a problem with OpenBSD has popped up over here that manifests
> itself in "random" connection failures after some time. Network
> diagram:
> 
>  workstation (1) --- (3b) firewall (3a) --- Internet --- www.example.com (2)
> 
> You surf from your workstation to www.example.com. On the firewall, you
> can see packets flowing, on the exterior interface.
> 
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
> 
> and so on. Everything works just fine. Now, with nothing changed except
> for the firewall being up some days (currently: 13 days), and having
> pushed some traffic already, connections start to fail:
> 
> On (3a), you see "almost" the same packet sequence like shown above,
> shortened for brevity:
> 
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)    <- point where the connection fails
>  (2) -> (1)
>  (2) -> (1)
>  (2) -> (1)
>  (2) -> (1)
> 
> but on (3b), you see:
> 
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
>  (2) -> (1)
>  (1) -> (2)
> 
> and then nothing more, like if the web server on the other side had
> stopped sending packets. I can't see the packets on pflog0, either, and
> using slightly different networking to "bypass" the firewall,
> everything works still fine, but "fixing" the problem involves powering
> down the firewall.  Simply rebooting it w/o powering it down, does not
> fix the problem.

investigating further, the firewall starts to send ICMP packets on (3a)
to the other sides "(2)", claiming that

1.2.3.4 > 92.122.217.187: icmp: 1.2.3.24 unreachable - need to frag (mtu 1420)

Which is wrong. I've verified that the complete path supports an MTU of
1500 bytes. FWIW, the machines 1.2.3.4 and 1.2.3.24 are connected via a
100MBit/s Ethernet (machine - switch - machine, << 10 m of cabling).
I've also verified the pf configuration, and there's nothing in there
that lowers the MTU:

# grep -v '^#' pf* |grep -F 1420
pf.os:4096:64:0:44:M1420:               NewtonOS:2.1::NewtonOS 2.1
#

quoted text
> This problem first occurred for us with 4.6-stable on both i386 and
> amd64, and now also occurred on -current with kernel 448 on i386. I'm
> underway trying to get yet-more-recent stuff installed to see whether
> the problem is fixed.

Experimenting with several variations of OpenBSD, like 4.6-stable and
some snapshots yields that with ever more recent versions of -current,
the problem seems to deteriorate, going down from almost two weeks with
kernel #448 on i386 to less than one week with kernel #148 on amd64,
with about no change in configuration except adding a few more packet
filter rules, currently around 500-600 after replacing many macros with
tables. The machine has moved some 300-400 mio packets since booting
about six days ago.

I'm probably going to try yesterday's snapshot, too.

Since I also swapped in a brand new, bigger machine to do the job, I am
confident that there is no hardware problem involved. The remaining
idea from my side is to suspect some kind of kernel memory corruption.

Unfortunately, I can so far only detect the problem on the outside of
the network by observing the packets emitted by the machine(s) in
question, but I failed to find a way to diagnose the problem from
within the machine(s) itself (except for using tcpdump, which is a bit
besides the point).


-- 
Kind regards,
--Toni++