Re: CARP node crashing reproducibly (4.3-stable)

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: misc <misc@...>
Cc: <lists-openbsd@...>, Stephan A. Rickauer <stephan.rickauer@...>, <reyk@...>
Date: Monday, July 14, 2008 - 6:38 am

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Henning Brauer wrote:
| * Stephan A. Rickauer [2008-07-11
16:59]:
|> Here's all data I was able to get off our crashing machine, the backup
|> node of our CARP cluster, that used to run flawlessly since 3.7.
|>
|> We can reproduce the problem
|
| if you follow http://www.benzedrine.cx/crashreport.html we have a
| chance to actually fix the bug...
|

Hello,
I'm a colleague of Stephan Rickauer and I've been taking a look at this
problem.

It's a NULL pointer bug!

dmesg shows
kernel: page fault trap, code=0
Stopped at pf_send_icmp+0x2b: orb

and ddb trace shows:

$0x1,0x32(%eax)pf_send_icmp(d62f3200,3,3,2,d67191b8,d115d500,2,db2a4eb8)
at pf_send_icmp+0x2b

ddb registers shows (among others):

eax 0
eip 0xd02f56db pf_send_icmp+0x2b

and helpfully disassembles the faulting instruction thus:

pf_send_icmp+0x2b: orb $0x1,0x32(%eax)

which is from line 1726 in pf_send_icmp() in pf.c:

m0->m_pkthdr.pf.flags |= PF_TAG_GENERATED;

The beginning of this function (up to the line with the or) is as follows:

pf_send_icmp(struct mbuf *m, u_int8_t type, u_int8_t code, sa_family_t af,
~ struct pf_rule *r)
{
struct mbuf *m0;

m0 = m_copy(m, 0, M_COPYALL);
m0->m_pkthdr.pf.flags |= PF_TAG_GENERATED;

Thus we have m_copy (actually m_copym, since m_copy is a macro defined
in /usr/src/sys/sys/mbuf.h in terms of m_copym, which itself is a
one-line wrapper around m_copym0) returning a NULL pointer in eax (= m0)
and the subsequent OR getting a page fault when it tries to use it.

Looking at m_copym0, it looks like it can legitimately fail and return
NULL (it even increments a global variable MCFail when it does so) and
therefore the bug is that its return value is not being checked in
pf_send_icmp.

As far as I can see, the precise nature of the packet being handled at
the time of the crash is not important. Using ddb on the crashed
machine, it looks as if the packet being handled at the time is a
(relatively) innocent UDP broadcast as follows:

IP header:
45 0 0 1d
0 0 0 0
40 11 1b a2
ac 10 3 f
ac 10 3 ff

ip header length = 5 32-bit words
length = 29
id = 0
flags = 0
fragmentation offset = 0
TTL = 64
Protocol = 17, UDP
Source address = 172.16.3.15 (zynapse.lan.ini.uzh.ch)
Dest address = 172.16.3.255

UDP header:
bb b5 22 3d
0 9 a5 ba

source port = bbb5 = 48053
dest port = 223d = 8765 (Ultraseek HTTP ?)
length = 9

Data:
1d

Adrian

- --
Adrian M. Whatley
Universitaet/ETH Zuerich,
Institut fuer Neuroinformatik,
Winterthurerstrasse 190,
CH-8057 Zuerich, Switzerland.
Phone: +41 44 635 3067 Fax: +41 44 635 3053
Email: amw@ini.phys.ethz.ch WWW: http://www.ini.uzh.ch/~amw/
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFIeyy7Lgk3RqYSp9YRAlgfAJ4wYygStPwwScv9eScXXjIRtwc4oQCghkTb
rUhs3B5ZZPkyMQwXxyg9Xys=
=0Dyq
-----END PGP SIGNATURE-----

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
CARP node crashing reproducibly (4.3-stable), Stephan A. Rickauer, (Fri Jul 11, 10:55 am)
Re: CARP node crashing reproducibly (4.3-stable), Henning Brauer, (Fri Jul 11, 3:32 pm)
Re: CARP node crashing reproducibly (4.3-stable), Adrian M. Whatley, (Mon Jul 14, 6:38 am)
Re: CARP node crashing reproducibly (4.3-stable), Henning Brauer, (Mon Jul 14, 8:22 am)
Re: CARP node crashing reproducibly (4.3-stable), Stephan A. Rickauer, (Mon Jul 14, 11:12 am)
Re: CARP node crashing reproducibly (4.3-stable), Henning Brauer, (Mon Jul 14, 11:38 am)
Re: CARP node crashing reproducibly (4.3-stable), Stephan A. Rickauer, (Wed Jul 23, 2:44 pm)
Re: CARP node crashing reproducibly (4.3-stable), Stephan A. Rickauer, (Fri Jul 11, 3:47 pm)
Re: CARP node crashing reproducibly (4.3-stable), Reyk Floeter, (Fri Jul 11, 11:09 am)
Re: CARP node crashing reproducibly (4.3-stable), Stephan A. Rickauer, (Fri Jul 11, 11:25 am)
Re: CARP node crashing reproducibly (4.3-stable), Giancarlo Razzolini, (Fri Jul 11, 2:42 pm)