ok it seems to be pf.c line 3397 (mybe littleoffset for you,i have

chnges in my tree),which is

                                pool_put(&pf_src_tree_pl, nsn);

in the second if block after the cleanup label. this is in source node

tracking, as in, not new or changed code. crash there means we either

tried to pool_put something invalid (double free style?) or we have

pool corrpution.
* Josh  [2008-06-08 17:11]:

quoted text
> I have had both firewalls in a carp/pfsync pair which are running the same snapshot crash;

>

>

> uvm_fault(0xd07f81e0, 0x0, 0, 1) -> e

> kernel: page fault trap, code=0

> Stopped at      pf_test_rule+0x8a0:     movl    0x58(%eax),%ecx

> ddb> pf_test_rule(d51efd64,d51efd5c,1,d0ba8500,db53ad00) at pf_test_rule+0x8a0

> pf_test(1,d0c54800,d51efe64,0) at pf_test+0x8c1

> ipv4_input(db53ad00,d0b9a180,68400000,7c10c) at ipv4_input+0x124

> ipintr(58,10,d51e0010,d0480010,7c10c) at ipintr+0x64

> Bad frame pointer: 0xd51efe7c

> ddb>    PID   PPID   PGRP    UID  S       FLAGS  WAIT          COMMAND

>  29392  24940  24940      0  2     0x40100                sendmail

>    897  22729    897      0  2      0x4002                top

>  22729  16691  22729      0  3      0x4082  pause         ksh

>  16691   9570  16691      0  2      0x4180                sshd

>  17835   3928  17835      0  3      0x4082  ttyin         ksh

>   3928   9570   3928      0  2      0x4180                sshd

>   6950      1   6950      0  3      0x4082  ttyin         getty

>  27487      1  27487      0  3      0x4082  ttyin         getty

>   1375      1   1375      0  3      0x4082  ttyin         getty

>  11547      1  11547      0  3      0x4082  ttyin         getty

>   3455      1   3455      0  3      0x4082  ttyin         getty

>   4587      1   4587      0  2           0                cron

>   9570      1   9570      0  3        0x80  select        sshd

>  24940      1  24940      0  2     0x40100                sendmail

>    138      1    138      0  3       0x180  select        inetd

>  30566  14663  14663     83  2       0x100                ntpd

>  14663      1  14663      0  2           0                ntpd

>  10627  31688  31688     70  3       0x100  uvn_getpage   named

>  31688      1  31688      0  3       0x180  netio         named

>   5097   8724   8724     74  2       0x100                pflogd

>   8724      1   8724      0  3        0x80  netio         pflogd

>  16751   7245   7245     73  2       0x100                syslogd

>   7245      1   7245      0  3        0x88  netio         syslogd

>     12      0      0      0  3    0x100200  bored         crypto

>     11      0      0      0  3    0x100200  aiodoned      aiodoned

>     10      0      0      0  2    0x100200                update

>      9      0      0      0  3    0x100200  cleaner       cleaner

>      8      0      0      0  3    0x100200  reaper        reaper

> *    7      0      0      0  7    0x100200                pagedaemon

>      6      0      0      0  2    0x100600                pfpurge

>      5      0      0      0  3    0x100200  acpi_idle     acpi0

>      4      0      0      0  3    0x100200  bored         syswq

>      3      0      0      0  3    0x100200                idle0

>      2      0      0      0  3    0x100200  km_alloc1w    kmthread

>      1      0      1      0  3      0x4080  wait          init

>      0     -1      0      0  3     0x80200  scheduler     swapper

>

>

>

> Here is the last pf related info I have off of one of the machines

> before it crashed (less than 30 minutes later):

>

>

> Sun Jun  8 01:30:03 NZST 2008

>

> Status: Enabled for 5 days 05:53:11             Debug: Misc

>

> State Table                          Total             Rate

>   current entries                      513

>   searches                       112428343          248.1/s

>   inserts                           919608            2.0/s

>   removals                          919095            2.0/s

> Counters

>   match                             973538            2.1/s

>   bad-offset                             0            0.0/s

>   fragment                               0            0.0/s

>   short                                  0            0.0/s

>   normalize                              0            0.0/s

>   memory                                 0            0.0/s

>   bad-timestamp                          0            0.0/s

>   congestion                             0            0.0/s

>   ip-option                              0            0.0/s

>   proto-cksum                            0            0.0/s

>   state-mismatch                       108            0.0/s

>   state-insert                           0            0.0/s

>   state-limit                            0            0.0/s

>   src-limit                              0            0.0/s

>   synproxy                               0            0.0/s

>

> Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg

> Maxpg Idle

> pfiaddrpl    100        4    0        0     1     0     1     1     0

> 8    0

> pfrulepl     848       30    0       10     8     0     8     8     0

> 8    2

> pfstatepl    192  5122566    0  5122068    49     0    49    49     0

> 477   24

> pfstatekeypl  64  5585028    0  1378048 66778     0 66778 66778     0

> 8    0

> pfstateitempl 12  1378790    0  1378048     5     0     5     5     0

> 8    2

> pfpooladdrpl  68        4    0        0     1     0     1     1     0

> 8    0

> pfrktable   1240       13    0        6     4     0     4     4     0

> 334    0

> pfrkentry    156       10    0        4     1     0     1     1     0

> 7693    0

> pfrkentry2   156        4    0        0     1     0     1     1     0

> 8    0

> pfosfpen     108     1392    0      696    30    11    19    19     0

> 8    0

> pfosfp        28      814    0      407     3     0     3     3     0

> 8    0

>

> carp:

>         5854725 packets received (IPv4)

>         3607274 packets received (IPv6)

>                 9461993 discarded for unknown vhid

>         897315 packets sent (IPv4)

>         2 transitions to master

>

> pfsync:

>         9080346 packets received (IPv4)

>                 8406004 failed state lookup/inserts

>         10660708 packets sent (IPv4)

>

>

> OpenBSD 4.3-current (GENERIC) #884: Sat May 31 11:49:16 MDT 2008

>     deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC

> cpu0: Intel Celeron ("GenuineIntel" 686-class, 128KB L2 cache) 496 MHz

> cpu0:

> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR

> real mem  = 334065664 (318MB)

> avail mem = 314511360 (299MB)

> mainbus0 at root

> bios0 at mainbus0: AT/286+ BIOS, date 01/10/99, BIOS32 rev. 0 @ 0xeba00,

> SMBIOS rev. 2.3 @ 0xfa248 (43 entries)

> bios0: vendor Compaq version "686J1 v1.10" date 08/12/1999

> bios0: Compaq Deskpro SFF Series

> acpi0 at bios0: rev 0

> acpi0: tables DSDT FACP SSDT SSDT SSDT

> acpi0: wakeup devices COM1(S4) COM2(S4) USB0(S1) PCI0(S4) PBTN(S4)

> acpitimer0 at acpi0: 3579545 Hz, 24 bits

> acpiprt0 at acpi0: bus 0 (PCI0)

> acpicpu0 at acpi0: C2

> acpibtn0 at acpi0: PBTN

> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x1000

> 0xe0000/0x10000!

> cpu0 at mainbus0

> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)

> pchb0 at pci0 dev 0 function 0 "Intel 82810-DC100 Host" rev 0x02

> vga1 at pci0 dev 1 function 0 "Intel 82810-DC100 Video" rev 0x02

> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)

> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)

> agp0 at vga1: aperture at 0x44000000, size 0x4000000

> ppb0 at pci0 dev 30 function 0 "Intel 82801AA Hub-to-PCI" rev 0x01

> pci1 at ppb0 bus 1

> fxp0 at pci1 dev 2 function 0 "Intel 8255x" rev 0x08, i82559: irq 10,

> address 00:50:8b:79:7d:f4

> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4

> fxp1 at pci1 dev 9 function 0 "Intel 8255x" rev 0x08, i82559: irq 9,

> address 00:04:ac:23:9b:f8

> inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4

> ichpcib0 at pci0 dev 31 function 0 "Intel 82801AA LPC" rev 0x01

> pciide0 at pci0 dev 31 function 1 "Intel 82801AA IDE" rev 0x01: DMA,

> channel 0 wired to compatibility, channel 1 wired to compatibility

> wd0 at pciide0 channel 0 drive 0:

> wd0: 16-sector PIO, LBA, 6149MB, 12594960 sectors

> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2

> pciide0: channel 1 disabled (no drives)

> uhci0 at pci0 dev 31 function 2 "Intel 82801AA USB" rev 0x01: can't map

> i/o space

> auich0 at pci0 dev 31 function 5 "Intel 82801AA AC97" rev 0x01: can't

> map codec i/o space

> isa0 at ichpcib0

> isadma0 at isa0

> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo

> pckbc0 at isa0 port 0x60/5

> pckbd0 at pckbc0 (kbd slot)

> pckbc0: using irq 1 for kbd slot

> wskbd0 at pckbd0: console keyboard, using wsdisplay0

> pcppi0 at isa0 port 0x61

> midi0 at pcppi0:

> spkr0 at pcppi0

> lpt0 at isa0 port 0x378/4 irq 7

> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16

> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2

> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec

> biomask f96d netmask ff6d ttymask ffff

> mtrr: Pentium Pro MTRR support

> softraid0 at root

> root on wd0a swap on wd0b dump on wd0b

> 

--

Henning Brauer, hb@bsws.de, henning@openbsd.org

BS Web Services, http://bsws.de

Full-Service ISP - Secure Hosting, Mail and DNS Services

Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam