Jeff Ross wrote:
quoted text
> On Sat, 30 Jan 2010, Bob Beck wrote:
> 
>> Ooooh. nice one.  Obviously ami couldn't get memory mappings and 
>> freaked out.
>>
>> While not completely necessary, I'd love for you to file that whole
>> thing into sendbug() in a pr so we don't
>> forget it. but that one I need to pester krw, art, dlg, and maybe
>> marco about what ami is doing.
>>
>> note that the behaviour you see wrt free memory dropping but not
>> hitting swap is what I expect.
>> basically that makes the buffer cache subordinate to working set
>> memory between 10 and 90% of
>> physmem. the buffer cache will throw away pages before allowing the
>> system to swap.
>>
>> Drop it back to 70% and tell me if you still get the same panic
>> please.  and if you have a fixed test
>> case that reproduces this on your machine ( a load generator for
>> postgres with clients) I'd love to
>> have a copy in the pr as well.
> 
> 70% produces the same panic.
> 
> panic: pmap_enter: no pv entries available
> Stopped at      Debugger+0x4:   leave
> RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
> IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
> DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
> ddb{0}> trace
> Debugger(7149,dfe3ede8,d08edf18,c,0) at Debugger+0x4
> panic(d077d740,0,7000,d08ad6e0,ffffffff) at panic+0x55
> pmap_enter(d08f3520,e0031000,7149000,3,13) at pmap_enter+0x2e5
> _bus_dmamem_map(d0875c40,d505fc44,1,6344,d505fc58,1,dfe3eebc,1) at 
> _bus_dmamem_map+0x9c
> ami_allocmem(d49b5800,6344,20,d0753ddc) at ami_allocmem+0x92
> ami_mgmt(d49b5800,a1,4,0,0,6344,d49cd000,1) at ami_mgmt+0x268
> ami_refresh_sensors(d49b5800,da987028,da987050,80000000,da987028) at 
> ami_refresh_sensors+0x25
> sensor_task_work(d49b3d80,0,50,200286) at sensor_task_work+0x1f
> workq_thread(d0863100) at workq_thread+0x32
> Bad frame pointer: 0xd0a32e78
> ddb{0}>
> 
> I'll skip the ps for this go round since it should be pretty much the 
> same and go directly to sendbug, including the pg_bench script I use to 
> trigger it.
> 
> Thanks!
> 
> Jeff

Okay, I've been testing.  I brought everything up to current, applied the
ami.c patch sent by David Gwynne as modified by Phillip Guenther, and the 
patch to bus_dma.c sent by Kenneth Westerback.

I started by setting kern.bufcachepercent=60 and then moving down by 10 after
each panic. Anything 20 or greater triggers the same panic as above.

I then set it to 10 to see what would happen.  The load ran okay, but I did
get three uvm_mapent_alloc: out of static map entries entries into the console
that seems to coincide with the end of one of the three pgbench runs and the
start of the next.

So I set it to 11 and got this:

ddb{2}> show panic
malloc: out of space in kmem_map
ddb{2}> trace
Debugger(3fff,c,d488a000,4,4000) at Debugger+0x4
panic(d0752c20,0,4000,0,0) at panic+0x55
malloc(4000,7f,0,da8980b4,70000) at malloc+0x76
vfs_getcwd_scandir(e002eee8,e002eecc,e002eed0,d4e80400,da24f184) at vfs_getcwd_
scandir+0x123
vfs_getcwd_common(da898154,0,e002ef10,d4e80400,200,1,da24f184,22) at vfs_getcwd
_common+0x1f0
sys___getcwd(da24f184,e002ef68,e002ef58,da24f184) at sys___getcwd+0x62
syscall() at syscall+0x12b
--- syscall (number 304) ---
0x1c028f25:

I reported a similar panic back in December
	http://kerneltrap.org/mailarchive/openbsd-misc/2009/12/14/6309363
and was told I'd twisted the knobs too hard ;-)

Here are the sysctl values I'm currently using:

kern.maxproc=10240
kern.maxfiles=20480
kern.maxvnodes=6000

kern.shminfo.shmseg=32
kern.seminfo.semmni=256
kern.seminfo.semmns=2048
kern.shminfo.shmall=512000
kern.shminfo.shmmax=768000000

About that time Owain Ainsworth sent his version of a fix to bus_dma.c so I 
applied that and built a new kernel and I still get panics when I adjust 
kern.bufcachepercent above 15 or so.

Here's the latest panic, trace and ps with kern.bufcachepercent set to 20:
ddb{0}> show panic
pmap_enter: no pv entries available
ddb{0}> trace
Debugger(b6f30,dff02e1c,2000,c,0) at Debugger+0x4
panic(d077d7a0,0,0,d0e7a980,1) at panic+0x55
pmap_enter(d08f3520,e002d000,b6f30000,7,13) at pmap_enter+0x2e5
uvm_km_alloc1(d08ad720,2000,0,1) at uvm_km_alloc1+0xd5
fork1(da8c5834,14,1,0,0) at fork1+0x100
sys_fork(da8c5834,dff02f68,dff02f58,da8c5834) at sys_fork+0x38
syscall() at syscall+0x12b
--- syscall (number 2) ---
ddb{0}> ps
    PID   PPID   PGRP    UID  S       FLAGS  WAIT          COMMAND
  18032   3768  18032    503  3   0x2000008  flt_pmfail2   postgres
  28815   3768  28815    503  3   0x2000008  inode         postgres
  16262   3768  16262    503  2   0x2000008                postgres
  15301   3768  15301    503  3   0x2000008  inode         postgres
  16712   3768  16712    503  2   0x2000008                postgres
   5959   3768   5959    503  3   0x2000008  flt_pmfail2   postgres
  24166   3768  24166    503  3   0x2000008  flt_pmfail2   postgres
  23692   3768  23692    503  3   0x2000008  flt_pmfail2   postgres
  21841   3768  21841    503  3   0x2000008  flt_pmfail2   postgres
  23838   3768  23838    503  3   0x2000008  flt_pmfail2   postgres
  25423   3768  25423    503  3   0x2000008  inode         postgres
   5075   3768   5075    503  3   0x2000008  inode         postgres
  23008   3768  23008    503  3   0x2000008  inode         postgres
  10527   3768  10527    503  3   0x2000008  flt_pmfail2   postgres
  17391   3768  17391    503  3   0x2000008  flt_pmfail2   postgres
  27363   3768  27363    503  3   0x2000008  flt_pmfail2   postgres
   4858   3768   4858    503  2   0x2000008                postgres
  18716   3768  18716    503  2   0x2000008                postgres
   8073   3768   8073    503  2   0x2000008                postgres
  30893   3768  30893    503  3   0x2000008  flt_pmfail2   postgres
  13741   3768  13741    503  3   0x2000008  flt_pmfail2   postgres
  14272   3768  14272    503  2   0x2000008                postgres
   1962   3768   1962    503  3   0x2000008  inode         postgres
   4988   3768   4988    503  3   0x2000008  flt_pmfail2   postgres
   8452   3768   8452    503  3   0x2000008  inode         postgres
  15633   3768  15633    503  3   0x2000008  flt_pmfail2   postgres
  17648   3768  17648    503  3   0x2000008  flt_pmfail2   postgres
  27751   3768  27751    503  3   0x2000008  inode         postgres
  12932   3768  12932    503  7   0x2000008                postgres
  21367   3768  21367    503  2   0x2000008                postgres
  13583   3768  13583    503  3   0x2000008  flt_pmfail2   postgres
  11813   3768  11813    503  3   0x2000008  inode         postgres
   6094   3768   6094    503  3   0x2000008  inode         postgres
  13686   3768  13686    503  3   0x2000008  flt_pmfail2   postgres
  10011   3768  10011    503  3   0x2000008  flt_pmfail2   postgres
  30652   3768  30652    503  7   0x2000008                postgres
   5358   3768   5358    503  3   0x2000008  flt_pmfail2   postgres
  24385   3768  24385    503  3   0x2000008  inode         postgres
  29773   3768  29773    503  3   0x2000008  flt_pmfail2   postgres
  22599   3768  22599    503  3   0x2000008  flt_pmfail2   postgres
  14231   3768  14231    503  3   0x2000008  flt_pmfail2   postgres
  14083   3768  14083    503  3   0x2000008  flt_pmfail2   postgres
   9729   3768   9729    503  2   0x2000008                postgres
  11748   3768  11748    503  3   0x2000008  inode         postgres
   9097   3768   9097    503  3   0x2000008  flt_pmfail2   postgres
  16057   3768  16057    503  3   0x2000008  inode         postgres
   1067   3768   1067    503  3   0x2000008  flt_pmfail2   postgres
  27445   3768  27445    503  3   0x2000008  flt_pmfail2   postgres
  24037   3768  24037    503  3   0x2000008  flt_pmfail2   postgres
   7000   3768   7000    503  2   0x2000008                postgres
  29367   3768  29367    503  3   0x2000008  inode         postgres
  25020   3768  25020    503  3   0x2000008  flt_pmfail2   postgres
  20564   3768  20564    503  3   0x2000008  flt_pmfail2   postgres
   4090   3768   4090    503  3   0x2000008  flt_pmfail2   postgres
   8865   3768   8865    503  3   0x2000008  flt_pmfail2   postgres
  29631   3768  29631    503  3   0x2000008  inode         postgres
  16278   3768  16278    503  7   0x2000008                postgres
  20854   3768  20854    503  2   0x2000008                postgres
  31943   3768  31943    503  3   0x2000008  flt_pmfail2   postgres
  18137   3768  18137    503  2   0x2000008                postgres
  10523   4345  10523      0  3   0x2044182  poll          systat
   4345      1   4345      0  3   0x2004082  pause         ksh
   2520      1   2520      0  3   0x2004082  ttyin         getty
  32389      1  32389      0  3   0x2004082  ttyin         getty
  21553      1  21553      0  3   0x2000080  select        cron
  22285      1  22285      0  3   0x2000080  select        sshd
  19843      1  19843      0  3   0x2000180  select        inetd
  24948  27412  17184     83  3   0x2000180  poll          ntpd
  27412  17184  17184     83  3   0x2000180  poll          ntpd
  17184      1  17184      0  3   0x2000080  poll          ntpd
   7640  25372  25372     74  3   0x2000180  bpf           pflogd
  25372      1  25372      0  3   0x2000080  netio         pflogd
  25643  12522  12522     73  3   0x2000180  poll          syslogd
  12522      1  12522      0  3   0x2000088  netio         syslogd
  31132   3768  31132    503  3   0x2000088  poll          postgres
   6731   3768   6731    503  3   0x2000088  select        postgres
   3796   3768   3796    503  2         0x8                postgres
  10178   3768  10178    503  3   0x2000088  select        postgres
  23616   3768  23616    503  3   0x2000088  select        postgres
   1109  22145  22399   7794  3   0x2004082  piperd        qmail-clean
   4633  22145  22399   7795  3   0x2004082  select        qmail-rspawn
  26222  22145  22399      0  3   0x2004082  select        qmail-lspawn
* 3768  27937  22399    503  7   0x200400a                postgres
   3493  26189  22399     73  3   0x2004082  piperd        multilog
  19747  10749  22399   7792  3   0x2004082  piperd        multilog
  31915  18366  22399   1002  3   0x2004082  piperd        multilog
    611  19270  22399     73  3   0x2004182  netio         socklog
   6710      1   6710      0  3   0x2004082  ttyin         getty
  30305      1  30305      0  3   0x2004082  ttyin         getty
   1774      1   1774      0  3   0x2004082  ttyin         getty
  25968  31940  22399   1001  3   0x2004182  poll          dnscache
  30323   3543  22399     73  3   0x2004082  piperd        multilog
  22145  15017  22399   7796  3   0x2004082  select        qmail-send
  26189  30365  22399      0  3   0x2004082  poll          supervise
  27937  30365  22399      0  3   0x2004082  poll          supervise
   3543  30365  22399      0  3   0x2004082  poll          supervise
  19270  30365  22399      0  3   0x2004082  poll          supervise
  10749  30365  22399      0  3   0x2004082  poll          supervise
  15017  30365  22399      0  3   0x2004082  poll          supervise
  18366  30365  22399      0  3   0x2004082  poll          supervise
  31940  30365  22399      0  3   0x2004082  poll          supervise
  30671  22399  22399      0  3   0x2004082  piperd        readproctitle
  30365  22399  22399      0  3   0x2004082  nanosleep     svscan
  22399      1  22399      0  3   0x2004082  pause         sh
     19      0      0      0  3   0x2100200  bored         crypto
     18      0      0      0  3   0x2100200  aiodoned      aiodoned
     17      0      0      0  3   0x2100200  syncer        update
     16      0      0      0  3   0x2100200  cleaner       cleaner
     15      0      0      0  3    0x100200  reaper        reaper
     14      0      0      0  2   0x2100200                pagedaemon
     13      0      0      0  3   0x2100200  pftm          pfpurge
     12      0      0      0  3   0x2100200  usbevt        usb2
     11      0      0      0  3   0x2100200  usbevt        usb1
     10      0      0      0  3   0x2100200  usbtsk        usbtask
      9      0      0      0  3   0x2100200  usbevt        usb0
      8      0      0      0  3   0x2100200  acpi_idle     acpi0
      7      0      0      0  3  0x40100200                idle3
      6      0      0      0  3  0x40100200                idle2
      5      0      0      0  3  0x40100200                idle1
      4      0      0      0  3   0x2100200  bored         syswq
      3      0      0      0  3  0x40100200                idle0
      2      0      0      0  3   0x2100200  kmalloc       kmthread
      1      0      1      0  3   0x2004080  wait          init
      0     -1      0      0  3   0x2080200  scheduler     swapper

  Anybody got an idea what I can test from here?

Thanks,

Jeff