login
Search
All,
I'm getting quite a lot of these errors in /var/log/messages and can't
seem to find an appropriate fix in the archives:
May 14 21:05:54 svr02 /bsd: uvm_mapent_alloc: out of static map entries
May 14 21:57:47 svr02 /bsd: uvm_mapent_alloc: out of static map entries
May 14 23:00:05 svr02 /bsd: uvm_mapent_alloc: out of static map entries
May 15 07:27:53 svr02 /bsd: uvm_mapent_alloc: out of static map entries
May 15 07:39:59 svr02 /bsd: uvm_mapent_alloc: out of static map entries
N.B. This machine serves mirror content for various F/OSS projects in
addition to standard www content, so it quite often has >350 users
concurrently connected downloading mirrored content (in addition to
visitors who're actually visiting the site).
These messages correspond almost exactly with two things:
1.) the sites having quite a few visitors
2.) the sites becoming unavailable. In most cases, it fixes itself
when the freeloaders (errr downloaders ;-) complete their file
transfers.
Possibly worth noting:
1.) We've had to crank various settings in Apache to keep serving
traffic, as the stock settings were too low: we were reaching the max
daemons for Apache so new visitors were just out-of-luck.
2.) When the system begins to knuckle under load, I'm taking a
snapshot of various bits like the following.
Here's one example:
load averages: 0.45, 0.47, 0.40 07:40:00
247 processes: 245 idle, 2 on processor
CPU0 states: 7.2% user, 0.0% nice, 2.6% system, 2.2% interrupt, 88.0% idle
CPU1 states: 3.6% user, 0.0% nice, 0.3% system, 1.9% interrupt, 94.3% idle
Memory: Real: 339M/737M act/tot Free: 1272M Swap: 0K/518M used/tot
From the archives this seems to be something for which a fix *used* to
be cranking up the following:
maxusers 64
option BUFCACHEPERCENT=25
option MULTIPROCESSOR
option MAX_KMAPENT=4000
This hardly seems a real fix though--especially given everyone's
hatred of knobs, custom kernels, and such though ...What output to you get from 'netstat -m'? I might get yelled at for this as you mentioned people seem to hate custom kernels. But i've had good luck with the following options, I'm not sure which are still relevant, but they help. option NKMEMPAGES_MAX=81920 option NKMEMPAGES=81920 option MAX_KMAPENT=8192 I've always received that error you described on any high load openbsd box. Even with the above changes, you will eventually get the same error as your new limits are reached. If you come up with any better solutions, please let me know, i'd be very interested to hear them. -Darrian
2867 mbufs in use:
2566 mbufs allocated to data
274 mbufs allocated to packet headers
27 mbufs allocated to socket names and addresses
1129/5450/6144 mbuf clusters in use (current/peak/max)
13028 Kbytes allocated to network (22% in use)
0 requests for memory denied
0 requests for memory delayed
I see Allen beat me to the reply with the requested netstat data below, but in the mean time, I'm going to do the unthinkable and build a custom kernel with your mods and see where the chips fall. :-) Thanks for the suggestion. Kevin
Based on that netstat output, things look OK on your system. On some of my heavier loaded systems, I will see the peak mbuf use hit the max. Good luck, and as I said if you come up with something better, please let me know. -Darrian
sure, they help. at least if you want to believe they do. randomly pushing buttons you don't understand until it feels better is going to help how? btw, two of the three up there are completely unrelated to the problem at hand and useless these days. -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Are you using squid as well? You may try doing something like restarting apache. The problem seems related to certain long running processes with fragmented address spaces. Basically, in order to manage address spaces, the kernel keeps track of a bunch of maps. Entries in these maps are stored in... map entries. In certain situations, the kernel can't wait to allocate a map entry, so it grabs one from a static list. Previously, when they ran out, the kernel paniced. Now it just says uh oh. The kernel will merrily go on making more static entries as needed. I'd keep track of how often the message appears. At some point, it should stop. But it's not really alarming, unless it continues to print that continuously.
Funny you should ask. Yes and no. We are proxying some of the site's content, but it's with apache's mod_proxy. (No way around this from what we can see as it solves some business needs in terms of content delivery and is an easy fix to an otherwise vexing problem.) Restarting apache always solves the problem, but that's hardly a fix. Sure, I could crontab it to do so automatically and just periodically kick everyone off, but that's super yucky and still doesn't really *solve* the problem.... I'd feel good about that being the only answer I'll go out on a limb and assume this is the case since some of the files being downloaded are certainly ~100mb or more... some are entire DVD ISOs. I'd say these downloads qualify as "long running processes," It isn't alarming per se, but the sites on the server *definitely* stop accepting new visitors at some point. This seems to correlate directly to the uvm_mapent_alloc log events. If it were only the mirror visitors who were getting turned away that would be one thing, but it's actually interfering with regular traffic, too. :-( In short, I'm trying to find a way to: 1.) serve the oodles of mirror content (since we ourselves rely so heavily on F/OSS we want to make sure the mirrors are running both for ourselves and others) and 2.) also keep our normal site traffic humming along, too. I'm hoping to get to the point where "It Just Works", and it sure seems like the server itself has the horsepower to do it. If the CPUs were sweating hard or we were swapping heavily, it would make sense, but for it to be knuckling under what seems to me to be relatively light load seems like there's something else I can do to make it happy. Knobs, dials, levers, custom kernels, and custom apache builds they may be, but at this point I'm open to juuuuust about anything and everything including witch doctors, Chinese herbalists, and/or exorcists to get the problem solved. :-) Thanks much, Kevin
well, use a httpd that is better designed than apache. at least for the static content that should be kinda easy with a couple of redirects and a second IP. lighttpd is a good pick. -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
If talking about serving static content: mathopd is doing really good job here. -- Janusz Gumkowski http://www.am.torun.pl/~ja
Hi Kevin, Take a look at nginx. We found this server to be quite nice to use, and it has a small memory footprint, too. For caching + proxying stuff, you might want to take a look at Varnish, although I don't have first-hand experience with it yet (nginx alone can proxy just fine, only not cache anything). Kind regards, --Toni++
I believe cache is in the works, but of course this does no good right now. Hopefully soon. -- Darrin Chandler | Phoenix BSD User Group | MetaBUG dwchandler@stilyagin.com | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
Hi Darrin, I'm not sure whether I want another jack-of-all-trades kind of a web server, but would have different priorities for it, anyway (like eg. full IPv6 support - don't know it's in 0.6.x already). I mainly wanted to point the original poster to the - imho - top contender for lighttpd, which I personally don't like that much, despite using it for a number of years. Btw, thanks for providing the port. Kind regards, --Toni++
I wouldn't like nginx to include every possible thing either. I doubt that such a thing will happen. But I think adding cache makes some sense, since if you're using it as proxy then it's a central place, and it's also a very common configuration. Kind of like the old question of whether 'sort' should have the '-u' option or not. 'sort|uniq' is such a You're welcome :) -- Darrin Chandler | Phoenix BSD User Group | MetaBUG dwchandler@stilyagin.com | http://phxbug.org/ | http://metabug.org/ http://www.stilyagin.com/ | Daemons in the Desert | Global BUG Federation
the problem is not the message. problem is unneeded
kmem_map fragmentation that is caused byt misuse
by the bad code.
cu
--
paranoic mickey (my employers have changed but, the name has remained)
Hi Mickey, I think I do not understand: I'm not worried about the message, but about the crash I recently experienced on a 4.2 box, after seeing this message. IOW, I'd be happy already if 4.3 prevents the crash and only leaves a warning message. Making the code better to not experience the problem in the first place is, of course, a laudable goal, but unfortunately, I have no qualification to actually do anything about it. Kind regards, --Toni++
both explanation of the problem and a big diff that fixes it
had been already posted (:
cu
--
paranoic mickey (my employers have changed but, the name has remained)
Hi Mickey, thank you. I've seen that diff already (before sending the first), but don't quite understand whether the crash was already fixed in stock 4.3, with the diff also fixing the memory fragmentation, thus avoiding the warning message, or whether stock 4.3 is still susceptible to crashes, and the patch fixes both problems. To understand that, I'll have to give the diff a much closer look. Kind regards, --Toni++
the problem is not in the user land.
the problem is in i386 pmap which abuses kmem_map that is there
for malloc(9)s use and allocates pv_entries from it.
this leads to enormous kmem_map fragmentation and unaccounted
allocations that does not show up in the vmstat and as well leads
to livelocks (sleeping on kmem_map) and out of space in kmem_map
panics as well. there is a number of measures to remediate the
situation proper
- convert pv_entries allocations to pool (i have a diff if you wanna)
- backend malloc w/ pool (filed in sendbug)
- a number of uvm fixes (such as amap ops) that reduce fragmentation.
cu
--
paranoic mickey (my employers have changed but, the name has remained)
Can you please point me to where the diffs you refer to reside? I'd definitely like to try them out. Thank you, Darrian
most of these are filed in sendbug (some for months) already...
here is a cumulative diff also w/ a bonus himem high-quality
software (in caase you managed to squeeze more than 4g of memory
in your box ;).
cu
--
paranoic mickey (my employers have changed but, the name has remained)
Index: arch/i386/conf/GENERIC
===================================================================
RCS file: /cvs/src/sys/arch/i386/conf/GENERIC,v
retrieving revision 1.603
diff -u -r1.603 GENERIC
--- arch/i386/conf/GENERIC 25 Feb 2008 23:16:47 -0000 1.603
+++ arch/i386/conf/GENERIC 7 May 2008 12:55:43 -0000
@@ -37,6 +37,8 @@
config bsd swap generic
mainbus0 at root
+himem0 at root # himem.sys
+scsibus* at himem?
cpu0 at mainbus?
bios0 at mainbus0
Index: arch/i386/conf/files.i386
===================================================================
RCS file: /cvs/src/sys/arch/i386/conf/files.i386,v
retrieving revision 1.172
diff -u -r1.172 files.i386
--- arch/i386/conf/files.i386 4 Mar 2008 21:14:29 -0000 1.172
+++ arch/i386/conf/files.i386 7 May 2008 12:55:43 -0000
@@ -440,6 +440,10 @@
attach esm at mainbus
file arch/i386/i386/esm.c esm needs-flag
+device himem: scsi
+attach himem at root
+file arch/i386/i386/himem.c himem needs-flag
+
#
# VESA
#
Index: arch/i386/i386/autoconf.c
===================================================================
RCS file: /cvs/src/sys/arch/i386/i386/autoconf.c,v
retrieving revision 1.78
diff -u -r1.78 autoconf.c
--- arch/i386/i386/autoconf.c 27 Dec 2007 18:04:27 -0000 1.78
+++ arch/i386/i386/autoconf.c 7 May 2008 12:55:43 -0000
@@ -71,6 +71,7 @@
#include <dev/cons.h>
#include "ioapic.h"
+#include "himem.h"
#if NIOAPIC > 0
#include <machine/i82093var.h>
@@ -117,6 +118,10 @@
if (config_rootfound("mainbus", NULL) == NULL)
panic("cpu_configure: mainbus not configured");
+
+#if NHIMEM > 0
+ config_rootfound("himem", NULL);
+#endif
#if NIOAPIC > 0
if (nioapics > 0)
Index: ...Yes, please. Definitely... and thanks. FWIW I can bring a spare server online this weekend to keep in the wings in case something goes completely nutty with the diff, so no worries about this affecting production per se. :-)
