On Thu, May 15, 2008 at 7:07 PM, Ted Unangst <ted.unangst@gmail.com> wrote:
quoted text
> On 5/15/08, Kevin <spytag@gmail.com> wrote:
>> All,
>>
>>  I'm getting quite a lot of these errors in /var/log/messages and can't
>>  seem to find an appropriate fix in the archives:
>>
>>      May 14 21:05:54 svr02 /bsd: uvm_mapent_alloc: out of static map entries
>>      May 14 21:57:47 svr02 /bsd: uvm_mapent_alloc: out of static map entries
>>      May 14 23:00:05 svr02 /bsd: uvm_mapent_alloc: out of static map entries
>>      May 15 07:27:53 svr02 /bsd: uvm_mapent_alloc: out of static map entries
>>      May 15 07:39:59 svr02 /bsd: uvm_mapent_alloc: out of static map entries
>>
>>  N.B. This machine serves mirror content for various F/OSS projects in
>>  addition to standard www content, so it quite often has >350 users
>>  concurrently connected downloading mirrored content (in addition to
>>  visitors who're actually visiting the site).
>
> Are you using squid as well?  You may try doing something like
> restarting apache.
Funny you should ask. Yes and no. We are proxying some of the site's
content, but it's with apache's mod_proxy.

(No way around this from what we can see as it solves some business
needs in terms of content delivery and is an easy fix to an otherwise
vexing problem.)

Restarting apache always solves the problem, but that's hardly a fix.
Sure, I could crontab it to do so automatically and just periodically
kick everyone off, but that's super yucky and still doesn't really
*solve* the problem.... I'd feel good about that being the only answer
if this were 1998, and we were running IIS on Windows NT 4.0. :-)

quoted text
> The problem seems related to certain long running processes with
> fragmented address spaces.
I'll go out on a limb and assume this is the case since some of the
files being downloaded are certainly ~100mb or more... some are entire
DVD ISOs. I'd say these downloads qualify as "long running processes,"
no?

quoted text
> Basically, in order to manage address spaces, the kernel keeps track
> of a bunch of maps.  Entries in these maps are stored in... map
> entries.  In certain situations, the kernel can't wait to allocate a
> map entry, so it grabs one from a static list.  Previously, when they
> ran out, the kernel paniced.  Now it just says uh oh.  The kernel will
> merrily go on making more static entries as needed.
>
> I'd keep track of how often the message appears.  At some point, it
> should stop.  But it's not really alarming, unless it continues to
> print that continuously.

It isn't alarming per se, but the sites on the server *definitely*
stop accepting new visitors at some point. This seems to correlate
directly to the uvm_mapent_alloc log events.

If it were only the mirror visitors who were getting turned away that
would be one thing, but it's actually interfering with regular
traffic, too. :-(

In short, I'm trying to find a way to:
1.) serve the oodles of mirror content (since we ourselves rely so
heavily on F/OSS we want to make sure the mirrors are running both for
ourselves and others) and
2.) also keep our normal site traffic humming along, too.

I'm hoping to get to the point where "It Just Works", and it sure
seems like the server itself has the horsepower to do it.

If the CPUs were sweating hard or we were swapping heavily, it would
make sense, but for it to be knuckling under what seems to me to be
relatively light load seems like there's something else I can do to
make it happy.

Knobs, dials, levers, custom kernels, and custom apache builds they
may be, but at this point I'm open to juuuuust about anything and
everything including witch doctors, Chinese herbalists, and/or
exorcists to get the problem solved. :-)


Thanks much,
Kevin