On Thu, 3 Jan 2008, Dag-Erling Sm=F8rgrav wrote:

quoted text
> Jason Evans <jasone@freebsd.org> writes:
>> [sbrk is broken]
>
> The real question is why we would revert perfectly good code (jemalloc) f=
rom=20
quoted text
> using a modern interface to using one that has been obsolete for twenty=
=20
quoted text
> years, and marked as such in the man page for seven years.
>
> If rwatson@ wants malloc() to respect resource limits, he can bloody well=
=20
quoted text
> fix mmap().  Until he does, the datasize limit is a joke anyway, as anyon=
e=20
quoted text
> can circumvent it by either using mmap() instead of malloc() or setting=
=20
quoted text
> _malloc_options before calling malloc().

The issue here was that there were a number of reports that out-of-control=
=20
applications were toasting systems that weren't getting toasted under 6.x. =
 I=20
experienced this on my web server, but the ports build cluster has been=20
running into it for months.  The symptom is that a single application exhau=
sts=20
swap, causing all sorts of things to break (tm), killing of other large=20
processes, etc.  To be clear, in the new world order, instead of getting NU=
LL=20
back from malloc(3), SIGKILL is delivered to large processes.

When I e-mailed Jason Evans and Alan Cox about it, I suggested that we=20
actually teach malloc(3) to enforce an allocation limit itself by querying =
a=20
limit once at process startup, and then using its own accounting to decide=
=20
when to start failing requests.  As an alternative model that would require=
=20
some more infrastructural changes, I suggested a new mmap() flag that hinte=
d=20
to the kernel that the page should count against a swap/anonymous memory=20
limit, but that we should avoid more serious changes at the last minute bef=
ore=20
a release.  Alan suggested the the model Jason ended up implementing as a=
=20
lower risk way to restore the 6.x resource limits non-disruptively.  As it=
=20
turned out, this proved much more complicated than expected.

The right answer is presumably to introduce a new LIMIT_SWAP, which limits =
the=20
allocation of anonymous memory by processes, and size it to something like =
90%=20
of swap space by default.  Since that won't be happening before 7.0, I beli=
eve=20
the consensus is to simply not MFC the changes for 7 and proceed with the=
=20
release.  However, having a resource limit on swap use in order to prevent =
the=20
above scenario is actually quite important: SIGKILL of arbitrary processes =
is=20
not a good way to deal with one run-away process, and the virtual memory si=
ze=20
limit, while also useful, prevents you from limiting the allocation of swap=
=20
without also limiting memory mapping.  So wouldn't help, for example, to li=
mit=20
swap used by a web cache that memory mapped cache files.

Robert N M Watson
Computer Laboratory
University of Cambridge