In a recent email, OpenBSD creator Theo de Raadt [interview] described a number of modifications to how OpenBSD allocates memory. In preparation for the upcoming 3.8 release, Theo asked for people to beta test -current as the recent modifications will likely cause instabilities in many applications. One of the modifications was to make the mmap system call return a random memory address, as well ensuring "that two objects are not mapped next to each other; in effect, this creates unallocated memory which we call a 'guard page'." Another was to update the malloc function to use mmap to obtain memory. Finally, the free function was updated to immediately return memory to the kernel and un-allocate it from the calling process. Additional changes were also made, but unlike these three the additional changes are not enabled by default as they are "too dangerous for normal software or cause too much of a slowdown".

Theo points out that these changes have a couple of significant impacts. He explains that for over a decade efforts have been made to find and fix buffer overflows, and more recently bugs have been found in which software is reading before the start of a buffer, or beyond the end of the buffer. With these recent memory allocation changes, such an attempt will cause the application to coredump with a SIGSEGV signal. Additionally, now that memory is unmapped as soon as it is freed, any attempt to access freed memory will also cause the application to coredump with a SIGSEGV signal. He explained, "we expect that our malloc will find more bugs in software, and this might hurt our user community in the short term. We know that what this new malloc is doing is perfectly legal, but that realistically some open source software is of such low quality that it is just not ready for these things to happen." Hence the request for beta testers to help track down these misbehaving applications. Theo concluded, "instead of saying that OpenBSD is busted in this regard, please realize that the software which is crashing is showing how shoddily it was written. Then help us fix it. For everyone.. not just OpenBSD users."

From: Theo de Raadt [email blocked]
To:  misc
Subject: 3.8 beta requests
Date: Mon, 22 Aug 2005 17:33:40 -0600

We are heading towards making the real 3.8 release soonish.  I would
like to ask the community to do lots of testing over the next week if
they can.

This release will bring a lot of new ideas from us.  One of them in
particular is somewhat risky.  I think it is time to talk about that
one, and let people know what is ahead on our road.

Traditionally, Unix malloc(3) has always just "extended the brk",
which means extending the traditional Unix process data segment to
allocate more memory.  malloc(3) would simply extend the data segment,
and then calve off little pieces to requesting callers as needed.  It
also remembered which pieces were which, so that free(3) could do it's
job.

The way this was always done in Unix has had a number of consequences,
some of which we wanted to get rid of.  In particular, malloc & free
have not been able to provide strong protection against overflows or
other corruption.

Our malloc implementation is a lot more resistant (than Linux) to
"heap overflows in the malloc arena", but we wanted to improve things
even more.

Starting a few months ago, the following changes were made:

- We made the mmap(2) system call return random memory addresses.  As well
  the kernel ensures that two objects are not mapped next to each other;
  in effect, this creates unallocated memory which we call a "guard page".

- We have changed malloc(3) to use mmap(2) instead of extending the data
  segment via brk()

- We also changed free(3) to return memory to the kernel, un-allocating
  them out of the process.

- As before, objects smaller than a page are allocated within shared
  pages that malloc(3) maintains.  But their allocation is now somewhat
  randomized as well.

- A number of other similar changes which are too dangerous for normal
  software or cause too much of a slowdown are available as malloc options
  as described in the manual page.  These are very powerful for debugging
  buggy applications.

Other results:

- When you free an object that is >= 1 page in size, it is actually
  returned to the system.  Attempting to read or write to it after
  you free is no longer acceptable.  That memory is unmapped.  You get
  a SIGSEGV.

- For a decade and a bit, we have been fixing software for buffer overflows.
  Now we are finding a lot of software that reads before the start of the
  buffer, or reads too far off the end of the buffer.  You get a SIGSEGV.

To some of you, this will sound like what the Electric Fence toolkit
used to be for.  But these features are enabled by default.  Electric
Fence was also very slow.  It took nearly 3 years to write these
OpenBSD changes since performance was a serious consideration.  (Early
versions caused a nearly 50% slowdown).

Our changes have tremendous benefits, but until some bugs in external
packages are found and fixed, there are some risks as well.  Some
software making incorrect assumptions will be running into these new
security technologies.

I discussed this in talks I have given before: I said that we were
afraid to go ahead with guard pages, because a lot of software is just
written to such low standards.  Applications over-read memory all the
time, go 1 byte too far, read 1 byte too early, access memory after free,
etc etc etc.

Oh well -- we've decided that we will try to ship with this protection
mechanism in any case, and try to solve the problems as we run into
them.

Two examples:

Over the last two months, some OpenBSD users noticed that the X server
was crashing occasionally.  Two bugs have been diagnosed and fixed by
us.  One was a use-after-free bug in the X shared library linker.  The
other was a buffer-over-read bug deep down in the very lowest level
fb* pixmap compositing routines.  The latter bug in particular was
very difficult to diagnose and fix, and is about 10 years old.  We
have found other bugs like this in other external software, and even a
few in the base OpenBSD tree (though those were found a while back,
even as we started experimenting with the new malloc code).

I would bet money that the X fb* bug has crashed Linux (and other) X
servers before.  It is just that it was very rare, and noone ever
chased it.  The new malloc we have just makes code get lucky less
often, which lets us get to the source of a bug easier.  As a
programmer, I appreciate anything which makes bugs easier to
reproduce.

We expect that our malloc will find more bugs in software, and this
might hurt our user community in the short term.  We know that what
this new malloc is doing is perfectly legal, but that realistically
some open source software is of such low quality that it is just not
ready for these things to happen.

We ask our users to help us uncover and fix more of these bugs in
applications.  Some will even be exploitable.  Instead of saying that
OpenBSD is busted in this regard, please realize that the software
which is crashing is showing how shoddily it was written.  Then help
us fix it.  For everyone.. not just OpenBSD users.

• Archive of above thread
• KernelTrap interview with Theo de Raadt