In an interesting conversation spun off an earlier discussion about how and where to report Linux kernel security issues [story], Linux kernel guru Ingo Molnar [interview] offered some insights into the differences between PaX and ExecShield [story].

Linus Torvalds began the conversation noting, "quite frankly, nobody should ever depend on the kernel having zero holes. We do our best, but if you want real security, you should have other shields in place. exec-shield is one." In a followup email he explained, "what is the real issue with MS security? Is it that NT is fundamentally a weak kernel? Hey, maybe. Or maybe not. More likely it's the mindset that you trust everything, regardless of where they are." And finally he added, "containment is what real security is about. Everybody knows bugs happen, and that people do stupid things. Developers, users, whatever. We all do."

The conversation then led into some insightful comments from ExecShield authors Arjan van de Ven and Ingo Molnar, further explaining how Exec Shield works, and some of its design differences compared to PaX. Looking at the disadvantages of both efforts, Ingo highlighted three disadvantages of PaX: it halves the per-process VM space from 3GB to 1.5GB, its mirrored vma code is quite complex and thus potentially difficult to maintain, and it requires manual tagging of applications. In contrast, Ingo highlighted the main disadvantage of ExecShield being that certain VM allocation patterns can reduce its effectiveness, going on to point out that such instances are quite rare and have mostly been addressed. Ingo added that one of the goals with ExecShield is to "'make it as seemless and automatic as possible, so that people actually end up using our stuff.'"

From: Linus Torvalds [email blocked]
To: Dave Jones [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Wed, 12 Jan 2005 20:48:57 -0800 (PST)



On Wed, 12 Jan 2005, Dave Jones wrote:
> 
> For us thankfully, exec-shield has trapped quite a few remotely
> exploitable holes, preventing the above.

One thing worth considering, but may be abit _too_ draconian, is a
capability that says "can execute ELF binaries that you can write to".

Without that capability set, you can only execute binaries that you cannot
write to, and that you cannot _get_ write permission to (ie you can't be
the owner of them either - possibly only binaries where the owner is
root).

Sure, that's clearly not viable for a developer or even somebody who
maintains his own machine, but it _is_ probably viable for pretty much any
user that is afraid of compiling stuff him/herself and just gets signed
rpm's that install as root anyway. And it should certainly be viable for
somebody like "nobody" or "ftp" or "apache".

And I suspect there is almost zero overlap between the "developer
workstation" kind of setup (where the above is just not workable) and
"server or end-user desktop" setup where it might work.

A lot of the local root exploits depend on being able to run code that
doesn't come pre-installed on the system. A hole in a user-level server
may get you local shell access, but you generally need another stage to
get elevated privileges and _really_ mess with the machine.

Quite frankly, nobody should ever depend on the kernel having zero holes.  
We do our best, but if you want real security, you should have other
shields in place. exec-shield is one. So is using a compiler that puts
guard values on the stack frame (immunix, I think). But so is saying "you
can't just compile or download your own binaries, nyaah, nyaah, nyaah".

As I've already made clear, I don't believe one whit in the "secrecy"  
approach to security. I believe that "security through obscurity" can
actually be one valid level of security (after all, in the extreme case,
that's all a password ever really is).

So I believe that in the case of hiding vulnerabilities, any "security
gain" from the obscurity is more than made up for by all the security you
lose though delaying action and not giving people information about the
problem. 

I realize people disagree with me, which is also why I don't in any way
take vendor-sec as a personal affront or anything like that: I just think
it's a mistake, and am very happy to be vocal about it, but hey, the
fundamental strength of open source is exactly the fact that people don't
have to agree about everything.

			Linus



From: Christoph Hellwig [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 13 Jan 2005 08:23:20 +0000

On Wed, Jan 12, 2005 at 08:48:57PM -0800, Linus Torvalds wrote:
> Without that capability set, you can only execute binaries that you cannot
> write to, and that you cannot _get_ write permission to (ie you can't be
> the owner of them either - possibly only binaries where the owner is
> root).

I think this is called "mount user-writeable filesystems with -noexec" ;-)



From: Linus Torvalds [email blocked]
To: Christoph Hellwig [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 13 Jan 2005 08:38:03 -0800 (PST)



On Thu, 13 Jan 2005, Christoph Hellwig wrote:
>2B
> On Wed, Jan 12, 2005 at 08:48:57PM -0800, Linus Torvalds wrote:
> > Without that capability set, you can only execute binaries that you cannot
> > write to, and that you cannot _get_ write permission to (ie you can't be
> > the owner of them either - possibly only binaries where the owner is
> > root).
> 
> I think this is called "mount user-writeable filesystems with -noexec" ;-)

You miss the point.

It wouldn't be a global flag. It's a per-process flag. For example, many 
people _do_ need to execute binaries in their home directory. I do it all 
the time. I know what a compiler is.

Others do not necessarily do that. Sure, you could mount each users home 
directory separately with a bind mount, but that's not only inconvenient, 
it also misses the point - it's not about _where_ the binary is, it's 
about _who_ runs it.

What is the real issue with MS security? Is it that NT is findamentally a 
weak kernel? Hey, maybe. Or maybe not. More likely it's the mindset that 
you trust everything, regardless of where they are. Most users are admins, 
and you run any code you see (or don't see) by default, whether it's in an 
email attachement or whatever.

Containment is what real security is about. Everybody knows bugs happen, 
and that people do stupid things. Developers, users, whatever. We all do.

For example, in many environments it could possibly be a good idea to make
even _root_ have the "can run non-root binaries flag" clear by default. 
Imagine a system that booted up that way, and used PAM to enable non-root 
binaries on a per-user basis (for developers who need it or otherwise 
people who are trusted to have their own binaries). Think about what that 
means...

Every single deamon in the system would have the flag clear by default.  
You take over the web-server, and the most you have to play with are the
binaries that are already installed on the system (and the code you can
inject directly into the web server process from outside - that's likely
to be the _real_ security hazard).

It's just another easy containment. It's not real security in itself, but
_no_ single thing is "real security". You just add containment, to the
point where it gets increasingly difficult to get to some state where you
can do lots of damage (in a perfect world, exponentially more so, but
these containments are seldom independent or each other).

NOTE! I'd personally hate some of the security things. For example, I
think the "randomize code addresses" is absolutely horrible, just because
of the startup overhead it implies (specifically no pre-linking). I also
immensely dislike exec-shield because of the segment games it plays - I
think it makes sense in the short run but not in the long run, so I much
prefer that one as a "vendor feature", not as a "core feature". 

So when I talk about security, I have this double-standard where I end up 
convinced that many features are things that _I_ should not do, but 
others likely should ;)

		Linus



From: Arjan van de Ven [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 13 Jan 2005 18:01:02 +0100

On Thu, 2005-01-13 at 08:38 -0800, Linus Torvalds wrote:
> 
> NOTE! I'd personally hate some of the security things. For example, I
> think the "randomize code addresses" is absolutely horrible, just
> because
> of the startup overhead it implies (specifically no pre-linking). I
> also
> immensely dislike exec-shield because of the segment games it plays -
> I
> think it makes sense in the short run but not in the long run, so I
> much
> prefer that one as a "vendor feature", not as a "core feature". 

I think you are somewhat misguided on these: the randomisation done in
FC does NOT prohibit prelink for working, with the exception of special
PIE binaries. Does this destroy the randomisation? No: prelink *itself*
randomizes the addresses when creating it's prelink database (which is
in fedora once every two weeks with a daily incremental run inbetween;
the bi-weekly run is needed anyway to properly deal with new and updated
software, the daily runs are stopgapping only). This makes all *systems*
different, even though runs of the same app on the same machine right
after eachother are the same for the library addresses only.
That does not destroy the value of randomisation; it limits it slightly,
since this ONLY matters for libraries, not for the stack or heap and the
other things that get randomized. 

As for the segment limits (you call them execshield, but execshield is
actually a whole bunch of stuff that happens to include segment limits;
a bit like tree and forrest ;) yes they probably should remain a vendor
feature, no argument about that.



From: Linus Torvalds [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 13 Jan 2005 09:19:16 -0800 (PST)



On Thu, 13 Jan 2005, Arjan van de Ven wrote:
> 
> I think you are somewhat misguided on these: the randomisation done in
> FC does NOT prohibit prelink for working, with the exception of special
> PIE binaries. Does this destroy the randomisation? No: prelink *itself*
> randomizes the addresses when creating it's prelink database

There was a kernel-based randomization patch floating around at some 
point, though. I think it's part of PaX. That's the one I hated. 

Although I haven't seen it in a long time, so you may well be right that 
that one too is fine. 

My point was really more about the generic issue of me being two-faced: 
I'll encourage people to do things that I don't actually like myself in 
the standard kernel. 

I just think that forking at some levels is _good_. I like the fact that 
different vendors have different objectives, and that there are things 
like Immunix and PaX etc around. Of course, the problem that sometimes 
results in is the very fact that because I encourage others to have 
special patches, they en dup not even trying to feed back _parts_ of them.

In this case I really believe that was the case. There are fixes in PaX
that make sense for the standard kernel. But because not _all_ of PaX
makes sense for the standard kernel, and because I will _not_ take their
patch whole-sale, they apparently believe (incorrectly) that I wouldn't
even take the non-intrusive fixes, and haven't really even tried to feed
them back.

(Yes, Brad Spengler has talked to me about PaX, but never sent me 
individual patches, for example. People seem to expect me to take all or 
nothing - and there's a _lot_ of pretty extreme people out there that 
expect everybody else to be as extreme as they are..)

		Linus



From: Ingo Molnar [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Wed, 19 Jan 2005 11:30:20 +0100


* John Richard Moser [email blocked] wrote:

> > There was a kernel-based randomization patch floating around at some 
> > point, though. I think it's part of PaX. That's the one I hated. 
> 
> PaX and Exec Shield both have them; personally I believe PaX is a more
> mature technology, since it's 1) still actively developed, and 2) been
> around since late 2000.  The rest of the community dissagrees with me
> of course, [...]

might this disagreement be based on the fact that exec-shield _is_ being
actively developed and is in active use in Fedora/RHEL, and that split
out portions of exec-shield (e.g. flexmmap, PT_GNU_STACK, NX) are
already in the upstream kernel?

(but no doubt PaX is fine and protects against exploits at least as
effectively as (and in some cases more effectively than) exec-shield, so
you've definitely not made a bad choice.)

	Ingo



From: John Richard Moser [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Wed, 19 Jan 2005 12:20:39 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Ingo Molnar wrote:
> * John Richard Moser [email blocked] wrote:
> 
> 
>>>There was a kernel-based randomization patch floating around at some 
>>>point, though. I think it's part of PaX. That's the one I hated. 
>>
>>PaX and Exec Shield both have them; personally I believe PaX is a more
>>mature technology, since it's 1) still actively developed, and 2) been
>>around since late 2000.  The rest of the community dissagrees with me
>>of course, [...]
> 
> 
> might this disagreement be based on the fact that exec-shield _is_ being
> actively developed and is in active use in Fedora/RHEL, and that split
> out portions of exec-shield (e.g. flexmmap, PT_GNU_STACK, NX) are
> already in the upstream kernel?
> 

ES has been actively developed since it was poorly implemented in 2003.
 PaX has been actively developed since it was poorly implemented in
2000.  PaX has had about 4 times longer to go from a poor
proof-of-concept NX emulation patch based on the plex86 announcement to
a full featured security system, and is written by a competant security
developer rather than a competant scheduler developer.

Split-out portions of PaX (and of ES) don't make sense.  ASLR can be
evaded pretty easily:  inject code, read %efp, find the GOT, read
addresses.  The NX protections can be evaded by using ret2libc.  on x86,
you need emulation to make an NX bit or the NX protections are useless.
 So every part prevents every other part from being pushed gently aside.

PT_GNU_STACK annoys me :P  I'm more interested in 1) PaX' full set of
markings (-ps for NX, -m for mprotect(), r for randmmap, x for
randexec), 2) getting rid of the need for anything but -m, and 3)
eliminating relocations.  Sometimes they don't patch GLIBC here and
Firefox won't load flash or Java because they're PT_GNU_STACK and don't
really need it (the java executables are marked, but the java plug-in
doesn't need PT_GNU_STACK).

I guess it works on Exec Shield, but it frightens me that I have to
audit every library an executable uses for a PT_GNU_STACK marking to see
if it has an executable stack.

> (but no doubt PaX is fine and protects against exploits at least as
> effectively as (and in some cases more effectively than) exec-shield, so
> you've definitely not made a bad choice.)
> 

Either or if it stops an exploit; there's no "stopping an exploit
better," just stopping more of them and having fewer loopholes.  As I
understand, ES' NX approximation fails if you relieve protections on a
higher mapping-- which confuses me, isn't vsyscall() a high-address
executable mapping, which would disable NX protection for the full
address space?

PaX disables vsyscall when using PAGEEXEC on x86 because (since 2.6.6 or
so) pipacs uses the same method as ExecShield as a best-effort, falling
back to kernel-assisted MMU walking if that fails.  Wasted effort with
vsyscall.

PaX though gives me powerful, flexible administrative control over
executable space protections as a privileged resource.
mprotect(PROT_EXEC|PROT_WRITE) isn't something normal programs need; so
it's not something I allow everyone to do.

Aside from that, I just trust the PaX developer more.  He's already got
a more developed product; he's a security developer instead of a
scheduler developer; and he reads CPU manuals for breakfast.  I think a
lot of PaX is developed without real hardware-- I know he at least
doesn't have an AMD64 (which is what I use PaX on-- and yes I use the
regression tests), and he does a fine job anyway.  This indicates to me
that this is a serious project with someone who knows what he's doing,
so I trust it more.

> 	Ingo
> 

- --
All content of all messages exchanged herein are left in the
Public Domain, unless otherwise explicitly stated.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFB7pbmhDd4aOud5P8RAuV2AJ44dE9gvqZ9xwfENaWA6Hm81ALcfQCaA7mk
QFZejeyBBLd1sdtSj3o4Avk=
=hNuJ
-----END PGP SIGNATURE-----




From: Ingo Molnar [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Wed, 19 Jan 2005 18:47:09 +0100


* John Richard Moser [email blocked] wrote:

> Split-out portions of PaX (and of ES) don't make sense. [...]

which shows that you dont know the exec-shield patch at all, nor those
split-out portions. At which point it becomes pretty pointless to 
discuss any technical details, dont you think?

> PT_GNU_STACK annoys me :P  I'm more interested in 1) PaX' full set of
> markings (-ps for NX, -m for mprotect(), r for randmmap, x for
> randexec), [...]
>
> I guess it works on Exec Shield, but it frightens me that I have to
> audit every library an executable uses for a PT_GNU_STACK marking to
> see if it has an executable stack.

here there are two misconceptions:

1) you claim that the manual setting of markings is better than the
_automatic_ setting of markings in Fedora. Manual setting is a support
and maintainance nightmare, there can be false positives and false
negatives as well. Also, manual setting of markings assumes code review
or 'does this application break' type of feedback - neither is as
reliable as automatic detection done by the compiler.

2) you claim that you have to audit everything. You dont have to. It's
all automatic. _Fedora developers_ (not you) then check the markings and
reduce the number of executable stacks as much as possible.

> [...] ES' NX approximation fails if you relieve protections on a
> higher mapping-- which confuses me, isn't vsyscall() a high-address
> executable mapping, which would disable NX protection for the full
> address space?

another misconception. Read the patch and you'll see how it's solved. 

> Aside from that, I just trust the PaX developer more.  He's already
> got a more developed product; he's a security developer instead of a
> scheduler developer; and he reads CPU manuals for breakfast. 

this is your choice, and i respect it. Please show the same amount of
respect for the choice of others as well, without badmouthing anything
just because their choice is different from yours.

	Ingo



From: Ingo Molnar [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 20 Jan 2005 09:35:59 +0100


* John Richard Moser [email blocked] wrote:

> On a final note, isn't PaX the only technology trying to apply NX
> protections to kernel space? [...]

NX protection for kernel-space overflows on x86 has been part of the
mainline kernel as of June 2004 (released in 2.6.8), on CPUs that
support the NX bit - i.e. latest AMD and Intel CPUs. Let me quote from
the commit log:

http://linux.bkbits.net:8080/linux-2.5/cset@1.1757.49.12

  [...]
  furthermore, the patch also implements 'NX protection' for kernelspace
  code: only the kernel code and modules are executable - so even
  kernel-space overflows are harder (in some cases, impossible) to
  exploit. Here is how kernel code that tries to execute off the stack is
  stopped:

   kernel tried to access NX-protected page - exploit attempt? (uid: 500)
   Unable to handle kernel paging request at virtual address f78d0f40
    printing eip:
   ...

implemented, split out and brought to you by yours truly, as part
of the exec-shield project. (You know, the one not developed by that 
'scheduler developer' ;-)

	Ingo



From: Ingo Molnar [email blocked]
Subject: Re: thoughts on kernel security issues
Date: 	Thu, 20 Jan 2005 11:44:51 +0100


* John Richard Moser [email blocked] wrote:

> I respect you as a kernel developer as long as you're doing preemption
> and schedulers; [...]

actually, 'preemption and schedulers' ignores 80% of my contributions to
Linux so i'm not sure what to make of your comment :-| Here's a list of
bigger stuff i worked on in the past 3-4 years:

  http://redhat.com/~mingo/

as you can readily notice from the directory names alone, 'preemption
and schedulers' is pretty much in the minority :-|

and that list i think sums up the main difference in mindset: to me,
exec-shield is 'just' another kernel feature (amongst dozens), which
solves a problem. I'm not attached to the concept/patch emotionally, i
only want to see a solution for a problem in a pragmatic way. Playing
with lowlevel segment details is not nice and not always fun and results
in tradeoffs i dont like, but it's pretty much the only technique that
works on older x86 CPUs (as PaX has proven it too). If something better
comes along, then more power to it.

> [...] but I honestly think PaX is the better technology, and I think
> it's important that the best security technology be in place. [...]

i like PaX's completeness, and it has different tradeoffs. There is one
major and two medium tradeoffs that PaX has, from a distributor's POV:

1) the halving of the per-process VM space from 3GB to 1.5GB.

[ 2) the technique PaX uses (mirrored vmas) is pretty complex in terms
     of MM code. ]

[ 3) requires manual tagging of applications. ]

The technique exec-shield uses (to track the per-process 'highest
executable address') is pretty simple and non-intrusive on the
implementational level, but it also results in exec-shield's main
tradeoff:

   certain VM allocation patterns (e.g. doing mprotect() on an area that
   was allocated not via PROT_EXEC and was thus mapped high) can reduce
   exec-shield to 'only protects the stack against execution', or if
   the application needs an executable stack then reduces exec-shield to
   'no protection'.

it turns out these cases where exec-shield gets reduced are quite rare
and dont happen in critical applications. (partly because we fixed
affected critical applications - such fixes made sense even when not
considering the exec-shield impact.)

If a 'generic' distribution (i.e. one that has a significant userbase,
has thousands of packages that do get used) deviates from mainline it
wants to do it as simply as possible. (otherwise it would have the
overhead of porting/testing those deviations all the time.) In fact,
most of the extra patches that distribution kernels apply are patches
that they think will go mainline soon. If they apply any patch they know
wont be merged anytime soon they only do it if it is really needed, and
even then they try chose the variant that is smaller and easier to
maintain. Another important aspect is that extra patches should
obviously _widen_ the utility of the system, not narrow it.

On x86, VM space is scarce so PaX's halving of the VM space is a
'reduced utility' problem. (yes, you can turn it off per application and
get processes that have 3GB of address space, but that removes security
for those processes. Also, you cannot know in advance whether an
application will use more than 1.5GB of VM - different systems have
different usage patterns.)

[ PaX's #2 tradeoff is a maintainance overhead issue. Not an
  insurmountable issue because it is well-written kernel code, but
  combined with #1 it can tip the scale. PaX's #3 tradeoff is fixable -
  it could very well use the PT_GNU_STACK code now upstream. ]

you seem to be arguing for a 'no prisoners taken' approach to security,
and that is a perfectly fine approach if you maintain your own variant
of a distribution.

the other approach to security (which Fedora follows) is to 'make it as
seemless and automatic as possible, so that people actually end up using
our stuff.'

so while exec-shield is not "complete" in the sense of PaX, in practice
it is like 99% complete. E.g. on my Fedora desktop box:

  $ lsexec --all | grep 'execshield enabled' | wc -l
  86
  $ lsexec --all | grep 'execshield disabled' | wc -l
  0

and that's what really matters at the end of the day. (Anyway, you dont
have to believe and/or follow any of this, you are free to run your own
distribution, and if it's good then people will inevitably end up using
it.)

	Ingo

• Archive of above thread
• KernelTrap interview with Ingo Molnar