Colin Percival continues the discussion regarding the shared-cache vulnerability inherent in multi-core processors [story], offering potential mitigation techniques in the form of fixes to the FreeBSD schedulers. Based on Percival's original discovery, information leakage between threads which share a processor core and the subsequent opportunity to monitor memory access patterns can be prevented by eliminating the co-scheduling of threads that have differing privileges. Additionally, Percival advises that a currently scheduled in-kernel thread should be capable of telling its siblings (who would likely run with the same privileges) to sleep in cases when it is handling sensitive data in a "non-oblivious" manner - IPSec being a good example of this. This would further secure sensitive data from monitoring. For these two solutions, he suggests the use of p_candebug(9) for first and an as yet unimplemented IPI (Interprocessor Interrupt) mechanism for the second:

"1. The scheduler must be taught to not run threads on the same
processor core unless they p_candebug() each other. For reasons
of performance and locking, this is probably best accomplished by
only allowing threads to share a processor core if they belong
to the same process."

"2. When a thread is in the kernel, there must be a mechanism for it to IPI its siblings and put them to sleep, and then wake them
up later. This would be used any time when a thread in the kernel
is about to handle sensitive data in a non-oblivious manner; IPsec
is a good example of where this would be necessary."

Stephen Uphoff, referring to the relative difficulties of accomplishing this work among other scheduler development tasks, comments:

"Currently we don't even know if a thread running on another virtual CPU is in the kernel or not. Throwing preemption, interrupt and kernel threads, pinned threads, priority inheritance and the IPIs in the mix make correct and efficient hyperthreading safe scheduling difficult. This also looks like a lot of work and I am wondering if it would not be better spend on other scheduler improvements."

Other developers comment on the proposed solution and offer alternatives. While there will likely to be more deliberation, the availability of a fix other than disabling HTT may take some time to develop. Interested users should stay tuned. Read on for the partial thread and a link to the full discussion.

From: Colin Percival [email blocked]
To: freebsd-arch
Subject: Scheduler fixes for hyperthreading
Date: 2005-05-21 23:11:07

As you are probably all aware by now, HyperThreading has been
disabled on the stable and security branches due to a problem
with information leakage between threads which are scheduled
simultaneously on the two processor cores.  Clearly, some people
(and at least one large company) are unhappy about us having
hyperthreading disbaled, so the security team would like to see
hyperthreading re-enabled by default as soon as we believe that
this can be done safely.

  The following must be done before hyperthreading is re-enabled:

1. The scheduler must be taught to not run threads on the same
processor core unless they p_candebug() each other.  For reasons
of performance and locking, this is probably best accomplished by
only allowing threads to share a processor core if they belong
to the same process.
2. When a thread is in the kernel, there must be a mechanism for
it to IPI its siblings and put them to sleep, and then wake them
up later.  This would be used any time when a thread in the kernel
is about to handle sensitive data in a non-oblivious manner; IPsec
is a good example of where this would be necessary.

  Does anyone want to step forward to work on this?

Colin Percival



From: Marcel Moolenaar [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 0:34:37

On May 21, 2005, at 4:11 PM, Colin Percival wrote:

*snip*
>   The following must be done before hyperthreading is re-enabled:
>
> 1. The scheduler must be taught to not run threads on the same
> processor core unless they p_candebug() each other.  For reasons
> of performance and locking, this is probably best accomplished by
> only allowing threads to share a processor core if they belong
> to the same process.
> 2. When a thread is in the kernel, there must be a mechanism for
> it to IPI its siblings and put them to sleep, and then wake them
> up later.  This would be used any time when a thread in the kernel
> is about to handle sensitive data in a non-oblivious manner; IPsec
> is a good example of where this would be necessary.
>
>   Does anyone want to step forward to work on this?

Maybe it's a better idea to describe the problem in much more
detail, rather than dictate what you want someone else to do?
A pointer to where the problem is described/discussed would
do.

Just a thought,

-- 
  Marcel Moolenaar

From: Colin Percival [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 0:49:20

Marcel Moolenaar wrote:
> On May 21, 2005, at 4:11 PM, Colin Percival wrote:
>>   The following must be done before hyperthreading is re-enabled:
>> [snip]
> 
> Maybe it's a better idea to describe the problem in much more
> detail, rather than dictate what you want someone else to do?
> A pointer to where the problem is described/discussed would
> do.

The problem is described in my paper "Cache missing for fun and profit":
http://www.daemonology.net/papers/htt.pdf

Put simply, threads which share a processor core can monitor each others'
memory access patterns, so we need to ensure that such co-scheduling never
happens between threads which have different privileges.

The reason I cut through to explaining what needed to be done is that
I discussed this at length with several people from the FreeBSD security
team before and during BSDCan; but these discussions were obviously not
public, so I can't give a reference to them.

Colin Percival



From: Marcel Moolenaar [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 1:04:23

On May 21, 2005, at 5:49 PM, Colin Percival wrote:

> Marcel Moolenaar wrote:
>> On May 21, 2005, at 4:11 PM, Colin Percival wrote:
>>>   The following must be done before hyperthreading is re-enabled:
>>> [snip]
>>
>> Maybe it's a better idea to describe the problem in much more
>> detail, rather than dictate what you want someone else to do?
>> A pointer to where the problem is described/discussed would
>> do.
>
> The problem is described in my paper "Cache missing for fun and 
> profit":
> http://www.daemonology.net/papers/htt.pdf

Thanks.

> Put simply, threads which share a processor core can monitor each 
> others'
> memory access patterns, so we need to ensure that such co-scheduling 
> never
> happens between threads which have different privileges.

I'll be studying your paper to see if it can be shown that the HT
implementation in Itanium is affected as well. If not, any solution
must be sufficiently machine dependent.

> The reason I cut through to explaining what needed to be done is that
> I discussed this at length with several people from the FreeBSD 
> security
> team before and during BSDCan; but these discussions were obviously not
> public, so I can't give a reference to them.

I can only assume that the discussion was i386 centric (as this is
typically the case). Hence my request for a problem description.

-- 
  Marcel Moolenaar

From: Colin Percival [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 1:59:36

Marcel Moolenaar wrote:
> On May 21, 2005, at 5:49 PM, Colin Percival wrote:
>> Put simply, threads which share a processor core can monitor each others'
>> memory access patterns, so we need to ensure that such co-scheduling
>> never
>> happens between threads which have different privileges.
> 
> I'll be studying your paper to see if it can be shown that the HT
> implementation in Itanium is affected as well.

My understanding is that there are no currently released ia64 processors
with hyperthreading support, but that some future ia64 processor(s) are
likely to be affected.

> I can only assume that the discussion was i386 centric (as this is
> typically the case). Hence my request for a problem description.

In addition to i386 and amd64, which are certainly affected, and ia64,
which will probably be affected, there is a good chance that some powerpc
processors are affected... the problem is a general one with shared caches
and probably affects all currently existing simultaneous multithreading
processors.

I think the "right solution" is to make the basic functionality machine
independent, but have the machine dependent initialization code determine
which sets of threads share caches.

Colin Percival



From: Stephen Uphoff [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 2:44:25

On Sat, 2005-05-21 at 19:11, Colin Percival wrote: 
>   As you are probably all aware by now, HyperThreading has been
> disabled on the stable and security branches due to a problem
> with information leakage between threads which are scheduled
> simultaneously on the two processor cores.  Clearly, some people
> (and at least one large company) are unhappy about us having
> hyperthreading disbaled, so the security team would like to see
> hyperthreading re-enabled by default as soon as we believe that
> this can be done safely.
> 
>   The following must be done before hyperthreading is re-enabled:
> 
> 1. The scheduler must be taught to not run threads on the same
> processor core unless they p_candebug() each other.  For reasons
> of performance and locking, this is probably best accomplished by
> only allowing threads to share a processor core if they belong
> to the same process.
> 2. When a thread is in the kernel, there must be a mechanism for
> it to IPI its siblings and put them to sleep, and then wake them
> up later.  This would be used any time when a thread in the kernel
> is about to handle sensitive data in a non-oblivious manner; IPsec
> is a good example of where this would be necessary.
> 
>   Does anyone want to step forward to work on this?
>
> Colin Percival

While I have been to your talk I have not read your paper yet and the
following may be totally uninformed (Please be gentle :-) :

Would it be enough to disable access to RDTSC for user processes?
I believe the attack needs a very exact time source.

Beside benchmarking - is there any other real use for RDTSC ?
Is there any use of RDTSC that system requiring the security cannot live
without? (We could even try to emulate the instruction if we really need
to)

I have to think more about possible scheduler changes.
Currently we don't even  know if a thread running on another virtual CPU
is in the kernel or not.
Throwing preemption, interrupt and kernel threads, pinned
threads,priority inheritance and the IPIs in the mix make correct and
efficient hyperthreading safe scheduling difficult.
This also looks like a lot of work and I am wondering if it would not be
better spend on other scheduler improvements.

Stephan

From: Bruce M Simpson [email blocked]
Subject: Re: Scheduler fixes for hyperthreading
Date: 2005-05-22 2:59:15

On Sat, May 21, 2005 at 10:44:25PM -0400, Stephan Uphoff wrote:
> Beside benchmarking - is there any other real use for RDTSC ?
> Is there any use of RDTSC that system requiring the security cannot live
> without? (We could even try to emulate the instruction if we really need
> to)

A number of ports use RDTSC for high-resolution timing. The most obvious
examples being machine emulators mostly used for gaming (UAE and MAME
spring to mind, possibly also dosbox and others).

I daresay VMware probably uses RDTSC too.

BMS

• Archive of the Above Thread
• Cache Missing for Fun and Profit
• Hyper-Threading Considered Harmful