Colin Percival continues the discussion regarding the shared-cache vulnerability inherent in multi-core processors [story [1]], offering potential mitigation techniques in the form of fixes to the FreeBSD schedulers. Based on Percival's original discovery [2], information leakage between threads which share a processor core and the subsequent opportunity to monitor memory access patterns can be prevented by eliminating the co-scheduling of threads that have differing privileges. Additionally, Percival advises that a currently scheduled in-kernel thread should be capable of telling its siblings (who would likely run with the same privileges) to sleep in cases when it is handling sensitive data in a "non-oblivious" manner - IPSec being a good example of this. This would further secure sensitive data from monitoring. For these two solutions, he suggests the use of p_candebug(9) [3] for first and an as yet unimplemented IPI (Interprocessor Interrupt) mechanism for the second:
"1. The scheduler must be taught to not run threads on the same
processor core unless they p_candebug() each other. For reasons
of performance and locking, this is probably best accomplished by
only allowing threads to share a processor core if they belong
to the same process."
"2. When a thread is in the kernel, there must be a mechanism for it to IPI its siblings and put them to sleep, and then wake them
up later. This would be used any time when a thread in the kernel
is about to handle sensitive data in a non-oblivious manner; IPsec
is a good example of where this would be necessary."
Stephen Uphoff, referring to the relative difficulties of accomplishing this work among other scheduler development tasks, comments:
"Currently we don't even know if a thread running on another virtual CPU is in the kernel or not. Throwing preemption, interrupt and kernel threads, pinned threads, priority inheritance and the IPIs in the mix make correct and efficient hyperthreading safe scheduling difficult. This also looks like a lot of work and I am wondering if it would not be better spend on other scheduler improvements."
Other developers comment on the proposed solution and offer alternatives. While there will likely to be more deliberation, the availability of a fix other than disabling HTT may take some time to develop. Interested users should stay tuned. Read on for the partial thread and a link to the full discussion.
From: Colin Percival [email blocked] To: freebsd-arch Subject: Scheduler fixes for hyperthreading Date: 2005-05-21 23:11:07 As you are probably all aware by now, HyperThreading has been disabled on the stable and security branches due to a problem with information leakage between threads which are scheduled simultaneously on the two processor cores. Clearly, some people (and at least one large company) are unhappy about us having hyperthreading disbaled, so the security team would like to see hyperthreading re-enabled by default as soon as we believe that this can be done safely. The following must be done before hyperthreading is re-enabled: 1. The scheduler must be taught to not run threads on the same processor core unless they p_candebug() each other. For reasons of performance and locking, this is probably best accomplished by only allowing threads to share a processor core if they belong to the same process. 2. When a thread is in the kernel, there must be a mechanism for it to IPI its siblings and put them to sleep, and then wake them up later. This would be used any time when a thread in the kernel is about to handle sensitive data in a non-oblivious manner; IPsec is a good example of where this would be necessary. Does anyone want to step forward to work on this? Colin Percival
From: Marcel Moolenaar [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 0:34:37 On May 21, 2005, at 4:11 PM, Colin Percival wrote: *snip* > The following must be done before hyperthreading is re-enabled: > > 1. The scheduler must be taught to not run threads on the same > processor core unless they p_candebug() each other. For reasons > of performance and locking, this is probably best accomplished by > only allowing threads to share a processor core if they belong > to the same process. > 2. When a thread is in the kernel, there must be a mechanism for > it to IPI its siblings and put them to sleep, and then wake them > up later. This would be used any time when a thread in the kernel > is about to handle sensitive data in a non-oblivious manner; IPsec > is a good example of where this would be necessary. > > Does anyone want to step forward to work on this? Maybe it's a better idea to describe the problem in much more detail, rather than dictate what you want someone else to do? A pointer to where the problem is described/discussed would do. Just a thought, -- Marcel Moolenaar From: Colin Percival [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 0:49:20 Marcel Moolenaar wrote: > On May 21, 2005, at 4:11 PM, Colin Percival wrote: >> The following must be done before hyperthreading is re-enabled: >> [snip] > > Maybe it's a better idea to describe the problem in much more > detail, rather than dictate what you want someone else to do? > A pointer to where the problem is described/discussed would > do. The problem is described in my paper "Cache missing for fun and profit": http://www.daemonology.net/papers/htt.pdf [4] Put simply, threads which share a processor core can monitor each others' memory access patterns, so we need to ensure that such co-scheduling never happens between threads which have different privileges. The reason I cut through to explaining what needed to be done is that I discussed this at length with several people from the FreeBSD security team before and during BSDCan; but these discussions were obviously not public, so I can't give a reference to them. Colin Percival
From: Marcel Moolenaar [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 1:04:23 On May 21, 2005, at 5:49 PM, Colin Percival wrote: > Marcel Moolenaar wrote: >> On May 21, 2005, at 4:11 PM, Colin Percival wrote: >>> The following must be done before hyperthreading is re-enabled: >>> [snip] >> >> Maybe it's a better idea to describe the problem in much more >> detail, rather than dictate what you want someone else to do? >> A pointer to where the problem is described/discussed would >> do. > > The problem is described in my paper "Cache missing for fun and > profit": > http://www.daemonology.net/papers/htt.pdf [5] Thanks. > Put simply, threads which share a processor core can monitor each > others' > memory access patterns, so we need to ensure that such co-scheduling > never > happens between threads which have different privileges. I'll be studying your paper to see if it can be shown that the HT implementation in Itanium is affected as well. If not, any solution must be sufficiently machine dependent. > The reason I cut through to explaining what needed to be done is that > I discussed this at length with several people from the FreeBSD > security > team before and during BSDCan; but these discussions were obviously not > public, so I can't give a reference to them. I can only assume that the discussion was i386 centric (as this is typically the case). Hence my request for a problem description. -- Marcel Moolenaar From: Colin Percival [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 1:59:36 Marcel Moolenaar wrote: > On May 21, 2005, at 5:49 PM, Colin Percival wrote: >> Put simply, threads which share a processor core can monitor each others' >> memory access patterns, so we need to ensure that such co-scheduling >> never >> happens between threads which have different privileges. > > I'll be studying your paper to see if it can be shown that the HT > implementation in Itanium is affected as well. My understanding is that there are no currently released ia64 processors with hyperthreading support, but that some future ia64 processor(s) are likely to be affected. > I can only assume that the discussion was i386 centric (as this is > typically the case). Hence my request for a problem description. In addition to i386 and amd64, which are certainly affected, and ia64, which will probably be affected, there is a good chance that some powerpc processors are affected... the problem is a general one with shared caches and probably affects all currently existing simultaneous multithreading processors. I think the "right solution" is to make the basic functionality machine independent, but have the machine dependent initialization code determine which sets of threads share caches. Colin Percival
From: Stephen Uphoff [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 2:44:25 On Sat, 2005-05-21 at 19:11, Colin Percival wrote: > As you are probably all aware by now, HyperThreading has been > disabled on the stable and security branches due to a problem > with information leakage between threads which are scheduled > simultaneously on the two processor cores. Clearly, some people > (and at least one large company) are unhappy about us having > hyperthreading disbaled, so the security team would like to see > hyperthreading re-enabled by default as soon as we believe that > this can be done safely. > > The following must be done before hyperthreading is re-enabled: > > 1. The scheduler must be taught to not run threads on the same > processor core unless they p_candebug() each other. For reasons > of performance and locking, this is probably best accomplished by > only allowing threads to share a processor core if they belong > to the same process. > 2. When a thread is in the kernel, there must be a mechanism for > it to IPI its siblings and put them to sleep, and then wake them > up later. This would be used any time when a thread in the kernel > is about to handle sensitive data in a non-oblivious manner; IPsec > is a good example of where this would be necessary. > > Does anyone want to step forward to work on this? > > Colin Percival While I have been to your talk I have not read your paper yet and the following may be totally uninformed (Please be gentle :-) : Would it be enough to disable access to RDTSC for user processes? I believe the attack needs a very exact time source. Beside benchmarking - is there any other real use for RDTSC ? Is there any use of RDTSC that system requiring the security cannot live without? (We could even try to emulate the instruction if we really need to) I have to think more about possible scheduler changes. Currently we don't even know if a thread running on another virtual CPU is in the kernel or not. Throwing preemption, interrupt and kernel threads, pinned threads,priority inheritance and the IPIs in the mix make correct and efficient hyperthreading safe scheduling difficult. This also looks like a lot of work and I am wondering if it would not be better spend on other scheduler improvements. Stephan From: Bruce M Simpson [email blocked] Subject: Re: Scheduler fixes for hyperthreading Date: 2005-05-22 2:59:15 On Sat, May 21, 2005 at 10:44:25PM -0400, Stephan Uphoff wrote: > Beside benchmarking - is there any other real use for RDTSC ? > Is there any use of RDTSC that system requiring the security cannot live > without? (We could even try to emulate the instruction if we really need > to) A number of ports use RDTSC for high-resolution timing. The most obvious examples being machine emulators mostly used for gaming (UAE and MAME spring to mind, possibly also dosbox and others). I daresay VMware probably uses RDTSC too. BMS
Related Links:
- Archive of the Above Thread [6]
- Cache Missing for Fun and Profit [7]
- Hyper-Threading Considered Harmful [8]