FreeBSD: Lock Order Reversal Documentation

Submitted by njc
on August 11, 2004 - 12:53pm

While FreeBSD -current is still moving toward more stable footing, many users have posted issues with panics and deadlocks in recent kernel builds. Bjoern A. Zeeb has kindly compiled a running list of lock order reversals, links to relevant threads, PRs, and existing patches. Lock order reversal messages are the result of FreeBSD's lock validation facility, witness(4), notifying the system of potential deadlocks as a means for developers to isolate bugs. Robert Watson explains in a Dec. 2003 thread:

"[...]Among other things, Witness performs run-time lock order
verification using a combination of hard coded lock orders, and run-time
detected lock orders, and generates console warnings when lock orders are
violated. The intent of this is to detect the potential for deadlocks due
to lock order violations; it's worth observing that Witness is actually
slightly conservative, and so it's possible to get false positives...."

Read on for relevant links.


From:Robert Watson [email blocked]
To: freebsd-current
Subject: lock order reversals - what do they mean?
Date: Sun Dec 14 09:35:21 PST 2003

On Sun, 14 Dec 2003, C. Kukulies wrote:

> I'm getting 
> lock order reversal
> 1st 0xc09524c0 UMA lock (UMA lock) @ /u/src/sys/vm/uma_core.c:1200
> 2nd 0xc1431100 system map (system map) @ /u/src/sys/vm/vm_map.c:2210
> Stack backtrace:
> 
> This happenend while doing a find . something on my notebook hd. 

These warnings are generated by Witness, a run-time lock diagnostic system
found in FreeBSD 5-CURRENT kernels (but removed in releases).  You can
read more about Witness in the WITNESS(4) man page, which talks about its
capabilities.  Among other things, Witness performs run-time lock order
verification using a combination of hard coded lock orders, and run-time
detected lock orders, and generates console warnings when lock orders are
violated.  The intent of this is to detect the potential for deadlocks due
to lock order violations; it's worth observing that Witness is actually
slightly conservative, and so it's possible to get false positives.  In
the event that Witness is accurately reporting a lock order problem, it's
basically saying "If you were unlucky, a deadlock would have happened
here".  There are a couple of "well known" false positives, which we need
to do a better job of documenting to prevent spurious reports.  The
non-well-known ones typically correspond to bugs in newly added locking,
as lock order reversals usually get fixed pretty quickly because Witness
is busy generating warnings :-).  I believe the reversal you've reported
is a false positive.

Thanks,

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects



Related Links: