Re: Unified tracing buffer

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Linus Torvalds
Date: Monday, September 22, 2008 - 9:05 pm

On Mon, 22 Sep 2008, Mathieu Desnoyers wrote:

Sure (if you have a barrier - not all architectures will imply that for an 
incrment).

But that still doesn't mean a thing.

You have two events (a) and (b), and you put trace-points on each. In your 
trace, you see (a) before (b) by comparing the numbers. But what does that 
mean? 

The actual event that you traced is not the trace-point - the trace-point 
is more like a fancy "printk". And the fact that one showed up before 
another in the trace buffer, doesn't mean that the events _around_ the 
trace happened in the same order.

You can use the barriers to make a partial ordering, and if you have a 
separate tracepoint for entry into a region and exit, you can perhaps show 
that they were totally disjoint. Or maybe they were partially overlapping, 
and you'll never know exactly how they overlapped.

Example:

	trace(..);
	do_X();

being executed on two different CPU's. In the trace, CPU#1 was before 
CPU#2. Does that mean that "do_X()" happened first on CPU#1?

No.

The only way to show that would be to put a lock around the whole trace 
_and_ operation X, ie

	spin_lock(..);
	trace(..);
	do_X();
	spin_unlock(..);

and now, if CPU#1 shows up in the trace first, then you know that do_X() 
really did happen first on CPU#1. Otherwise you basically know *nothing*, 
and the ordering of the trace events was totally and utterly meaningless.

See? Trace events themselves may be ordered, but the point of the trace 
event is never to know the ordering of the trace itself - it's to know the 
ordering of the code we're interested in tracing. The ordering of the 
trace events themselves is irrelevant and not useful.

And I'd rather see people _understand_ that, than if they think the 
ordering is somehow something they can trust.

Btw, if you _do_ have locking, then you can also know that the "do_X()" 
operations will be essentially as far apart in some theoretical notion of 
"time" (let's imagine that we do have global time, even if we don't) as 
the cost of the trace operation and do_X() itself.

So if we _do_ have locking (and thus a valid ordering that actually can 
matter), then the TSC doesn't even have to be synchronized on a cycle 
basis across CPU's - it just needs to be close enough that you can tell 
which one happened first (and with ordering, that's a valid thing to do). 

So you don't even need "perfect" synchronization, you just need something 
reasonably close, and you'll be able to see ordering from TSC counts 
without having that horrible bouncing cross-CPU thing that will impact 
performance a lot.

Quite frankly, I suspect that anybody who wants to have a global counter 
might as well almost just have a global ring-buffer. The trace events 
aren't going to be CPU-local anyway if you need to always update a shared 
cacheline - and you might as well make the shared cacheline be the ring 
buffer head with a spinlock in it.

That may not be _quite_ true, but it's probably close enough.

		Linus
--
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Unified tracing buffer, Martin Bligh, (Fri Sep 19, 2:33 pm)
Re: Unified tracing buffer, Randy Dunlap, (Fri Sep 19, 2:42 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 2:57 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 3:09 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 3:19 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Fri Sep 19, 3:28 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Fri Sep 19, 3:41 pm)
Re: Unified tracing buffer, Frank Ch. Eigler, (Fri Sep 19, 4:18 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Fri Sep 19, 5:07 pm)
Re: Unified tracing buffer, Marcel Holtmann, (Fri Sep 19, 5:26 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Sat Sep 20, 1:10 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:26 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:29 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:50 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 2:03 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 4:40 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 4:44 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 6:37 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 6:51 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 6:55 am)
Re: Unified tracing buffer, Arjan van de Ven, (Sat Sep 20, 7:12 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 7:54 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Sun Sep 21, 8:09 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 2:57 am)
Re: Unified tracing buffer, K.Prasad, (Mon Sep 22, 6:57 am)
Re: Unified tracing buffer, K.Prasad, (Mon Sep 22, 7:07 am)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 7:45 am)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 9:29 am)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 9:36 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 11:45 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 11:52 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 12:45 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 1:13 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 1:50 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 2:39 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 3:25 pm)
Re: Unified tracing buffer, Darren Hart, (Mon Sep 22, 4:11 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 4:16 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 5:04 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 5:05 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 5:12 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 5:39 pm)
Re: Unified tracing buffer, Roland Dreier, (Mon Sep 22, 6:26 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 6:39 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:02 pm)
Re: Unified tracing buffer, Darren Hart, (Mon Sep 22, 7:26 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:30 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:31 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:36 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:49 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:05 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 8:06 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 8:26 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:27 pm)
Re: Unified tracing buffer, Andi Kleen, (Mon Sep 22, 8:33 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:36 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 8:43 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 8:47 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 9:05 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 9:10 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 9:17 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 9:19 pm)
Re: Unified tracing buffer, Andi Kleen, (Mon Sep 22, 10:04 pm)
Re: Unified tracing buffer, Tom Zanussi, (Mon Sep 22, 10:25 pm)
[PATCH 1/3] relay - clean up subbuf switch, Tom Zanussi, (Mon Sep 22, 10:27 pm)
[PATCH 2/3] relay - make subbuf switch replaceable, Tom Zanussi, (Mon Sep 22, 10:27 pm)
[PATCH 3/3] relay - add channel flags, Tom Zanussi, (Mon Sep 22, 10:27 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Tue Sep 23, 2:31 am)
Re: Unified tracing buffer, Steven Rostedt, (Tue Sep 23, 3:53 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 6:50 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 7:00 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 7:12 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Tue Sep 23, 7:36 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 7:49 am)
Re: Unified tracing buffer, Frank Ch. Eigler, (Tue Sep 23, 8:02 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 8:04 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:21 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:23 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:30 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 8:46 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 9:01 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 10:04 am)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 10:30 am)
Re: Unified tracing buffer, K.Prasad, (Tue Sep 23, 10:55 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Tue Sep 23, 10:59 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 11:13 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 11:27 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 11:28 am)
Re: Unified tracing buffer, Christoph Lameter, (Tue Sep 23, 11:33 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 11:56 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 11:59 am)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 12:36 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 12:38 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 12:41 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 12:50 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 1:03 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 1:03 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 1:08 pm)
Re: [PATCH 1/3] relay - clean up subbuf switch, Andrew Morton, (Tue Sep 23, 1:15 pm)
Re: [PATCH 2/3] relay - make subbuf switch replaceable, Andrew Morton, (Tue Sep 23, 1:17 pm)
Re: [PATCH 3/3] relay - add channel flags, Andrew Morton, (Tue Sep 23, 1:20 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 2:02 pm)
Re: Unified tracing buffer, Tom Zanussi, (Tue Sep 23, 8:50 pm)
Re: [PATCH 3/3] relay - add channel flags, Tom Zanussi, (Tue Sep 23, 8:57 pm)
Re: Unified tracing buffer, K.Prasad, (Tue Sep 23, 10:42 pm)
[RFC PATCH 0/8] current relay cleanup patchset, Tom Zanussi, (Wed Sep 24, 11:07 pm)
Re: Unified tracing buffer, Jason Baron, (Thu Oct 2, 8:28 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 9:11 am)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 11:37 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 12:10 pm)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 12:25 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 12:56 pm)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 1:25 pm)
Re: Unified tracing buffer, Frank Ch. Eigler, (Fri Oct 3, 2:52 pm)