Unified tracing buffer

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Martin Bligh
Date: Friday, September 19, 2008 - 2:33 pm

During kernel summit and Plumbers conference, Linus and others
expressed a desire for a unified
tracing buffer system for multiple tracing applications (eg ftrace,
lttng, systemtap, blktrace, etc) to use.
This provides several advantages, including the ability to interleave
data from multiple sources,
not having to learn 200 different tools, duplicated code/effort, etc.

Several of us got together last night and tried to cut this down to
the simplest usable system
we could agree on (and nobody got hurt!). This will form version 1.
I've sketched out a few
enhancements we know that we want, but have agreed to leave these
until version 2.
The answer to most questions about the below is "yes we know, we'll
fix that in version 2"
(or 3). Simplicity was the rule ...

Sketch of design.  Enjoy flaming me. Code will follow shortly.


STORAGE
-------

We will support multiple buffers for different tracing systems, with
separate names, event id spaces.
Event ids are 16 bit, dynamically allocated.
A "one line of text" print function will be provided for each event,
or use the default (probably hex printf)
Will provide a "flight data recorder" mode, and a "spool to disk" mode.

Circular buffer per cpu, protected by per-cpu spinlock_irq
Word aligned records.
Variable record length, header will start with length record.
Timestamps in fixed timebase, monotonically increasing (across all CPUs)


INPUT_FUNCTIONS
---------------

allocate_buffer (name, size)
        return buffer_handle

register_event (buffer_handle, event_id, print_function)
        You can pass in a requested event_id from a fixed set, and
will be given it, or an error
        0 means allocate me one dynamically
        returns event_id     (or -E_ERROR)

record_event (buffer_handle, event_id, length, *buf)


OUTPUT
------

Data will be output via debugfs, and provide the following output streams:

/debugfs/tracing/<name>/buffers/text
    clear text stream (will merge the per-cpu streams via insertion
sort, and use the print functions)

/debugfs/tracing/<name>/buffers/binary[cpu_number]
    per-cpu binary data


CONTROL
-------

Sysfs style tree under debugfs

/debugfs/tracing/<name>/buffers/enabed         <--- binary value

/debugfs/tracing/<name>/<event1>
/debugfs/tracing/<name>/<event2>
    etc ...
    provides a way to enable/disable events, see what's available, and
what's enabled.


KNOWN ISSUES / PLANS
-------------------

No way to unregister buffers and events.
    Will provide an unregister_buffer and unregister_event call


Generating systemwide time is hard on some platforms
    Yes. Time-based output provides a lot of simplicity for the user though
    We won't support these platforms at first, we'll add functionality
to make it work for them later.
    (plan based on tick-based ms timing, plus counter offset from that
if needed).

Spinlock_irq is ineffecient, and doesn't support tracing in NMIs
    True. We'll implement a lockless scheme later (see lttng)

Putting a length record in every event is inefficient
    True. Fixed record length with optional extensions is better, but
more complex. v2.

Putting a full timestamp rather than an offset in every event is inefficient
    See above. True, but v2.

Relayfs already exists! use that!
    People were universally not keen on that idea. Complexity, interface, etc.
    We're also providing some higher level shared functions for time &
event ids.

There's no way to decode the binary data stream
    Code will be shared from the kernel to decode it, so that we can
get the compact binary
    format and decode it later. That code will be kept in the kernel
tree (it's a trivial piece of C).
    Version 1.1 ;-)
--
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Unified tracing buffer, Martin Bligh, (Fri Sep 19, 2:33 pm)
Re: Unified tracing buffer, Randy Dunlap, (Fri Sep 19, 2:42 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 2:57 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 3:09 pm)
Re: Unified tracing buffer, Martin Bligh, (Fri Sep 19, 3:19 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Fri Sep 19, 3:28 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Fri Sep 19, 3:41 pm)
Re: Unified tracing buffer, Frank Ch. Eigler, (Fri Sep 19, 4:18 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Fri Sep 19, 5:07 pm)
Re: Unified tracing buffer, Marcel Holtmann, (Fri Sep 19, 5:26 pm)
Re: Unified tracing buffer, Olaf Dabrunz, (Sat Sep 20, 1:10 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:26 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:29 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 1:50 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 2:03 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 4:40 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 4:44 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 6:37 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 6:51 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Sat Sep 20, 6:55 am)
Re: Unified tracing buffer, Arjan van de Ven, (Sat Sep 20, 7:12 am)
Re: Unified tracing buffer, Steven Rostedt, (Sat Sep 20, 7:54 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Sun Sep 21, 8:09 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 2:57 am)
Re: Unified tracing buffer, K.Prasad, (Mon Sep 22, 6:57 am)
Re: Unified tracing buffer, K.Prasad, (Mon Sep 22, 7:07 am)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 7:45 am)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 9:29 am)
Re: Unified tracing buffer, Peter Zijlstra, (Mon Sep 22, 9:36 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 11:45 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 11:52 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 12:45 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 1:13 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 1:50 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 2:39 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 3:25 pm)
Re: Unified tracing buffer, Darren Hart, (Mon Sep 22, 4:11 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 4:16 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 5:04 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 5:05 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 5:12 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 5:39 pm)
Re: Unified tracing buffer, Roland Dreier, (Mon Sep 22, 6:26 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 6:39 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:02 pm)
Re: Unified tracing buffer, Darren Hart, (Mon Sep 22, 7:26 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:30 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:31 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:36 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 7:49 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:05 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 8:06 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 8:26 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:27 pm)
Re: Unified tracing buffer, Andi Kleen, (Mon Sep 22, 8:33 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Mon Sep 22, 8:36 pm)
Re: Unified tracing buffer, Steven Rostedt, (Mon Sep 22, 8:43 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 8:47 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 9:05 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Mon Sep 22, 9:10 pm)
Re: Unified tracing buffer, Martin Bligh, (Mon Sep 22, 9:17 pm)
Re: Unified tracing buffer, Linus Torvalds, (Mon Sep 22, 9:19 pm)
Re: Unified tracing buffer, Andi Kleen, (Mon Sep 22, 10:04 pm)
Re: Unified tracing buffer, Tom Zanussi, (Mon Sep 22, 10:25 pm)
[PATCH 1/3] relay - clean up subbuf switch, Tom Zanussi, (Mon Sep 22, 10:27 pm)
[PATCH 2/3] relay - make subbuf switch replaceable, Tom Zanussi, (Mon Sep 22, 10:27 pm)
[PATCH 3/3] relay - add channel flags, Tom Zanussi, (Mon Sep 22, 10:27 pm)
Re: Unified tracing buffer, Peter Zijlstra, (Tue Sep 23, 2:31 am)
Re: Unified tracing buffer, Steven Rostedt, (Tue Sep 23, 3:53 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 6:50 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 7:00 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 7:12 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Tue Sep 23, 7:36 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 7:49 am)
Re: Unified tracing buffer, Frank Ch. Eigler, (Tue Sep 23, 8:02 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 8:04 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:21 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:23 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 8:30 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 8:46 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 9:01 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 10:04 am)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 10:30 am)
Re: Unified tracing buffer, K.Prasad, (Tue Sep 23, 10:55 am)
Re: Unified tracing buffer, KOSAKI Motohiro, (Tue Sep 23, 10:59 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Tue Sep 23, 11:13 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 11:27 am)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 11:28 am)
Re: Unified tracing buffer, Christoph Lameter, (Tue Sep 23, 11:33 am)
Re: Unified tracing buffer, Linus Torvalds, (Tue Sep 23, 11:56 am)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 11:59 am)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 12:36 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 12:38 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 12:41 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 12:50 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 1:03 pm)
Re: Unified tracing buffer, Masami Hiramatsu, (Tue Sep 23, 1:03 pm)
Re: Unified tracing buffer, Thomas Gleixner, (Tue Sep 23, 1:08 pm)
Re: [PATCH 1/3] relay - clean up subbuf switch, Andrew Morton, (Tue Sep 23, 1:15 pm)
Re: [PATCH 2/3] relay - make subbuf switch replaceable, Andrew Morton, (Tue Sep 23, 1:17 pm)
Re: [PATCH 3/3] relay - add channel flags, Andrew Morton, (Tue Sep 23, 1:20 pm)
Re: Unified tracing buffer, Martin Bligh, (Tue Sep 23, 2:02 pm)
Re: Unified tracing buffer, Tom Zanussi, (Tue Sep 23, 8:50 pm)
Re: [PATCH 3/3] relay - add channel flags, Tom Zanussi, (Tue Sep 23, 8:57 pm)
Re: Unified tracing buffer, K.Prasad, (Tue Sep 23, 10:42 pm)
[RFC PATCH 0/8] current relay cleanup patchset, Tom Zanussi, (Wed Sep 24, 11:07 pm)
Re: Unified tracing buffer, Jason Baron, (Thu Oct 2, 8:28 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 9:11 am)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 11:37 am)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 12:10 pm)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 12:25 pm)
Re: Unified tracing buffer, Mathieu Desnoyers, (Fri Oct 3, 12:56 pm)
Re: Unified tracing buffer, Jason Baron, (Fri Oct 3, 1:25 pm)
Re: Unified tracing buffer, Frank Ch. Eigler, (Fri Oct 3, 2:52 pm)