Linus, please pull the latency tracer tree from:
   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git
Find the shortlog below.
This is the latency tracer from -rt, split up and much cleaned up by

Steve Rostedt, Arnaldo Carvalho and myself. The main motivation of this

tracer is to be utilized by high-level user tools such as LatencyTOP, to

analyze system behavior and to enable users to give feedback to kernel

developers.
It has been in -rt for years and was found to be very useful there, and

in the last month it has been posted to lkml by Steve about 10 times.

[This final version is simpler and cleaner than the last lkml version

(v10) - lets start simple. ]
It does include one very interesting new feature that deserves to be

mentioned outside of the shortlog: 'dynamic ftrace' - which is a

transparent kernel-image-patcher mechanism that lazily patches out

mcount callsites from all functions that get executed. [if tracing is

disabled] These patched out callsites are remembered, and are patched

back in when tracing is enabled.
This technique does not just accelelerate the "tracing disabled" case

enormously, we were in fact unable to measure _any_ performance

difference (within noise) between an mcount-enabled dyn-ftrace [but

tracing-disabled] and a vanilla kernel (!), on modern CPUs.
There is still the cost of the +5 byte function size that mcount causes,

and the resulting +~1% kernel text size increase, but the overhead was

not measurable in micro or macro benchmarks that we tried. (probably

because there are no branches added to the hot paths and the only

overhead is the NOP that is inserted into the prologue of the function -

which modern CPUs will just eat up as if it didnt exist.)
All in one, i think this is one of the most promising developments in

terms of Linux kernel instrumentation that happened in the past few

years:
- it gives us full, very meaningful instrumentation (there are

  70,000+ function calls in an a...
[ message continues ]

i've updated the git tree - find the shortlog below.
It's a few small changes (got rid of the DONT_TRACE_NMI complication,

Steve removed the max_mutex, removed 2 lines of leftover debugging

code), and streamlined the splitup to reduce the number of patches from

24 to 18. No bugs have been found in the tracer code in overnight

testing, and the queue still is bisectable at every patch point with

tracers on and off as well.
(i've attached the delta patch below the diffstat, relative to

yesterday's tree - the full patch would be too large for lkml.)
	Ingo
------------------>

Arnaldo Carvalho de Melo (2):

      ftrace: annotate core code that should not be traced

      ftrace: add basic support for gcc profiler instrumentation
Ingo Molnar (2):

      sched: add latency tracer callbacks to the scheduler

      tracing: add notrace to linkage.h
Steven Rostedt (14):

      rcu: add support for dynamic ticks and preempt rcu

      printk: dont wake up klogd with the rq locked

      ftrace: add preempt_enable/disable notrace macros

      ftrace: make the task state char-string visible to all

      x86: add notrace annotations to vsyscall.

      ftrace: add notrace annotations for NMI routines

      ftrace: latency tracer infrastructure

      ftrace: function tracer

      ftrace: add tracing of context switches

      ftrace: tracer for scheduler wakeup latency

      ftrace: trace irq disabled critical timings

      ftrace: trace preempt off critical timings

      ftrace: dynamic enabling/disabling of function calls

      ftrace: add ftrace_enabled sysctl to disable mcount function
 Makefile                          |    3 +

 arch/x86/Kconfig                  |    1 +

 arch/x86/kernel/Makefile          |    1 +

 arch/x86/kernel/entry_32.S        |   27 +

 arch/x86/kernel/entry_64.S        |   37 +

 arch/x86/kernel/ftrace.c          |  237 ++++++

 arch/x86/kernel/nmi_32.c          |    3 +-

 arch/x86/kernel/nmi_64.c          |    6 +-

 arch/x86/kernel/process_32....
[ message continues ]

I've never seen any of this code before and googling several of the

patch titles turns up this email and nothing else.
Could we get the patches out for review in the usual way please?
Bear in mind that people will actually want to understand how this

code works.   Particularly architecture maintainers.  (I assume it's

portable to non-x86?).
--

there might be some confusion here. Google for "mcount tracing utility"

- there's 2270 hits. There's been 8 full series posted to lkml in the

past month:
 Date: Thu, 03 Jan 2008 02:16:09 -0500

 Subject: [RFC PATCH 00/11] mcount tracing utility
 Date: Wed, 09 Jan 2008 18:29:14 -0500

 Subject: [RFC PATCH 00/22 -v2] mcount and latency tracing utility -v2
 Date: Tue, 15 Jan 2008 15:49:07 -0500

 Subject: [RFC PATCH 00/30 v3] mcount and latency tracing utility -v3
 Date: Mon, 21 Jan 2008 10:22:31 -0500

 Subject: [RFC PATCH 00/23 -v4] mcount and latency tracing utility -v4
 Date: Wed, 23 Jan 2008 11:02:36 -0500

 Subject: [PATCH 00/20 -v5] mcount and latency tracing utility -v5
 Date: Fri, 25 Jan 2008 23:21:52 -0500

 Subject: [PATCH 00/23 -v6] mcount and latency tracing utility -v6
 Date: Tue, 29 Jan 2008 22:15:21 -0500

 Subject: [PATCH 00/22 -v7] mcount and latency tracing utility -v7
 Date: Wed, 30 Jan 2008 16:03:57 -0500

 Subject: [PATCH 00/23 -v8] mcount and latency tracing utility -v8
it derives from the same latency tracer code that we wrote for -rt that

has been around for years:
   http://people.redhat.com/mingo/latency-tracing-patches/
we renamed the concept to 'ftrace' during pre-merge cleanups, perhaps

that is what caused you to not recognize this? (mcount is a confusing

name and ties it to a gcc feature while there's nothing gcc specific

about this concept.)
	Ingo

--

Oh.  That clang you heard was a penny dropping.
I'd been kind of ignoring those patches assuming I had a couple more

months.
--

hot off the presses: Steve has just finished doing a complete set of

lmbench and hbackbench runs (on 2.8GHz Xeons, dual socket). The

comparison results are at:
    http://rostedt.homelinux.com/dyn-ftrace-lmbench/lmbench-table
the results show zero lmbench/hackbench overhead from dyn-ftrace (with

thousands of functions instrumented) apart of the usual lmbench jitter.
The core kernel results for example [these used to be rather sensitive

on mcount overhead]:
Host                 OS  Mhz null null      open slct sig  sig  fork exec sh

                             call  I/O stat clos TCP  inst hndl proc proc proc

--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

ftrace-jm  Linux 2.6.24 2783 0.65 0.74 4.74 6.96 5.28 0.76 4.29 290. 866. 2626

ftrace-no  Linux 2.6.24 2784 0.65 0.74 4.75 7.14 5.31 0.76 3.92 282. 837. 2591

no-patche  Linux 2.6.24 2786 0.64 0.71 4.86 7.10 5.34 0.78 3.89 283. 829. 2572

patched-n  Linux 2.6.24 2784 0.64 0.71 4.72 6.98 5.35 0.78 3.94 284. 832. 2582
"ftrace-jm": dyn-ftrace, 2-byte JMPs patched in the mcount callsite

"ftrace-no": dyn-ftrace, 5-byte NOPs patched in (devel patch, not in

             this pull request)

"no-patche": no patches [vanilla Linus -git kernel]

"patched-n": patched with the ftrace tree, all tracer options off
i can see an outlier in the "Create" results - that's noise i think

because the patched+options-off kernel is in essence the same as

vanilla. "File reread" is noisy and unreliable as ever. 16-task

ctx-switch results are noisy too (especially the 64k working set ones) -

this is a HT system hence the multi-threaded results are fundamentally

noisy due to sibling interaction.
The hackbench results look good too.
enabling full tracing of all the tens of thousands of kernel functions

makes things measurably slower - but that is expected. (filtered

patching of a user-selected list of function names is an upcoming

feature)
the full raw results are at:
   [ message continues ]