Re: [PATCH] poll: Avoid extra wakeups in select/poll

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

To: Ingo Molnar <mingo@...>

Cc: Christoph Lameter <cl@...>, linux kernel <linux-kernel@...>, Andi Kleen <andi@...>, David Miller <davem@...>, <jesse.brandeburg@...>, <netdev@...>, <haoki@...>, <mchan@...>, <davidel@...>

Subject: Re: [PATCH] poll: Avoid extra wakeups in select/poll

Date: Thursday, April 30, 2009 - 10:08 am

Ingo Molnar a écrit :
quoted text> * Eric Dumazet <dada1@cosmosbay.com> wrote:
> 
>> Ingo Molnar a écrit :
>>> * Eric Dumazet <dada1@cosmosbay.com> wrote:
>>>
>>>> On uddpping, I had prior to the patch about 49000 wakeups per 
>>>> second, and after patch about 26000 wakeups per second (matches 
>>>> number of incoming udp messages per second)
>>> very nice. It might not show up as a real performance difference if 
>>> the CPUs are not fully saturated during the test - but it could show 
>>> up as a decrease in CPU utilization.
>>>
>>> Also, if you run the test via 'perf stat -a ./test.sh' you should 
>>> see a reduction in instructions executed:
>>>
>>> aldebaran:~/linux/linux> perf stat -a sleep 1
>>>
>>>  Performance counter stats for 'sleep':
>>>
>>>    16128.045994  task clock ticks     (msecs)
>>>           12876  context switches     (events)
>>>             219  CPU migrations       (events)
>>>          186144  pagefaults           (events)
>>>     20911802763  CPU cycles           (events)
>>>     19309416815  instructions         (events)
>>>       199608554  cache references     (events)
>>>        19990754  cache misses         (events)
>>>
>>>  Wall-clock time elapsed:  1008.882282 msecs
>>>
>>> With -a it's measured system-wide, from start of test to end of test 
>>> - the results will be a lot more stable (and relevant) statistically 
>>> than wall-clock time or CPU usage measurements. (both of which are 
>>> rather imprecise in general)
>> I tried this perf stuff and got strange results on a cpu burning 
>> bench, saturating my 8 cpus with a "while (1) ;" loop
>>
>>
>> # perf stat -a sleep 10
>>
>>  Performance counter stats for 'sleep':
>>
>>    80334.709038  task clock ticks     (msecs)
>>           80638  context switches     (events)
>>               4  CPU migrations       (events)
>>             468  pagefaults           (events)
>>    160694681969  CPU cycles           (events)
>>    160127154810  instructions         (events)
>>          686393  cache references     (events)
>>          230117  cache misses         (events)
>>
>>  Wall-clock time elapsed: 10041.531644 msecs
>>
>> So its about 16069468196 cycles per second for 8 cpus
>> Divide by 8 to get 2008683524 cycles per second per cpu,
>> which is not       3000000000  (E5450  @ 3.00GHz)
> 
> What does "perf stat -l -a sleep 10" show? I suspect your counters 
> are scaled by about 67%, due to counter over-commit. -l will show 
> the scaling factor (and will scale up the results).

Only difference I see with '-l' is cache misses not counted.

(tbench 8 running, so not one instruction per cycle)

# perf stat -l -a sleep 10

 Performance counter stats for 'sleep':

   80007.128844  task clock ticks     (msecs)
        6754642  context switches     (events)
              2  CPU migrations       (events)
            474  pagefaults           (events)
   160925719143  CPU cycles           (events)
   108482003620  instructions         (events)
     7584035056  cache references     (events)
  <not counted>  cache misses

 Wall-clock time elapsed: 10000.595448 msecs

# perf stat -a sleep 10

 Performance counter stats for 'sleep':

   80702.908287  task clock ticks     (msecs)
        6792588  context switches     (events)
             24  CPU migrations       (events)
           4867  pagefaults           (events)
   161957342744  CPU cycles           (events)
   109147553984  instructions         (events)
     7633190481  cache references     (events)
       22996234  cache misses         (events)

 Wall-clock time elapsed: 10087.502391 msecs



quoted text> 
> If so then i think this behavior is confusing, and i'll make -l 
> default-enabled. (in fact i just committed this change to latest 
> -tip and pushed it out)
> 
> To get only instructions and cycles, do:
> 
>    perf stat -e instructions -e cycles
> 

# perf stat -e instructions -e cycles -a sleep 10

 Performance counter stats for 'sleep':

   109469842392  instructions         (events)
   162012922122  CPU cycles           (events)

 Wall-clock time elapsed: 10124.943544 msecs

I am wondering if cpus are not running at 2 GHz ;)


quoted text>> It seems strange a "jmp myself" uses one unhalted cycle per 
>> instruction and 0.5 halted cycle ...
>>
>> Also, after using "perf stat", tbench results are 1778 MB/S 
>> instead of 2610 MB/s. Even if no perf stat running.
> 
> Hm, that would be a bug. Could you send the dmesg output of:
> 
>    echo p > /proc/sysrq-trigger 
>    echo p > /proc/sysrq-trigger 
> 
> with counters running it will show something like:
> 
> [  868.105712] SysRq : Show Regs
> [  868.106544] 
> [  868.106544] CPU#1: ctrl:       ffffffffffffffff
> [  868.106544] CPU#1: status:     0000000000000000
> [  868.106544] CPU#1: overflow:   0000000000000000
> [  868.106544] CPU#1: fixed:      0000000000000000
> [  868.106544] CPU#1: used:       0000000000000000
> [  868.106544] CPU#1:   gen-PMC0 ctrl:  00000000001300c0
> [  868.106544] CPU#1:   gen-PMC0 count: 000000ffee889194
> [  868.106544] CPU#1:   gen-PMC0 left:  0000000011e1791a
> [  868.106544] CPU#1:   gen-PMC1 ctrl:  000000000013003c
> [  868.106544] CPU#1:   gen-PMC1 count: 000000ffd2542438
> [  868.106544] CPU#1:   gen-PMC1 left:  000000002dd17a8e

They stay fix (but only CPU#0 is displayed)

Is perf able to display per cpu counters, and not aggregated values ?

[ 7894.426787] CPU#0: ctrl:       ffffffffffffffff
[ 7894.426788] CPU#0: status:     0000000000000000
[ 7894.426790] CPU#0: overflow:   0000000000000000
[ 7894.426792] CPU#0: fixed:      0000000000000000
[ 7894.426793] CPU#0: used:       0000000000000000
[ 7894.426796] CPU#0:   gen-PMC0 ctrl:  0000000000134f2e
[ 7894.426798] CPU#0:   gen-PMC0 count: 000000ffb91e31e1
[ 7894.426799] CPU#0:   gen-PMC0 left:  000000007fffffff
[ 7894.426802] CPU#0:   gen-PMC1 ctrl:  000000000013412e
[ 7894.426804] CPU#0:   gen-PMC1 count: 000000ff80312b23
[ 7894.426805] CPU#0:   gen-PMC1 left:  000000007fffffff
[ 7894.426807] CPU#0: fixed-PMC0 count: 000000ffacf54a68
[ 7894.426809] CPU#0: fixed-PMC1 count: 000000ffb71cfe02
[ 7894.426811] CPU#0: fixed-PMC2 count: 0000000000000000
[ 7905.522262] SysRq : Show Regs
[ 7905.522277]
[ 7905.522279] CPU#0: ctrl:       ffffffffffffffff
[ 7905.522281] CPU#0: status:     0000000000000000
[ 7905.522283] CPU#0: overflow:   0000000000000000
[ 7905.522284] CPU#0: fixed:      0000000000000000
[ 7905.522286] CPU#0: used:       0000000000000000
[ 7905.522288] CPU#0:   gen-PMC0 ctrl:  0000000000134f2e
[ 7905.522290] CPU#0:   gen-PMC0 count: 000000ffb91e31e1
[ 7905.522292] CPU#0:   gen-PMC0 left:  000000007fffffff
[ 7905.522294] CPU#0:   gen-PMC1 ctrl:  000000000013412e
[ 7905.522296] CPU#0:   gen-PMC1 count: 000000ff80312b23
[ 7905.522298] CPU#0:   gen-PMC1 left:  000000007fffffff
[ 7905.522299] CPU#0: fixed-PMC0 count: 000000ffacf54a68
[ 7905.522301] CPU#0: fixed-PMC1 count: 000000ffb71cfe02
[ 7905.522303] CPU#0: fixed-PMC2 count: 0000000000000000


quoted text> 
> the counts should stay put (i.e. all counters should be disabled). 
> If they move around - despite there being no 'perf stat -a' session 
> running, that would be a bug.

I rebooted my machine then got good results

# perf stat -e instructions -e cycles -a sleep 10

 Performance counter stats for 'sleep':

   240021659058  instructions         (events)
   240997984836  CPU cycles           (events) << OK >>

 Wall-clock time elapsed: 10041.499326 msecs

But if I use plain "perf stat -a sleep 10"
it seems I get wrong values again (16 G cycles/sec) for all next perf sessions


# perf stat -a sleep 10

 Performance counter stats for 'sleep':

   80332.718661  task clock ticks     (msecs)
          80602  context switches     (events)
              4  CPU migrations       (events)
            473  pagefaults           (events)
   160665397757  CPU cycles           (events)  << bad >>
   160079277974  instructions         (events)
         793665  cache references     (events)
         226829  cache misses         (events)

 Wall-clock time elapsed: 10041.302969 msecs

# perf stat -e cycles -a sleep 10

 Performance counter stats for 'sleep':

   160665176575  CPU cycles           (events)  << bad >>

 Wall-clock time elapsed: 10041.491503 msecs


quoted text> 
> Also, the overhead might be profile-able, via:
> 
> 	perf record -m 1024 sleep 10
> 
> (this records the profile into output.perf.)
> 
> followed by:
> 
> 	./perf-report | tail -20
> 
> to display a histogram, with kernel-space and user-space symbols 
> mixed into a single profile.
> 
> (Pick up latest -tip to get perf-report built by default.)

Thanks, this is what I use currently

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:

udp ping pong with various process bindings (and correct cpu..., Christoph Lameter, (Fri Apr 24, 4:10 pm)

[PATCH] net: Avoid extra wakeups of threads blocked in wait_..., Eric Dumazet, (Sat Apr 25, 11:47 am)

Re: [PATCH] net: Avoid extra wakeups of threads blocked in w..., David Miller, (Tue Apr 28, 5:26 am)

Re: [PATCH] net: Avoid extra wakeups of threads blocked in w..., David Miller, (Sun Apr 26, 5:04 am)

[PATCH] poll: Avoid extra wakeups, Eric Dumazet, (Sun Apr 26, 6:46 am)

Re: [PATCH] poll: Avoid extra wakeups, Andi Kleen, (Tue Apr 28, 10:21 am)

[PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 11:06 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Wed Apr 29, 5:16 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Wed Apr 29, 5:36 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Wed Apr 29, 6:27 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Wed Apr 29, 8:29 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Davide Libenzi, (Wed Apr 29, 11:53 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Wed Apr 29, 9:07 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Andrew Morton, (Wed Apr 29, 3:20 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Wed Apr 29, 3:39 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Wed Apr 29, 4:26 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Andi Kleen, (Wed Apr 29, 3:35 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Wed Apr 29, 5:22 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Wed Apr 29, 3:37 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Christoph Lameter, (Tue Apr 28, 3:05 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 4:05 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Christoph Lameter, (Tue Apr 28, 4:14 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 4:33 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Christoph Lameter, (Tue Apr 28, 4:49 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 5:04 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 5:11 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Wed Apr 29, 5:11 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Thu Apr 30, 6:49 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Ingo Molnar, (Thu Apr 30, 7:57 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Thu Apr 30, 10:08 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Paul E. McKenney, (Thu Apr 30, 5:24 pm)

[BUG] perf_counter: change cpu frequencies , Eric Dumazet, (Thu Apr 30, 12:07 pm)

Re: [BUG] perf_counter: change cpu frequencies, Eric Dumazet, (Sun May 3, 2:06 am)

Re: [BUG] perf_counter: change cpu frequencies, Ingo Molnar, (Sun May 3, 3:25 am)

Re: [BUG] perf_counter: change cpu frequencies, Eric Dumazet, (Mon May 4, 6:39 am)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Christoph Lameter, (Tue Apr 28, 5:00 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Eric Dumazet, (Tue Apr 28, 5:05 pm)

Re: [PATCH] poll: Avoid extra wakeups in select/poll, Christoph Lameter, (Tue Apr 28, 5:04 pm)

Re: [PATCH] poll: Avoid extra wakeups, Eric Dumazet, (Tue Apr 28, 10:58 am)

Re: [PATCH] poll: Avoid extra wakeups, David Miller, (Tue Apr 28, 5:15 am)

Re: [PATCH] poll: Avoid extra wakeups, Eric Dumazet, (Tue Apr 28, 5:24 am)

Re: [PATCH] poll: Avoid extra wakeups, Jarek Poplawski, (Sun Apr 26, 9:33 am)

Re: [PATCH] poll: Avoid extra wakeups, Eric Dumazet, (Sun Apr 26, 10:27 am)

Re: udp ping pong with various process bindings (and correct..., Eric Dumazet, (Fri Apr 24, 5:18 pm)


linux-kernel:
Linus Torvalds	Linux 2.6.27-rc8
Greg KH	[GIT PATCH] driver core patches against 2.6.24
Linus Torvalds	Linux 2.6.20-rc6
Mike Snitzer	Re: Distributed storage.
git:

linux-netdev:
Gerrit Renker	[PATCH 03/37] dccp: List management for new feature negotiation
Jarek Poplawski	[PATCH] pkt_sched: Destroy gen estimators under rtnl_lock().
David Miller	[GIT]: Networking
Herbert Xu	Re: Kernel oops with 2.6.26, padlock and ipsec: probably problem with fpu state ch...
openbsd-misc:

Re: [PATCH] poll: Avoid extra wakeups in select/poll

Online users