* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:

quoted text
> > > Hmm, readfds is NULL isn't it?!? Are you sure you straced the 
> > > right process?
> > 
> > yes, i'm stracing the task that is hung unexpectedly.
> 
> But that wasn't the receiving process? (I didn't quickly find into 
> which direction distcc ports go, so I couldn't confirm this). If you 
> still have that situation at hand, could you check which is the 
> receiving process (e.g., using netstat -p, the end which has Recv-Q is 
> the right one) and where it's stuck?

it wasnt the receiving process. There's no receiving process - which is 
weird:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0 207232 europe:37198                europe:distcc               ESTABLISHED 19578/distcc        
tcp        0      0 europe:ssh                  dione:36284                 ESTABLISHED -                   
tcp        0      0 europe:ssh                  e2:45910                    ESTABLISHED -                   
tcp    72283      0 europe:distcc               europe:37198                ESTABLISHED -                   

quoted text
> > i mean, sometimes a TCP connection can get 'unstuck' if you strace a 
> > task - that is because the TCP related syscall the task sits in gets 
> > interrupted. But in this case it's select() which doesnt explicitly 
> > take the socket, doesnt do any tcp_push_pending_frames() processing, 
> > etc. - it just its on the socket waitqueue AFAICS. And that's 
> > expected.
> 
> This is not in the sender end at all. It's correct behavior of the 
> flow control to stop the sender until more room is made available by 
> the reading end. Thus push_pending_frames couldn't send anything.

i just gave it as a general example of why sometimes stracing a task can 
'disturb' the observed system and can kick the TCP state machine out of 
a stall. I did not say it's occuring here.

quoted text
> ...It may still be that the receiving process is stuck due to the 
> non-net related changes you have there.

the socket does not seem to be owned. It should have closed down? 
Refcounting issue?

find below the sysrq-t dump.

	Ingo

-------------------------->
[11811.540000] SysRq : Show State
[11811.540000]   task                PC stack   pid father
[11811.540000] init          S f7c67b04     0     1      0
[11811.540000]        f7c70000 00000086 00000002 f7c67b04 f7c67b0c 00000000 00000046 f7ca08b8 
[11811.540000]        f7c67b2c 00119272 00000002 80130c53 00000000 01ca0000 f7c67b08 f7c70164 
[11811.540000]        82cbb080 0011907f 00000000 00000000 00000000 00000000 000000ff f7c67b2c 
[11811.540000] Call Trace:
[11811.540000]  [<80130c53>] internal_add_timer+0x103/0x110
[11811.540000]  [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540000]  [<801310e0>] process_timeout+0x0/0x40
[11811.540000]  [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540000]  [<8019868c>] do_select+0x3dc/0x500
[11811.540000]  [<801982b0>] do_select+0x0/0x500
[11811.540000]  [<80198d60>] __pollwait+0x0/0xe0
[11811.540000]  [<80120070>] default_wake_function+0x0/0x10
[11811.540000]  [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540000]  [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540000]  [<80185695>] check_object+0xe5/0x1f0
[11811.540000]  [<80185e08>] __slab_free+0x238/0x2e0
[11811.540000]  [<801202e4>] __wake_up+0x24/0x60
[11811.540000]  [<801e6717>] journal_stop+0x1a7/0x2a0
[11811.540000]  [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540000]  [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540000]  [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540000]  [<801987d1>] core_sys_select+0x21/0x350
[11811.540000]  [<802b0e1c>] selinux_inode_getattr+0x5c/0x70
[11811.540000]  [<80185e08>] __slab_free+0x238/0x2e0
[11811.540000]  [<802e62af>] copy_to_user+0x3f/0x70
[11811.540000]  [<8018e3e9>] cp_new_stat64+0xf9/0x110
[11811.540000]  [<80198f27>] sys_select+0xe7/0x1b0
[11811.540000]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540000]  =======================
[11811.540000] kthreadd      S f7c69fac     0     2      0
[11811.540000]        f7c70ed0 00000096 00000002 f7c69fac f7c69fb4 00000000 f7ca8000 00000003 
[11811.540000]        00000246 00000000 00000000 808e2c50 00000000 01000246 f7c69fb0 f7c71034 
[11811.540000]        82c68080 ffffcb74 00000000 00000000 00000000 00000000 000000ff 808e2c20 
[11811.540000] Call Trace:
[11811.540000]  [<8013b551>] kthreadd+0x151/0x160
[11811.540000]  [<8013b400>] kthreadd+0x0/0x160
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] migration/0   S 82c68090     0     3      2
[11811.540000]        f7c71da0 00000086 00000000 82c68090 f7c71da0 00000046 00000000 00000001 
[11811.540000]        00000046 00000000 00000000 82c68090 00000000 01000046 00000046 f7c71f04 
[11811.540000]        82c68080 82c68090 00000002 806b1e00 00000001 82c68080 82c6d620 82c687f8 
[11811.540000] Call Trace:
[11811.540000]  [<80124931>] migration_thread+0x141/0x220
[11811.540000]  [<801247f0>] migration_thread+0x0/0x220
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ksoftirqd/0   S f7c6bfa4     0     4      2
[11811.540000]        f7c72c70 00000082 00000002 f7c6bfa4 f7c6bfac 00000000 f7c6b000 802e5b64 
[11811.540000]        02223000 00000000 808dd400 80103f3c 00000000 01c68080 f7c6bfa8 f7c72dd4 
[11811.540000]        82c68080 00119094 00000000 00000000 00000000 00000000 000000ff 00000000 
[11811.540000] Call Trace:
[11811.540000]  [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540000]  [<80103f3c>] restore_nocheck_notrace+0x0/0xe
[11811.540000]  [<8012dbd5>] ksoftirqd+0xf5/0x100
[11811.540000]  [<8012dae0>] ksoftirqd+0x0/0x100
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] watchdog/0    S f7c6cf8c     0     5      2
[11811.540000]        f7c73b40 00000092 00000002 f7c6cf8c f7c6cf94 00000000 0011908c 00979c85 
[11811.540000]        00000abe 82c68880 ffff0600 00000abd 00000000 01140dab f7c6cf90 f7c73ca4 
[11811.540000]        82c68080 0011908d 00000000 00000000 00000000 00000000 000000ff eb400ed0 
[11811.540000] Call Trace:
[11811.540000]  [<8015d375>] watchdog+0x65/0x1d0
[11811.540000]  [<8015d310>] watchdog+0x0/0x1d0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] migration/1   S 82cbb090     0     6      2
[11811.540000]        f7c74a10 00000086 00000000 82cbb090 f7c74a10 00000046 00000000 00000001 
[11811.540000]        00000046 00000000 00000000 82cbb090 00000000 01000046 00000046 f7c74b74 
[11811.540000]        82cbb080 82cbb090 00000002 00000000 00000001 82cbb080 82c1a620 82cbb7f8 
[11811.540000] Call Trace:
[11811.540000]  [<80124931>] migration_thread+0x141/0x220
[11811.540000]  [<801247f0>] migration_thread+0x0/0x220
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ksoftirqd/1   S f7c7afa4     0     7      2
[11811.540000]        f7c758e0 00000082 00000002 f7c7afa4 f7c7afac 00000000 80a44e80 80a44e80 
[11811.540000]        f7c7a000 802e5b64 f7ca8000 00000001 00000000 01103f3c f7c7afa8 f7c75a44 
[11811.540000]        82cbb080 00119094 00000000 00000000 00000000 00000000 000000ff 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540000]  [<8012dbd5>] ksoftirqd+0xf5/0x100
[11811.540000]  [<8012dae0>] ksoftirqd+0x0/0x100
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] watchdog/1    S f7c7bf8c     0     8      2
[11811.540000]        f7c767b0 00000092 00000002 f7c7bf8c f7c7bf94 00000000 0011908d 0097de40 
[11811.540000]        00000abe 82cbb880 00979c80 00000abe 00000000 01140dab f7c7bf90 f7c76914 
[11811.540000]        82cbb080 0011908e 00000000 00000000 00000000 00000000 000000ff fffffffc 
[11811.540000] Call Trace:
[11811.540000]  [<8015d375>] watchdog+0x65/0x1d0
[11811.540000]  [<8015d310>] watchdog+0x0/0x1d0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] events/0      S f7c7ef90     0     9      2
[11811.540000]        f7ca8ed0 00000096 00000002 f7c7ef90 f7c7ef98 00000000 f7c11d48 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7c7ef94 f7ca9034 
[11811.540000]        82c68080 00119084 00000000 00000000 00000000 00000000 000000ff f7c11d10 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] events/1      S f7c7ff90     0    10      2
[11811.540000]        f7ca9da0 00000096 00000002 f7c7ff90 f7c7ff98 00000000 f7c11df8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7c7ff94 f7ca9f04 
[11811.540000]        82cbb080 001190a7 00000000 00000000 00000000 00000000 000000ff f7c11dc0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] khelper       S 82c1a620     0    11      2
[11811.540000]        f7caac70 00000096 00000afb 82c1a620 00000002 f7c88f8c f7c11ea8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7c88f94 f7caadd4 
[11811.540000]        82c68080 00114d91 00000000 00000000 00000000 00000afb 000000ff f7c11e70 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kblockd/0     S 00000046     0   183      2
[11811.540000]        f7cabb40 00000096 00000046 00000046 00000000 00000000 f7c8d508 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cabca4 
[11811.540000]        82c68080 00000002 00000001 00000001 f7c8d4f8 00000286 801388e0 f7c8d4d0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kblockd/1     S 00000046     0   184      2
[11811.540000]        f7caca10 00000096 00000046 00000046 00000000 00000000 f7c8d5b8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cacb74 
[11811.540000]        82cbb080 00000002 00000001 00000001 f7c8d5a8 00000286 801388e0 f7c8d580 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kacpid        S 00000046     0   185      2
[11811.540000]        f7cad8e0 00000096 00000046 00000046 00000000 00000000 f7c8d668 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cada44 
[11811.540000]        82cbb080 00000000 00000001 00000001 f7c8d658 00000286 801388e0 f7c8d630 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  [<8063e9ec>] ip6_mc_add_src+0x7c/0x2c0
[11811.540000]  =======================
[11811.540000] kacpi_notify  S f7ce5f90     0   186      2
[11811.540000]        f7cae7b0 00000096 00000002 f7ce5f90 f7ce5f98 00000000 f7c8d7c8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7ce5f94 f7cae914 
[11811.540000]        82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff f7c8d790 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ata/0         S f7d7cf90     0   276      2
[11811.540000]        f7db0000 00000096 00000002 f7d7cf90 f7d7cf98 00000000 f7d47b38 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7d7cf94 f7db0164 
[11811.540000]        82c68080 ffff8df0 00000000 00000000 00000000 00000000 000000ff f7d47b00 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ata/1         S 00000046     0   277      2
[11811.540000]        f7db0ed0 00000096 00000046 00000046 00000000 00000000 f7d47be8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7db1034 
[11811.540000]        82cbb080 00000002 00000001 00000001 f7d47bd8 00000286 801388e0 f7d47bb0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ata_aux       S f7d7ef90     0   278      2
[11811.540000]        f7db1da0 00000096 00000002 f7d7ef90 f7d7ef98 00000000 f7d47c98 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7d7ef94 f7db1f04 
[11811.540000]        82cbb080 ffff8db2 00000000 00000000 00000000 00000000 000000ff f7d47c60 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ksuspend_usbd S 00000046     0   280      2
[11811.540000]        f7db2c70 00000096 00000046 00000046 00000000 00000000 f7d47df8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7db2dd4 
[11811.540000]        82cbb080 00000002 00000001 00000001 f7d47de8 00000286 801388e0 f7d47dc0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] khubd         S 00000046     0   286      2
[11811.540000]        f7db3b40 00000086 00000046 00000046 00000000 00000000 80918b70 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000292 f7db3ca4 
[11811.540000]        82c68080 00000002 00000001 00000001 80918b60 00000292 f6ae3810 80918b40 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<804ecdcd>] hub_thread+0x1dd/0x1060
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<804ecbf0>] hub_thread+0x0/0x1060
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kseriod       S f7d84f80     0   289      2
[11811.540000]        f7db4a10 00000092 00000002 f7d84f80 f7d84f88 00000000 8094f8b0 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f7d84f84 f7db4b74 
[11811.540000]        82c68080 ffff8e59 00000000 00000000 00000000 00000000 000000ff 8094f880 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<805292b8>] serio_thread+0xa8/0x2d0
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<80529210>] serio_thread+0x0/0x2d0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kondemand/0   S f6dbef90     0   316      2
[11811.540000]        f7db58e0 00000096 00000002 f6dbef90 f6dbef98 00000000 f7d85b38 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6dbef94 f7db5a44 
[11811.540000]        82c68080 ffffcc75 00000000 00000000 00000000 00000000 000000ff f7d85b00 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kondemand/1   S f6dbff90     0   317      2
[11811.540000]        f7db67b0 00000096 00000002 f6dbff90 f6dbff98 00000000 f7d85be8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6dbff94 f7db6914 
[11811.540000]        82cbb080 ffffcc75 00000000 00000000 00000000 00000000 000000ff f7d85bb0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] pageattr-test S f67fdef0     0   340      2
[11811.540000]        f7cb0ed0 00000092 00000002 f67fdef0 f67fdef8 00000000 00000046 80b6a678 
[11811.540000]        f67fdf18 00119af6 00000002 80130c53 00000000 01b69d80 f67fdef4 f7cb1034 
[11811.540000]        82c68080 00118f3b 00000000 00000000 00000000 00000000 000000ff f67fdf18 
[11811.540000] Call Trace:
[11811.540000]  [<80130c53>] internal_add_timer+0x103/0x110
[11811.540000]  [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540000]  [<8011ae25>] print_split+0x85/0x1b0
[11811.540000]  [<8011b7c8>] change_page_attr_set_clr+0xc8/0x190
[11811.540000]  [<801310e0>] process_timeout+0x0/0x40
[11811.540000]  [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540000]  [<8011b8b0>] do_pageattr_test+0x20/0x4e0
[11811.540000]  [<8011b890>] do_pageattr_test+0x0/0x4e0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-0     S f7cb67b0     0   346      2
[11811.540000]        f7cb67b0 00000096 00000000 f7cb67b0 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df0a94 00000046 00000046 00000046 00000000 0112c7fe f6df0a94 f7cb6914 
[11811.540000]        82c68080 00000001 f6df0a84 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-1     S f7cb58e0     0   348      2
[11811.540000]        f7cb58e0 00000096 00000000 f7cb58e0 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df1014 00000046 00000046 00000046 00000000 0112c7fe f6df1014 f7cb5a44 
[11811.540000]        82c68080 00000001 f6df1004 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-2     S f7cb4a10     0   350      2
[11811.540000]        f7cb4a10 00000096 00000000 f7cb4a10 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df1594 00000046 00000046 00000046 00000000 0112c7fe f6df1594 f7cb4b74 
[11811.540000]        82c68080 00000001 f6df1584 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-3     S f7cb3b40     0   352      2
[11811.540000]        f7cb3b40 00000096 00000000 f7cb3b40 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df1b14 00000046 00000046 00000046 00000000 0112c7fe f6df1b14 f7cb3ca4 
[11811.540000]        82c68080 00000001 f6df1b04 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-4     S f7cb2c70     0   354      2
[11811.540000]        f7cb2c70 00000096 00000000 f7cb2c70 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df2094 00000046 00000046 00000046 00000000 0112c7fe f6df2094 f7cb2dd4 
[11811.540000]        82c68080 00000001 f6df2084 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-5     S f7cb1da0     0   356      2
[11811.540000]        f7cb1da0 00000096 00000000 f7cb1da0 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df2614 00000046 00000046 00000046 00000000 0112c7fe f6df2614 f7cb1f04 
[11811.540000]        82c68080 00000001 f6df2604 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-6     S f7cb0000     0   358      2
[11811.540000]        f7cb0000 00000096 00000000 f7cb0000 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df2b94 00000046 00000046 00000046 00000000 0112c7fe f6df2b94 f7cb0164 
[11811.540000]        82c68080 00000001 f6df2b84 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] rt-test-7     S f6df8000     0   360      2
[11811.540000]        f6df8000 00000096 00000000 f6df8000 806ab08b 00000046 00000000 00000000 
[11811.540000]        f6df3114 00000046 00000046 00000046 00000000 0112c7fe f6df3114 f6df8164 
[11811.540000]        82c68080 00000001 f6df3104 00000000 8014cef0 00000000 806ad940 00000001 
[11811.540000] Call Trace:
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000]  [<8014cef0>] test_func+0x0/0x90
[11811.540000]  [<8014cf49>] test_func+0x59/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] pdflush       S f6743f90     0   362      2
[11811.540000]        f6df8ed0 00000096 00000002 f6743f90 f6743f98 00000000 f6df8ed0 80124aa3 
[11811.540000]        00000046 00000000 00000000 808e6db0 00000000 01000046 f6743f94 f6df9034 
[11811.540000]        82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff f6743fc0 
[11811.540000] Call Trace:
[11811.540000]  [<80124aa3>] set_cpus_allowed_ptr+0x53/0xd0
[11811.540000]  [<8016f790>] pdflush+0x0/0x1a0
[11811.540000]  [<8016f842>] pdflush+0xb2/0x1a0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] pdflush       S f6751f90     0   363      2
[11811.540000]        f6df9da0 00000096 00000002 f6751f90 f6751f98 00000000 00000000 00000296 
[11811.540000]        00000046 00000000 00000000 808e6db0 00000000 01000046 f6751f94 f6df9f04 
[11811.540000]        82cbb080 00118f23 00000000 00000000 00000000 00000000 000000ff f6751fc0 
[11811.540000] Call Trace:
[11811.540000]  [<8016f790>] pdflush+0x0/0x1a0
[11811.540000]  [<8016f842>] pdflush+0xb2/0x1a0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] kswapd0       S f67f9f28     0   364      2
[11811.540000]        f6dfac70 00000082 00000002 f67f9f28 f67f9f30 00000000 808e6c54 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f67f9f2c f6dfadd4 
[11811.540000]        82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff 808e4c00 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<80172c80>] kswapd+0x0/0x400
[11811.540000]  [<8017304a>] kswapd+0x3ca/0x400
[11811.540000]  [<806ab08b>] schedule+0x25b/0x670
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<80172c80>] kswapd+0x0/0x400
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] aio/0         S 00000046     0   451      2
[11811.540000]        f6dfbb40 00000096 00000046 00000046 00000000 00000000 f6dc59d8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6dfbca4 
[11811.540000]        82c68080 00000002 00000001 00000001 f6dc59c8 00000286 801388e0 f6dc59a0 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] aio/1         S 00000046     0   452      2
[11811.540000]        f6dfca10 00000096 00000046 00000046 00000000 00000000 f6dc5a88 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6dfcb74 
[11811.540000]        82cbb080 00000002 00000001 00000001 f6dc5a78 00000286 801388e0 f6dc5a50 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] jfsIO         S 808ea3d0     0   476      2
[11811.540000]        f6dfd8e0 00000092 00000000 808ea3d0 f6dfd8e0 00000002 82c68080 f7c70ed0 
[11811.540000]        00000046 00000000 00000000 808ea3d0 00000000 01000046 00000046 f6dfda44 
[11811.540000]        82c68080 808ea3d0 00000002 80248770 00000001 808ea3c0 00000000 00000000 
[11811.540000] Call Trace:
[11811.540000]  [<80248770>] jfsIOWait+0x0/0x70
[11811.540000]  [<80248770>] jfsIOWait+0x0/0x70
[11811.540000]  [<802487c4>] jfsIOWait+0x54/0x70
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] jfsCommit     S 00000046     0   477      2
[11811.540000]        f6dfe7b0 00000092 80d7d174 00000046 00000046 00000000 00000000 808ea490 
[11811.540000]        00000046 00000000 00000001 00000001 00000000 01000046 808ea480 f6dfe914 
[11811.540000]        82c68080 00000002 00000000 00000001 80d7d164 00000286 8024c150 fffffffc 
[11811.540000] Call Trace:
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8024c304>] jfs_lazycommit+0x1b4/0x200
[11811.540000]  [<80120070>] default_wake_function+0x0/0x10
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] jfsCommit     S 00000046     0   478      2
[11811.540000]        f6780000 00000092 80d7d174 00000046 00000046 00000000 00000000 808ea490 
[11811.540000]        00000046 00000000 00000001 00000001 00000000 01000046 808ea480 f6780164 
[11811.540000]        82c68080 00000002 00000000 00000001 80d7d164 00000286 8024c150 fffffffc 
[11811.540000] Call Trace:
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8024c304>] jfs_lazycommit+0x1b4/0x200
[11811.540000]  [<80120070>] default_wake_function+0x0/0x10
[11811.540000]  [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] jfsSync       S f6776f9c     0   479      2
[11811.540000]        f6780ed0 00000096 00000002 f6776f9c f6776fa4 00000000 00000000 f6780ed0 
[11811.540000]        00000246 00000000 00000000 808ea470 00000000 01000246 f6776fa0 f6781034 
[11811.540000]        82c68080 ffff8b4b 00000000 00000000 00000000 00000000 000000ff fffffffc 
[11811.540000] Call Trace:
[11811.540000]  [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000]  [<8024b49a>] jfs_sync+0x12a/0x1c0
[11811.540000]  [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000]  [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] ocfs2_wq      S 00000046     0   482      2
[11811.540000]        f6781da0 00000096 00000046 00000046 00000000 00000000 f6773c98 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6781f04 
[11811.540000]        82c68080 00000002 00000001 00000001 f6773c88 00000286 801388e0 f6773c60 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] fakephp       S 00000046     0   488      2
[11811.540000]        f6782c70 00000096 00000046 00000046 00000000 00000000 f676b5b8 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6782dd4 
[11811.540000]        82c68080 00000002 00000001 00000001 f676b5a8 00000286 801388e0 f676b580 
[11811.540000] Call Trace:
[11811.540000]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540000]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000]  [<801388e0>] worker_thread+0x0/0x90
[11811.540000]  [<8013b3d2>] kthread+0x42/0x70
[11811.540000]  [<8013b390>] kthread+0x0/0x70
[11811.540000]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000]  =======================
[11811.540000] netxen        S f6e9ff90     0  1672      2
[11811.540000]        f6783b40 00000096 00000002 f6e9ff90 f6e9ff98 00000000 f6f20b38 00000046 
[11811.540000]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6e9ff94 f6783ca4 
[11811.540001]        82cbb080 ffff8d29 00000000 00000000 00000000 00000000 000000ff f6f20b00 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] sfc_refill/0  S f67b1f90     0  1675      2
[11811.540001]        f6784a10 00000096 00000002 f67b1f90 f67b1f98 00000000 f6f20c98 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f67b1f94 f6784b74 
[11811.540001]        82c68080 ffff8d1f 00000000 00000000 00000000 00000000 000000ff f6f20c60 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] sfc_refill/1  S f6ea6f90     0  1676      2
[11811.540001]        f67858e0 00000096 00000002 f6ea6f90 f6ea6f98 00000000 f6f20d48 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6ea6f94 f6785a44 
[11811.540001]        82cbb080 ffff8d29 00000000 00000000 00000000 00000000 000000ff f6f20d10 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_0     S 00000000     0  1697      2
[11811.540001]        f67867b0 00000086 00000046 00000000 00000046 00000001 8048944d f6fe8884 
[11811.540001]        00000002 00000286 00000001 f6fe8874 00000000 01fe8830 00000000 f6786914 
[11811.540001]        82c68080 00000000 80489479 00000286 00000292 f6fe8830 8048d230 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<80489479>] __scsi_iterate_devices+0x49/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_1     S f6774f64     0  1701      2
[11811.540001]        f6a88000 00000086 00000002 f6774f64 f6774f6c 00000000 8048944d f6fe90b4 
[11811.540001]        00000002 00000286 00000001 f6fe90a4 00000000 01fe9060 f6774f68 f6a88164 
[11811.540001]        82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_2     S f67b7f64     0  1703      2
[11811.540001]        f6a88ed0 00000086 00000002 f67b7f64 f67b7f6c 00000000 8048944d f6fe98e4 
[11811.540001]        00000002 00000286 00000001 f6fe98d4 00000000 01fe9890 f67b7f68 f6a89034 
[11811.540001]        82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_3     S f6772f64     0  1705      2
[11811.540001]        f6a89da0 00000086 00000002 f6772f64 f6772f6c 00000000 8048944d f6fea114 
[11811.540001]        00000002 00000286 00000001 f6fea104 00000000 01fea0c0 f6772f68 f6a89f04 
[11811.540001]        82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_4     S 00000000     0  1716      2
[11811.540001]        f6a8ac70 00000086 00000046 00000000 00000046 00000001 8048944d f6feb174 
[11811.540001]        00000002 00000286 00000001 f6feb164 00000000 01feb120 f6fec180 f6a8add4 
[11811.540001]        82c68080 f6fec188 80489484 00000286 f6fec180 f6feb120 8048d230 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<80489484>] __scsi_iterate_devices+0x54/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] scsi_eh_5     S f6f3af64     0  1718      2
[11811.540001]        f6a8bb40 00000086 00000002 f6f3af64 f6f3af6c 00000000 8048944d f6feb9a4 
[11811.540001]        00000002 00000286 00000001 f6feb994 00000000 01feb950 f6f3af68 f6a8bca4 
[11811.540001]        82cbb080 ffff8def 00000000 00000000 00000000 00000000 000000ff 00000292 
[11811.540001] Call Trace:
[11811.540001]  [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001]  [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] w1_control    S f6a09f4c     0  1852      2
[11811.540001]        f6a8ca10 00000096 00000002 f6a09f4c f6a09f54 00000000 00000046 f7ca0678 
[11811.540001]        f6a09f74 001190ca 00000002 80130c53 00000000 01ca0000 f6a09f50 f6a8cb74 
[11811.540001]        82cbb080 00119066 00000000 00000000 00000000 00000000 000000ff f6a09f74 
[11811.540001] Call Trace:
[11811.540001]  [<80130c53>] internal_add_timer+0x103/0x110
[11811.540001]  [<8053f990>] w1_control+0x0/0x1c0
[11811.540001]  [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540001]  [<806ab08b>] schedule+0x25b/0x670
[11811.540001]  [<801310e0>] process_timeout+0x0/0x40
[11811.540001]  [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540001]  [<80131576>] msleep_interruptible+0x26/0x40
[11811.540001]  [<8053f9be>] w1_control+0x2e/0x1c0
[11811.540001]  [<8053f990>] w1_control+0x0/0x1c0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] kmemstick     S f6a02f90     0  1875      2
[11811.540001]        f6a8d8e0 00000096 00000002 f6a02f90 f6a02f98 00000000 f6ac5f58 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6a02f94 f6a8da44 
[11811.540001]        82cbb080 ffff8e68 00000000 00000000 00000000 00000000 000000ff f6ac5f20 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] ib_mcast      S 82c6d620     0  1881      2
[11811.540001]        f6a8e7b0 00000096 00000818 82c6d620 00000002 f6ae7f8c f6a08248 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6ae7f94 f6a8e914 
[11811.540001]        82cbb080 ffff8e68 00000000 00000000 00000000 00000818 000000ff f6a08210 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] ib_cm/0       S f6afcf90     0  1883      2
[11811.540001]        f6a20000 00000096 00000002 f6afcf90 f6afcf98 00000000 f6a083a8 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6afcf94 f6a20164 
[11811.540001]        82c68080 ffff8e59 00000000 00000000 00000000 00000000 000000ff f6a08370 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] ib_cm/1       S 82c6d620     0  1884      2
[11811.540001]        f6a20ed0 00000096 00000818 82c6d620 00000002 f6adef8c f6a08458 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6adef94 f6a21034 
[11811.540001]        82cbb080 ffff8e69 00000000 00000000 00000000 00000818 000000ff f6a08420 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] iw_cm_wq      S f6adbf90     0  1885      2
[11811.540001]        f6a21da0 00000096 00000002 f6adbf90 f6adbf98 00000000 f6a08508 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6adbf94 f6a21f04 
[11811.540001]        82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a084d0 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] ib_addr       S f6adaf90     0  1886      2
[11811.540001]        f6a22c70 00000096 00000002 f6adaf90 f6adaf98 00000000 f6a08668 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6adaf94 f6a22dd4 
[11811.540001]        82cbb080 00118f4d 00000000 00000000 00000000 00000000 000000ff f6a08630 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] rdma_cm       S f6ec2f90     0  1887      2
[11811.540001]        f6a23b40 00000096 00000002 f6ec2f90 f6ec2f98 00000000 f6a087c8 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6ec2f94 f6a23ca4 
[11811.540001]        82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a08790 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] mthca_catas   S f6ec0f90     0  1888      2
[11811.540001]        f6a24a10 00000096 00000002 f6ec0f90 f6ec0f98 00000000 f6a08928 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 f6ec0f94 f6a24b74 
[11811.540001]        82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a088f0 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013895a>] worker_thread+0x7a/0x90
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801388e0>] worker_thread+0x0/0x90
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] kirqd         S f664df20     0  1909      2
[11811.540001]        f6a258e0 00000096 00000002 f664df20 f664df28 00000000 00000046 f7ca08b0 
[11811.540001]        f664df48 00119180 00000002 80130c53 00000000 01ca0000 f664df24 f6a25a44 
[11811.540001]        82cbb080 00118f8d 00000000 00000000 00000000 00000000 000000ff f664df48 
[11811.540001] Call Trace:
[11811.540001]  [<80130c53>] internal_add_timer+0x103/0x110
[11811.540001]  [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540001]  [<801310e0>] process_timeout+0x0/0x40
[11811.540001]  [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540001]  [<80114143>] balanced_irq+0x73/0x4e0
[11811.540001]  [<801140d0>] balanced_irq+0x0/0x4e0
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] kjournald     S 00000046     0  1913      2
[11811.540001]        f6a267b0 00000082 00000046 00000046 00000000 00000000 f6afb37c 00000046 
[11811.540001]        00000000 00000001 00000001 8013b87c 00000000 01000046 00000246 f6a26914 
[11811.540001]        82cbb080 00000002 00000001 00000001 00000001 f6afb254 f6afb254 f6afb240 
[11811.540001] Call Trace:
[11811.540001]  [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001]  [<801ec9e5>] kjournald+0x2f5/0x340
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801ec6f0>] kjournald+0x0/0x340
[11811.540001]  [<8013b3d2>] kthread+0x42/0x70
[11811.540001]  [<8013b390>] kthread+0x0/0x70
[11811.540001]  [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001]  =======================
[11811.540001] udevd         S f66efb04     0  1977      1
[11811.540001]        f6a61da0 00000086 00000002 f66efb04 f66efb0c 00000000 f7824380 00001000 
[11811.540001]        00000046 00000046 00000000 f6dd1190 00000000 01000046 f66efb08 f6a61f04 
[11811.540001]        82cbb080 00114d79 00000000 00000000 00000000 00000000 000000ff 00000000 
[11811.540001] Call Trace:
[11811.540001]  [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001]  [<80190af2>] pipe_poll+0x32/0xb0
[11811.540001]  [<8019868c>] do_select+0x3dc/0x500
[11811.540001]  [<801982b0>] do_select+0x0/0x500
[11811.540001]  [<80198d60>] __pollwait+0x0/0xe0
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<801d5c3e>] ext3_get_blocks_handle+0x8e/0xa60
[11811.540001]  [<801ad1b9>] __find_get_block_slow+0x49/0x110
[11811.540001]  [<801ad1d1>] __find_get_block_slow+0x61/0x110
[11811.540001]  [<801ad55d>] __find_get_block+0x9d/0x190
[11811.540001]  [<801d5c3e>] ext3_get_blocks_handle+0x8e/0xa60
[11811.540001]  [<801e1257>] ext3fs_dirhash+0x107/0x210
[11811.540001]  [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001]  [<801987d1>] core_sys_select+0x21/0x350
[11811.540001]  [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001]  [<801202e4>] __wake_up+0x24/0x60
[11811.540001]  [<8013b8d3>] remove_wait_queue+0x13/0x40
[11811.540001]  [<8012b335>] do_wait+0x1d5/0xa80
[11811.540001]  [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540001]  [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540001]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001]  =======================
[11811.540001] rsyslogd      R running      0  5059      1
[11811.540001]        f62e4a10 00000086 00000002 808e3e00 00000002 f6e8ce20 00000046 000000df 
[11811.540001]        f6e8cae8 00000000 00000000 f784ad28 00000000 0184ad18 00000046 f62e4b74 
[11811.540001]        82cbb080 00000000 00000000 f784ad28 00000046 00000000 00000001 00000000 
[11811.540001] Call Trace:
[11811.540001]  [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001]  [<805b7e87>] datagram_poll+0x17/0x100
[11811.540001]  [<8019868c>] do_select+0x3dc/0x500
[11811.540001]  [<801982b0>] do_select+0x0/0x500
[11811.540001]  [<80198d60>] __pollwait+0x0/0xe0
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<8012098e>] hrtick_set+0x2e/0x120
[11811.540001]  [<801209c9>] hrtick_set+0x69/0x120
[11811.540001]  [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001]  [<80184f32>] slab_pad_check+0x42/0x100
[11811.540001]  [<80185695>] check_object+0xe5/0x1f0
[11811.540001]  [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001]  [<805b7f7a>] skb_free_datagram+0xa/0x30
[11811.540001]  [<80621254>] unix_dgram_recvmsg+0x184/0x2c0
[11811.540001]  [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540001]  [<805aef4c>] sock_recvmsg+0x13c/0x190
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001]  [<801987d1>] core_sys_select+0x21/0x350
[11811.540001]  [<8018c4c3>] fget_light+0x53/0xe0
[11811.540001]  [<805ae922>] sockfd_lookup_light+0x32/0x60
[11811.540001]  [<805b02d4>] sys_recvfrom+0x154/0x160
[11811.540001]  [<806ad344>] _spin_unlock+0x14/0x20
[11811.540001]  [<8014a2d6>] futex_wake+0xc6/0xf0
[11811.540001]  [<8015ed90>] handle_edge_irq+0x0/0x120
[11811.540001]  [<80107035>] do_IRQ+0x85/0xe0
[11811.540001]  [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540001]  [<80103f3c>] restore_nocheck_notrace+0x0/0xe
[11811.540001]  [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540001]  [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001]  =======================
[11811.540001] rsyslogd      R running      0  5061      1
[11811.540001]        f62e1da0 00000082 00000000 00000046 00000046 00000000 00000000 f6e75e30 
[11811.540001]        00000046 00000000 00000001 00000001 00000000 01000046 f6e75e20 f62e1f04 
[11811.540001]        82cbb080 00000002 00000000 00000001 f6e75e20 00000282 00000000 80d4c0cc 
[11811.540001] Call Trace:
[11811.540001]  [<8014a175>] futex_wait+0x345/0x3b0
[11811.540001]  [<806ad970>] _read_unlock_irq+0x20/0x30
[11811.540001]  [<80167a7d>] find_get_pages_tag+0x6d/0x80
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<8014b01c>] do_futex+0xbc/0x950
[11811.540001]  [<806ad970>] _read_unlock_irq+0x20/0x30
[11811.540001]  [<80167a7d>] find_get_pages_tag+0x6d/0x80
[11811.540001]  [<8017009a>] pagevec_lookup_tag+0x2a/0x40
[11811.540001]  [<8016844f>] wait_on_page_writeback_range+0x5f/0x110
[11811.540001]  [<806ad344>] _spin_unlock+0x14/0x20
[11811.540001]  [<801a90cd>] sync_inode+0x2d/0x40
[11811.540001]  [<8014b947>] sys_futex+0x97/0x120
[11811.540001]  [<801ac37c>] do_fsync+0x8c/0xb0
[11811.540001]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001]  =======================
[11811.540001] rklogd        R running      0  5064      1
[11811.540001]        82cbb090 00000046 00000046 00000001 00000001 8011fc6f 82cbb090 00000046 
[11811.540001]        f62e4a10 82cbb090 806ad489 00000000 82cbb080 00000002 00000001 80186499 
[11811.540001]        82cbb080 802aa1b4 00000046 00000046 00000000 00000000 f6a6c304 f6a6c000 
[11811.540001] Call Trace:
[11811.540001]  [<80185695>] check_object+0xe5/0x1f0
[11811.540001]  [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001]  [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001]  [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540001]  [<8013b810>] prepare_to_wait_exclusive+0x20/0x70
[11811.540001]  [<8061fa91>] unix_wait_for_peer+0xb1/0xc0
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<8061ff71>] unix_dgram_sendmsg+0x331/0x4f0
[11811.540001]  [<805ae7a9>] sock_aio_write+0x109/0x120
[11811.540001]  [<8018aba5>] do_sync_write+0xd5/0x120
[11811.540001]  [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<802a53ec>] security_file_permission+0xc/0x10
[11811.540001]  [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540001]  [<8018b4a9>] vfs_write+0x119/0x120
[11811.540001]  [<8018bb8a>] sys_write+0x4a/0xa0
[11811.540001]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001]  =======================
[11811.540001] dbus-daemon   S 801312f7     0  5071      1
[11811.540001]        f7cdac70 00000082 00000001 801312f7 f7ca0010 f7ca0000 00000046 f63a5bcc 
[11811.540001]        806ad9f5 00000000 f7cdbb40 f5fc8328 00000000 01ca0000 00000046 f7cdadd4 
[11811.540001]        82cbb080 00000000 00000000 f5fc8328 00000046 00000000 00000001 00000000 
[11811.540001] Call Trace:
[11811.540001]  [<801312f7>] lock_timer_base+0x27/0x60
[11811.540001]  [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540001]  [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001]  [<8061f0a7>] unix_poll+0x17/0xb0
[11811.540001]  [<80198076>] do_sys_poll+0x246/0x390
[11811.540001]  [<80198d60>] __pollwait+0x0/0xe0
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001]  [<80122852>] enqueue_task_fair+0xc2/0xe0
[11811.540001]  [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540001]  [<8011e94b>] __wake_up_common+0x4b/0x80
[11811.540001]  [<806ad2f4>] _read_unlock+0x14/0x20
[11811.540001]  [<80620f76>] unix_stream_sendmsg+0x1d6/0x330
[11811.540001]  [<805ae7a9>] sock_aio_write+0x109/0x120
[11811.540001]  [<805ae6a0>] sock_aio_write+0x0/0x120
[11811.540001]  [<8018aa8e>] do_sync_readv_writev+0xce/0x110
[11811.540001]  [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001]  [<8018b28b>] do_readv_writev+0x10b/0x170
[11811.540001]  [<805ae6a0>] sock_aio_write+0x0/0x120
[11811.540001]  [<8019d56d>] dput+0x6d/0x120
[11811.540001]  [<801a20d8>] mntput_no_expire+0x18/0xe0
[11811.540001]  [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001]  [<801981ed>] sys_poll+0x2d/0x80
[11811.540001]  [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001]  =======================
[11811.540001] sshd          S f613cb04     0  5085      1
[11811.540001]        f7cd8000 00200086 00000002 f613cb04 f613cb0c 00000000 f7824380 00001000 
[11811.540001]        00200246 00000000 00000000 f784b928 00000000 01e339c0 00200046 f7cd8164 
[11811.540001]        82cbb080 00000000 00000000 f784b928 00200046 00000000 00000001 00000000 
[11811.540001] Call Trace:
[11811.540001]  [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001]  [<805e92a7>] tcp_poll+0x17/0x130
[11811.540001]  [<8019868c>] do_select+0x3dc/0x500
[11811.540001]  [<801982b0>] do_select+0x0/0x500
[11811.540001]  [<80198d60>] __pollwait+0x0/0xe0
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<80120070>] default_wake_function+0x0/0x10
[11811.540001]  [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001]  [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001]  [<80185695>] check_object+0xe5/0x1f0
[11811.540001]  [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001]  [<801202e4>] __wake_up+0x24/0x60
[11811.540001]  [<801e6717>] j