Home » Mailing list archives » linux-kernel » 2007 » November » 6

Re: 2.6.23.1: mdadm/raid5 hung/d-state

!MAILaRCHIVE_VOTE_RePLACE
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
[view in full thread]
From: Jeff Lessem <Jeff@...>
To: Dan Williams <dan.j.williams@...>
Cc: Justin Piszcz <jpiszcz@...>, Neil Brown <neilb@...>, <linux-kernel@...>, <linux-raid@...>, BERTRAND Joël <joel.bertrand@...>
Subject: Re: 2.6.23.1: mdadm/raid5 hung/d-state
Date: Tuesday, November 6, 2007 - 7:18 pm

Dan Williams wrote:
 > On 11/4/07, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
 >>
 >> On Mon, 5 Nov 2007, Neil Brown wrote:
 >>
 >>> On Sunday November 4, jpiszcz@lucidpixels.com wrote:
 >>>> # ps auxww | grep D
 >>>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME 
COMMAND
 >>>> root       273  0.0  0.0      0     0 ?        D    Oct21  14:40 
[pdflush]
 >>>> root       274  0.0  0.0      0     0 ?        D    Oct21  13:00 
[pdflush]
 >>>>
 >>>> After several days/weeks, this is the second time this has 
happened, while
 >>>> doing regular file I/O (decompressing a file), everything on the 
device
 >>>> went into D-state.
 >>> At a guess (I haven't looked closely) I'd say it is the bug that was
 >>> meant to be fixed by
 >>>
 >>> commit 4ae3f847e49e3787eca91bced31f8fd328d50496
 >>>
 >>> except that patch applied badly and needed to be fixed with
 >>> the following patch (not in git yet).
 >>> These have been sent to stable@ and should be in the queue for 2.6.23.2
 >>>
 >> Ah, thanks Neil, will be updating as soon as it is released, thanks.
 >>
 >
 > Are you seeing the same "md thread takes 100% of the CPU" that Joël is
 > reporting?

I'm also seeing something similar, but it only seems to cause a
problem if the file system is xfs.  Once I observed the md thread at
100% cpu, but usually the machine is just idle, with processes stuck
in the D state.

The system:

Quad Xeon with 4GB of ram running stock 2.6.23, x86_64.  Drives are
attached to an Adaptec AIC-9410W with aic94xx driver 1.0.3 and
firmware 1.1 (V17/10c6).  Unfortunately I can't try earlier kernels,
because the aic94xx driver didn't support SATA disks until 2.6.23.

I have 4 750GB drives in a RAID5 with LVM and an LV formated ext3
that works without problem.  I can't do extensive testing on those
drives because they contain important data.

I also have 4 400GB drives attached to the same controller.  I created
10GB partition on the 4 drives and a 30GB RAID5 across the drives.
Formatting this RAID as XFS and then running bonnie++ on it causes a
hang (stack trace at the bottom of this message).

Rebooting, letting the RAID resync, and reformating the partition ext3
allows bonnie++ to complete successfully.

bonnie++ completes successfully on an xfs formatted non-RAID partition
on one of the drives.

bonnie++ completes succesfully on an xfs formatted RAID0 across the 4
drives.

I should be able to provide any additional debugging information.  I
can also test any patches, either the previous one from this thread,
or new ones.

The following stack trace was performed after the RAID5 hung.  The
hung RAID is md3:

SysRq : Show State
   task                        PC stack   pid father
init          S 0000000000000000     0     1      0
  ffff81012fc47a18 0000000000000082 0000000000000000 ffff81012c73e9c0
  ffff81012c8b6478 ffff81012fc44000 ffff81012fcc2000 ffff81012fc44208
  0000000300001000 ffff81012fc47a28 00000000ffffffff 000000010030fe11
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff80259d96>] generic_file_buffered_write+0x575/0x695
  [<ffffffff80287d9d>] __d_lookup+0xb0/0x100
  [<ffffffff8027ec7d>] do_lookup+0x63/0x1ae
  [<ffffffff80287198>] dput+0x1c/0x10b
  [<ffffffff80280eb1>] __link_path_walk+0xbb7/0xd0c
  [<ffffffff8028bba8>] mntput_no_expire+0x1c/0x80
  [<ffffffff802810d4>] link_path_walk+0xce/0xe0
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff8028bba8>] mntput_no_expire+0x1c/0x80
  [<ffffffff8027ada4>] cp_new_stat+0xe7/0xff
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

kthreadd      S 0000000000000000     0     2      0
  ffff81012fc4bf20 0000000000000046 0000000000000000 0000000000000001
  ffff81011598bb98 ffff81012fc44720 ffffffff805354c0 ffff81012fc44928
  0000000000000000 ffff81011598bb90 00000000ffffffff 0000000000000286
Call Trace:
  [<ffffffff802410c4>] kthreadd+0x73/0x12e
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff803075b0>] acpi_ds_init_one_object+0x0/0x7c
  [<ffffffff80241051>] kthreadd+0x0/0x12e
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

migration/0   S 0000000000000000     0     3      2
  ffff81012fc4feb0 0000000000000046 0000000000000001 0000000000000001
  ffff8100a4e67e90 ffff81012fc44e40 ffff810129824000 ffff81012fc45048
  0000000000000000 ffff8100a4e67e88 ffff8100a4e67e90 0000000000000286
Call Trace:
  [<ffffffff8022c553>] migration_thread+0x185/0x21d
  [<ffffffff8022c3ce>] migration_thread+0x0/0x21d
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksoftirqd/0   S 0000000000000000     0     4      2
  ffff81012fc51f10 0000000000000046 0000000000000000 ffff81012fc45770
  0000000000000001 ffff81012fc45560 ffffffff805354c0 ffff81012fc45768
  000000002fc44720 ffff81012ea47560 00000000ffffffff 0000000000000000
Call Trace:
  [<ffffffff80234540>] ksoftirqd+0x0/0x9b
  [<ffffffff80234557>] ksoftirqd+0x17/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

migration/1   S 0000000000000001     0     5      2
  ffff81012fc55eb0 0000000000000046 0000000000000001 0000000000000001
  ffff8100a49e9e90 ffff81012fc52000 ffff810129a82e40 ffff81012fc52208
  0000000100000000 ffff8100a49e9e88 ffff8100a49e9e90 0000000000000286
Call Trace:
  [<ffffffff8022c553>] migration_thread+0x185/0x21d
  [<ffffffff8022c3ce>] migration_thread+0x0/0x21d
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksoftirqd/1   S 0000000000000000     0     6      2
  ffff81012fc5df10 0000000000000046 0000000000000000 ffff81012fc52930
  0000000100000001 ffff81012fc52720 ffff81012fc52e40 ffff81012fc52928
  00000001ffffffff ffff81012ea47560 00000000ffffffff 0000000000000001
Call Trace:
  [<ffffffff80234540>] ksoftirqd+0x0/0x9b
  [<ffffffff80234557>] ksoftirqd+0x17/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

migration/2   S 0000000000000002     0     7      2
  ffff81012fc87eb0 0000000000000046 0000000000000001 0000000000000001
  ffff8100a489be90 ffff81012fc53560 ffff810129824000 ffff81012fc53768
  0000000200000000 ffff8100a489be88 ffff8100a489be90 0000000000000286
Call Trace:
  [<ffffffff8022c553>] migration_thread+0x185/0x21d
  [<ffffffff8022c3ce>] migration_thread+0x0/0x21d
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksoftirqd/2   S 0000000000000000     0     8      2
  ffff81012fc91f10 0000000000000046 0000000000000000 ffff81012fc8a210
  0000000200000001 ffff81012fc8a000 ffff81012fc8a720 ffff81012fc8a208
  000000022fc44720 ffff81012ea47560 00000000ffffffff 0000000000000002
Call Trace:
  [<ffffffff80234540>] ksoftirqd+0x0/0x9b
  [<ffffffff80234557>] ksoftirqd+0x17/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

migration/3   S 0000000000000000     0     9      2
  ffff81012fcb9eb0 0000000000000046 0000000000000000 0000000000000001
  ffff8100a49e7e90 ffff81012fc8ae40 ffff81012fcc2000 ffff81012fc8b048
  0000000300000000 ffff8100a49e7e88 00000000ffffffff 0000000000000286
Call Trace:
  [<ffffffff8022c553>] migration_thread+0x185/0x21d
  [<ffffffff8022c3ce>] migration_thread+0x0/0x21d
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksoftirqd/3   S 0000000000000003     0    10      2
  ffff81012fcc1f10 0000000000000046 ffff81012d4cd560 ffff81012fc8b770
  0000000300000001 ffff81012fc8b560 ffff81012ea47560 ffff81012fc8b768
  000000032fc44720 ffff81012ea47560 00000000ffffffff 0000000000000003
Call Trace:
  [<ffffffff80234540>] ksoftirqd+0x0/0x9b
  [<ffffffff80234557>] ksoftirqd+0x17/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

events/0      S 0000000000000000     0    11      2
  ffff81012fceded0 0000000000000046 0000000000000000 ffffffff80237ddc
  ffffffff802147e3 ffff81012fcc2720 ffffffff805354c0 ffff81012fcc2928
  00000000ffffffff 00000000000000fa 00000000ffffffff ffffffff8023e5fe
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff802147e3>] mcheck_timer+0x0/0x7c
  [<ffffffff8023e5fe>] queue_delayed_work_on+0xae/0xbe
  [<ffffffff80261c6a>] vmstat_update+0x0/0x32
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

events/1      S 0000000000000000     0    12      2
  ffff81012fcf1ed0 0000000000000046 0000000000000000 ffffffff80237ddc
  ffff81012c74a9a0 ffff81012fcc2e40 ffff81012fc52e40 ffff81012fcc3048
  00000001ffffffff 00000000000000fa 00000000ffffffff ffffffff8023e5fe
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff8023e5fe>] queue_delayed_work_on+0xae/0xbe
  [<ffffffff80261c6a>] vmstat_update+0x0/0x32
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

events/2      S 0000000000000000     0    13      2
  ffff81012fcf3ed0 0000000000000046 0000000000000000 ffffffff80237ddc
  ffff81012d73cec8 ffff81012fcc3560 ffff81012fc8a720 ffff81012fcc3768
  00000002ffffffff 00000000000000fa 00000000ffffffff ffffffff8023e5fe
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff8023e5fe>] queue_delayed_work_on+0xae/0xbe
  [<ffffffff80261c6a>] vmstat_update+0x0/0x32
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

events/3      S 0000000000000000     0    14      2
  ffff81012fcf7ed0 0000000000000046 0000000000000000 ffffffff80237ddc
  0000000000000286 ffff81012fcf4000 ffff81012fcc2000 ffff81012fcf4208
  00000003ffffffff 00000000000000fa 00000000ffffffff ffffffff8023e5fe
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff8023e5fe>] queue_delayed_work_on+0xae/0xbe
  [<ffffffff80261c6a>] vmstat_update+0x0/0x32
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

khelper       S 0000000000000002     0    15      2
  ffff81012fcfbed0 0000000000000046 0000000000001351 ffff81012b6bc188
  0000000000000611 ffff81012fcf4720 ffff810129d44e40 ffff81012fcf4928
  000000028020cbfe 0000000000000010 0000000000000200 0000000000000000
Call Trace:
  [<ffffffff8023d7f4>] __call_usermodehelper+0x41/0x61
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kblockd/0     S 0000000000000000     0    54      2
  ffff81012fd83ed0 0000000000000046 0000000000000000 ffffffff802ec753
  ffff81012ee54000 ffff81012fd80000 ffffffff805354c0 ffff81012fd80208
  000000002ee54000 ffffffff802e8fbd 00000000ffffffff ffffffffffffffff
Call Trace:
  [<ffffffff802ec753>] kobject_get+0x12/0x17
  [<ffffffff802e8fbd>] cfq_kick_queue+0x0/0x35
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kblockd/1     S 0000000000000001     0    55      2
  ffff81012fd85ed0 0000000000000046 ffff81012ec9e1d8 ffffffff802ec753
  ffff81012ee54000 ffff81012fd80720 ffff810129824000 ffff81012fd80928
  000000012ee54000 ffffffff802e8fbd 0000000000000286 ffffffffffffffff
Call Trace:
  [<ffffffff802ec753>] kobject_get+0x12/0x17
  [<ffffffff802e8fbd>] cfq_kick_queue+0x0/0x35
  [<ffffffff802e8fdf>] cfq_kick_queue+0x22/0x35
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kblockd/2     S 0000000000000000     0    56      2
  ffff81012fd89ed0 0000000000000046 0000000000000000 ffffffff802ec753
  ffff81012e7505e0 ffff81012fd80e40 ffff81012fc8a720 ffff81012fd81048
  000000022e7505e0 ffffffff802e8fbd 00000000ffffffff ffffffffffffffff
Call Trace:
  [<ffffffff802ec753>] kobject_get+0x12/0x17
  [<ffffffff802e8fbd>] cfq_kick_queue+0x0/0x35
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kblockd/3     S 0000000000000003     0    57      2
  ffff81012fd8bed0 0000000000000046 ffff81012e8fd1d8 ffffffff802ec753
  ffff81012ee551a0 ffff81012fd81560 ffff81012ec75560 ffff81012fd81768
  000000032ee551a0 ffffffff802e8fbd 0000000000000286 ffffffffffffffff
Call Trace:
  [<ffffffff802ec753>] kobject_get+0x12/0x17
  [<ffffffff802e8fbd>] cfq_kick_queue+0x0/0x35
  [<ffffffff802e8fdf>] cfq_kick_queue+0x22/0x35
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kacpid        S 0000000000000000     0    59      2
  ffff81012fd97ed0 0000000000000046 ffff81012fd97e60 0000000000000000
  ffff81012fd90720 ffff81012fd90720 ffff81012fd90e40 ffff81012fd90928
  0000000000000000 ffffffff80448110 ffff81012fd97f20 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kacpi_notify  S 0000000000000000     0    60      2
  ffff81012fd99ed0 0000000000000046 ffff81012fd99e60 0000000000000000
  ffff81012fd90e40 ffff81012fd90e40 ffff81012fd91560 ffff81012fd91048
  0000000000000000 ffffffff80448110 ffff81012fd99f20 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ata/0         S 0000000000000000     0   189      2
  ffff81012fd9fed0 0000000000000046 0000000000000000 000000005800176d
  ffff81012fd9fee0 ffff81012fda3560 ffffffff805354c0 ffff81012fda3768
  0000000000000004 ffff81012ecd69e0 00000000ffffffff ffffffff805dd920
Call Trace:
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ata/1         S 0000000000000000     0   190      2
  ffff81012fd4bed0 0000000000000046 0000000000000000 00000000580017a9
  ffff81012fd4bee0 ffff81012fd48000 ffff81012fc52e40 ffff81012fd48208
  0000000100000004 ffff81012ecd69e0 00000000ffffffff ffffffff805dd920
Call Trace:
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ata/2         S 0000000000000000     0   191      2
  ffff81012fd53ed0 0000000000000046 0000000000000000 00000000580016c8
  ffff81012fd53ee0 ffff81012fd48720 ffff81012fc8a720 ffff81012fd48928
  0000000200000004 ffff81012ecd69e0 00000000ffffffff ffffffff805dd920
Call Trace:
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ata/3         S 0000000000000000     0   192      2
  ffff81012fd11ed0 0000000000000046 0000000000000000 00000000580017ea
  ffff81012fd11ee0 ffff81012fd48e40 ffff81012fcc2000 ffff81012fd49048
  0000000300000004 ffff81012ecd4ae0 00000000ffffffff ffffffff805dd920
Call Trace:
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ata_aux       S 0000000000000000     0   193      2
  ffff81012fd13ed0 0000000000000046 ffff81012fd13e60 0000000000000000
  ffff81012fd49560 ffff81012fd49560 ffff81012fd36000 ffff81012fd49768
  0000000000000000 ffffffff80448110 ffff81012fd13f20 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksuspend_usbd S 0000000000000000     0   194      2
  ffff81012fd05ed0 0000000000000046 ffff81012fd05e60 0000000000000000
  ffff81012fd36000 ffff81012fd36000 ffff81012fd36720 ffff81012fd36208
  0000000000000000 ffffffff80448110 ffff81012fd05f20 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

khubd         S 0000000000000000     0   200      2
  ffff81012fe03e40 0000000000000046 0000000000000000 ffffffff80228f46
  ffff81012ea8c000 ffff81012fe14e40 ffff81012fcc2000 ffff81012fe15048
  0000000300000003 ffff81012ea8c000 00000000ffffffff ffffffff8044985b
Call Trace:
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff8044985b>] __up_wakeup+0x35/0x67
  [<ffffffff80393d35>] hub_thread+0xb33/0xb93
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80393202>] hub_thread+0x0/0xb93
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kseriod       S 0000000000000000     0   203      2
  ffff81012fdefed0 0000000000000046 0000000000000000 ffffffff80446d57
  0000000000000000 ffff81012fdea720 ffffffff805354c0 ffff81012fdea928
  00000000805dd920 ffffffff80345f1d 00000000ffffffff ffffffff80346044
Call Trace:
  [<ffffffff80446d57>] klist_next+0x2d/0x83
  [<ffffffff80345f1d>] next_device+0x9/0x1f
  [<ffffffff80346044>] bus_for_each_dev+0x61/0x6e
  [<ffffffff803a4a0d>] serio_thread+0x2ac/0x2e3
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803a4761>] serio_thread+0x0/0x2e3
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

pdflush       D 0000000000000000     0   261      2
  ffff81012ecc7930 0000000000000046 ffff81012ecc78f8 0000000000000001
  0000000000000082 ffff81012ecbf560 ffff81012fc52e40 ffff81012ecbf768
  000000012cb33800 0000000000000046 00000000ffffffff ffffffff802e27e0
Call Trace:
  [<ffffffff802e27e0>] __generic_unplug_device+0x13/0x24
  [<ffffffff803b12cf>] get_active_stripe+0x22f/0x4ca
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff803b6b3b>] make_request+0x3f3/0x577
  [<ffffffff8025acf6>] mempool_alloc+0x24/0xda
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802e1728>] generic_make_request+0x1be/0x1f5
  [<ffffffff802e3caf>] submit_bio+0xb4/0xbb
  [<ffffffff80298e2f>] __bio_add_page+0x109/0x1b9
  [<ffffffff8827dcb4>] :xfs:xfs_submit_ioend_bio+0x1e/0x27
  [<ffffffff8827e0ce>] :xfs:xfs_submit_ioend+0x88/0xc6
  [<ffffffff8827ef51>] :xfs:xfs_page_state_convert+0x51e/0x56d
  [<ffffffff8827f0f2>] :xfs:xfs_vm_writepage+0xa7/0xe1
  [<ffffffff8025d022>] __writepage+0xa/0x23
  [<ffffffff8025d552>] write_cache_pages+0x176/0x2a3
  [<ffffffff8025d018>] __writepage+0x0/0x23
  [<ffffffff8025d6bb>] do_writepages+0x20/0x2d
  [<ffffffff80291fd1>] __writeback_single_inode+0x1d6/0x3a7
  [<ffffffff80228561>] update_curr+0xdf/0xfe
  [<ffffffff802924fa>] sync_sb_inodes+0x1cb/0x2af
  [<ffffffff80292961>] writeback_inodes+0x7d/0xd3
  [<ffffffff8025dafb>] background_writeout+0x84/0xb7
  [<ffffffff8025df26>] pdflush+0x0/0x1d8
  [<ffffffff8025e054>] pdflush+0x12e/0x1d8
  [<ffffffff8025da77>] background_writeout+0x0/0xb7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

pdflush       D 0000000000000000     0   262      2
  ffff81012eccbdb0 0000000000000046 0000000000000000 ffff81012ecc8000
  0000000000000286 ffff81012ecc8000 ffffffff805354c0 ffff81012ecc8208
  000000002eccbe90 ffff81012eccbdc0 00000000ffffffff 000000010030fe0d
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff804483fe>] io_schedule_timeout+0x28/0x33
  [<ffffffff802622f4>] congestion_wait+0x66/0x80
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80292987>] writeback_inodes+0xa3/0xd3
  [<ffffffff8025dc1c>] wb_kupdate+0xba/0x111
  [<ffffffff8025df26>] pdflush+0x0/0x1d8
  [<ffffffff8025e054>] pdflush+0x12e/0x1d8
  [<ffffffff8025db62>] wb_kupdate+0x0/0x111
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kswapd0       S 0000000000000000     0   263      2
  ffff81012eccfe20 0000000000000046 0000000000000000 ffffffff8022bc95
  ffff81012eccfde0 ffff81012ecc8720 ffff81012fcc2000 ffff81012ecc8928
  0000000300000001 ffff81012eccfe10 00000000ffffffff ffffffff80228561
Call Trace:
  [<ffffffff8022bc95>] set_cpus_allowed+0xa5/0xb2
  [<ffffffff80228561>] update_curr+0xdf/0xfe
  [<ffffffff80260c38>] kswapd+0x0/0x429
  [<ffffffff80260d0b>] kswapd+0xd3/0x429
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80260c38>] kswapd+0x0/0x429
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

aio/0         S 0000000000000000     0   318      2
  ffff81012ed81ed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012ecc8e40 ffff81012ecc8e40 ffffffff805354c0 ffff81012ecc9048
  0000000000000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

aio/1         S 0000000000000000     0   319      2
  ffff81012ed83ed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012ecc9560 ffff81012ecc9560 ffff81012fc52e40 ffff81012ecc9768
  0000000100000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

aio/2         S 0000000000000000     0   320      2
  ffff81012ed89ed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012ed86000 ffff81012ed86000 ffff81012fc8a720 ffff81012ed86208
  0000000200000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

aio/3         S 0000000000000000     0   321      2
  ffff81012ed8bed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012ed86720 ffff81012ed86720 ffff81012fcc2000 ffff81012ed86928
  000000032ecae000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_0     S 0000000000000000     0   497      2
  ffff81012ecb9e80 0000000000000046 ffff81012ecb9e48 0000000000000003
  ffff81012ecb9e40 ffff81012ed86e40 ffffffff805354c0 ffff81012ed87048
  0000000000000000 0000000000000001 00000000ffffffff ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_1     S 0000000000000000     0   499      2
  ffff81012ecb3e80 0000000000000046 0000000000000000 0000000000000003
  ffff81012ecb3e40 ffff81012ed87560 ffff81012fc52e40 ffff81012ed87768
  0000000100000000 0000000000000001 00000000ffffffff ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_2     S 0000000000000003     0   501      2
  ffff81012ee25e80 0000000000000046 0000000000000082 0000000000000003
  ffff81012ee25e40 ffff81012fdeb560 ffff81012fc44000 ffff81012fdeb768
  0000000300000000 0000000000000001 0000000000000246 ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_3     S 0000000000000003     0   503      2
  ffff81012ed93e80 0000000000000046 0000000000000082 0000000000000003
  ffff81012ed93e40 ffff81012fdeae40 ffff81012fc44000 ffff81012fdeb048
  0000000300000000 0000000000000001 0000000000000246 ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_4     S 0000000000000003     0   505      2
  ffff81012eca7e80 0000000000000046 0000000000000082 0000000000000003
  ffff81012eca7e40 ffff81012fdea000 ffff81012fc44000 ffff81012fdea208
  0000000300000000 0000000000000001 0000000000000246 ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_5     S 0000000000000003     0   507      2
  ffff81012ec97e80 0000000000000046 0000000000000082 0000000000000003
  ffff81012ec97e40 ffff81012fd36720 ffff81012fc44000 ffff81012fd36928
  0000000300000000 0000000000000001 0000000000000246 ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_6     S 0000000000000000     0   524      2
  ffff81012ecd3e80 0000000000000046 0000000000000000 0000000000000003
  ffff81012ec9f9d8 ffff81012fd37560 ffffffff805354c0 ffff81012fd37768
  0000000000000202 0000000000000000 00000000ffffffff ffffffff8035db6a
Call Trace:
  [<ffffffff8035db6a>] __scsi_iterate_devices+0x63/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_7     S 0000000000000000     0   526      2
  ffff81012ede5e80 0000000000000046 0000000000000000 0000000000000003
  ffff81012ede5e40 ffff81012fd36e40 ffff81012fcc2000 ffff81012fd37048
  0000000300000000 0000000000000001 00000000ffffffff ffffffff8035db5d
Call Trace:
  [<ffffffff8035db5d>] __scsi_iterate_devices+0x56/0x6f
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kcryptd/0     S 0000000000000000     0   544      2
  ffff81012edfded0 0000000000000046 0000000000000000 0000000000000000
  ffff81012fda2e40 ffff81012fda2e40 ffffffff805354c0 ffff81012fda3048
  0000000000000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kcryptd/1     S 0000000000000000     0   545      2
  ffff81012ed6bed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012fda2720 ffff81012fda2720 ffff81012fc52e40 ffff81012fda2928
  0000000100000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kcryptd/2     S 0000000000000000     0   546      2
  ffff81012ee59ed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012fda2000 ffff81012fda2000 ffff81012fc8a720 ffff81012fda2208
  0000000200000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kcryptd/3     S 0000000000000000     0   547      2
  ffff81012ecdfed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012fd90000 ffff81012fd90000 ffff81012fcc2000 ffff81012fd90208
  0000000300000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

ksnapd        S 0000000000000000     0   548      2
  ffff81012ed1bed0 0000000000000046 0000000000000000 0000000000000000
  ffff81012fd91560 ffff81012fd91560 ffff81012fcc2000 ffff81012fd91768
  0000000300000000 ffffffff80448110 00000000ffffffff 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_8     S 0000000000000003     0  1093      2
  ffff81012ebb3e80 0000000000000046 0000000000000001 ffff81012ec74768
  ffff8100052c0b80 ffff81012ec74720 ffff81012e645560 ffff81012ec74928
  000000032ebb3e50 ffff81012ec74720 ffff81012ec74720 ffffffffffffffff
Call Trace:
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_wq_8     S 0000000000000003     0  1094      2
  ffff81012e67fed0 0000000000000046 0000000000000286 ffffffff8036bbf9
  ffff81012eba7a00 ffff81012e645560 ffff81012e8eee40 ffff81012e645768
  000000032e6c4154 ffffffff8036f154 ffff81012e6c0f30 ffff81012e993180
Call Trace:
  [<ffffffff8036bbf9>] sas_rphy_add+0x133/0x13f
  [<ffffffff8036f154>] sas_discover_domain+0x344/0x3fc
  [<ffffffff8036ee10>] sas_discover_domain+0x0/0x3fc
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

scsi_eh_9     S 0000000000000001     0  1665      2
  ffff81012e4a1e80 0000000000000046 0000000000000001 ffff81012ea46e88
  ffff8100052aeb80 ffff81012ea46e40 ffff81012ea47560 ffff81012ea47048
  000000012e4a1e50 ffff81012ea46e40 ffff81012ea46e40 ffffffffffffffff
Call Trace:
  [<ffffffff803619ad>] scsi_error_handler+0x59/0x4b7
  [<ffffffff80361954>] scsi_error_handler+0x0/0x4b7
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

usb-storage   S 0000000000000000     0  1667      2
  ffff81012e681e20 0000000000000046 ffff81012e58fca0 ffffffff8806ea5f
  0000000000000000 ffff81012ea47560 ffff81012fc45560 ffff81012ea47768
  00000000000002d5 0000000000000001 0000000000000001 ffff81012e681e20
Call Trace:
  [<ffffffff8806ea5f>] :usb_storage:usb_stor_msg_common+0x110/0x13a
  [<ffffffff804499e5>] __down_interruptible+0xcb/0x137
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff804497e7>] __down_failed_interruptible+0x35/0x3a
  [<ffffffff8806fca5>] :usb_storage:usb_stor_control_thread+0x27/0x1e2
  [<ffffffff8806fc7e>] :usb_storage:usb_stor_control_thread+0x0/0x1e2
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

md0_raid1     S 0000000000000000     0  1836      2
  ffff81012eab1e80 0000000000000046 0000000000000000 ffff81012e8ef5a8
  ffff8100052aeb80 ffff81012e8ef560 ffff81012fcc2000 ffff81012e8ef768
  000000032eab1e50 ffff81012fcc2000 00000000ffffffff ffffffffffffffff
Call Trace:
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802414fd>] prepare_to_wait+0x15/0x5f
  [<ffffffff803c02fb>] md_thread+0xbb/0xf1
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803c0240>] md_thread+0x0/0xf1
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

md1_raid1     S 0000000000000000     0  1854      2
  ffff81012e5bde80 0000000000000046 0000000000000000 ffff81012e8ee048
  ffff8100052aeb80 ffff81012e8ee000 ffff81012fc8a720 ffff81012e8ee208
  000000022e5bde50 ffffffff805354c0 00000000ffffffff ffffffffffffffff
Call Trace:
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802414fd>] prepare_to_wait+0x15/0x5f
  [<ffffffff803c02fb>] md_thread+0xbb/0xf1
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803c0240>] md_thread+0x0/0xf1
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

md2_raid5     S 0000000000000000     0  1872      2
  ffff81012e479e80 0000000000000046 0000000000000000 0000000000000046
  ffff81012eaa1780 ffff81012e8ee720 ffff81012fc52e40 ffff81012e8ee928
  000000012eb1c800 ffff81012fc52e40 00000000ffffffff ffff81012e453400
Call Trace:
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802414fd>] prepare_to_wait+0x15/0x5f
  [<ffffffff803c02fb>] md_thread+0xbb/0xf1
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803c0240>] md_thread+0x0/0xf1
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kjournald     S 0000000000000000     0  1980      2
  ffff81012d503eb0 0000000000000046 0000000000000000 00000000802413e5
  00000fcc00000000 ffff81012e48c720 ffffffff805354c0 ffff81012e48c928
  0000000000000000 0000000000000001 00000000ffffffff 0000000000000003
Call Trace:
  [<ffffffff802d0936>] kjournald+0x165/0x1e6
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802d07d1>] kjournald+0x0/0x1e6
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff8020cf7c>] call_softirq+0x1c/0x28
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

udevd         S 0000000000000000     0  2189      1
  ffff81012e7f7a18 0000000000000086 0000000000000000 ffffffff80264e33
  80000001143fa065 ffff81012ecbee40 ffff81012fc52e40 ffff81012ecbf048
  0000000100000870 ffff81012fc52e40 00000000ffffffff ffff81012ecbee40
Call Trace:
  [<ffffffff80264e33>] handle_mm_fault+0x6f3/0x772
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff8029dde8>] inotify_poll+0x4f/0x56
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
sage repeated 3 times
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff802ef67d>] number+0x119/0x204
  [<ffffffff8025c800>] get_page_from_freelist+0x278/0x35e
  [<ffffffff80287198>] dput+0x1c/0x10b
  [<ffffffff8027eab8>] __follow_mount+0x26/0x7b
  [<ffffffff8027ec7d>] do_lookup+0x63/0x1ae
  [<ffffffff8028bba8>] mntput_no_expire+0x1c/0x80
  [<ffffffff8028098f>] __link_path_walk+0x695/0xd0c
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff80287d9d>] __d_lookup+0xb0/0x100
  [<ffffffff802910e3>] simple_empty+0x10/0x58
  [<ffffffff80241559>] remove_wait_queue+0x12/0x44
  [<ffffffff802846bc>] sys_select+0xc1/0x183
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff80278ad7>] sys_read+0x45/0x6e
  [<ffffffff8020bdee>] system_call+0x7e/0x83

kpsmoused     S 0000000000000001     0  2826      2
  ffff81012d0f3ed0 0000000000000046 ffff81012d0f3e60 0000000000000000
  ffff81012e599560 ffff81012e599560 ffff81012ecbee40 ffff81012e599768
  000000012fd2c768 ffffffff80448110 ffff81012d0f3f20 0000000000000046
Call Trace:
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kjournald     S 0000000000000000     0  3333      2
  ffff81012c8b1eb0 0000000000000046 0000000000000000 0000000000000c31
  ffff81012dbd9a98 ffff81012fcf4e40 ffffffff805354c0 ffff81012fcf5048
  0000000000000000 0000000000000001 00000000ffffffff 0000000000000003
Call Trace:
  [<ffffffff802d0936>] kjournald+0x165/0x1e6
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802d07d1>] kjournald+0x0/0x1e6
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kjournald     S 0000000000000000     0  3334      2
  ffff81012d0b1eb0 0000000000000046 ffff81012e09b898 00000000000003f0
  ffff81012e09b898 ffff81012e8eee40 ffff81012fcf5560 ffff81012e8ef048
  0000000000000000 0000000000000001 0000000000000282 0000000000000003
Call Trace:
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff802d0936>] kjournald+0x165/0x1e6
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802d07d1>] kjournald+0x0/0x1e6
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

kjournald     S 0000000000000000     0  3335      2
  ffff81012e3edeb0 0000000000000046 0000000000000000 0000000000000c31
  ffff81012e09be98 ffff81012e598000 ffffffff805354c0 ffff81012e598208
  0000000000000000 0000000000000001 00000000ffffffff 0000000000000003
Call Trace:
  [<ffffffff802d0936>] kjournald+0x165/0x1e6
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802d07d1>] kjournald+0x0/0x1e6
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

portmap       S 0000000000000000     0  3766      1
  ffff81012f703b28 0000000000000086 0000000000000000 ffffffff802eebcf
  ffff81012f0ab888 ffff81012d4cc720 ffff81012fc52e40 ffff81012d4cc928
  000000012dfa4600 ffff81012fc52e40 00000000ffffffff ffffffff00030002
Call Trace:
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff8040caaf>] tcp_poll+0x25/0x138
  [<ffffffff80283b12>] do_sys_poll+0x278/0x360
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8025885f>] file_read_actor+0xa0/0x118
  [<ffffffff8025897e>] find_get_page+0x21/0x50
  [<ffffffff80259207>] do_generic_mapping_read+0x3f4/0x406
  [<ffffffff802587bf>] file_read_actor+0x0/0x118
  [<ffffffff8025a61e>] generic_file_aio_read+0x11d/0x160
  [<ffffffff8025c2bf>] __pagevec_free+0x21/0x2e
  [<ffffffff8025e950>] release_pages+0x14a/0x157
  [<ffffffff80263edc>] unmap_vmas+0x3ec/0x710
  [<ffffffff8026d4d5>] free_pages_and_swap_cache+0x73/0x8f
  [<ffffffff802679e6>] unmap_region+0x114/0x12a
  [<ffffffff80267a40>] remove_vma+0x44/0x4b
  [<ffffffff80268687>] do_munmap+0x254/0x276
  [<ffffffff80283c2c>] sys_poll+0x32/0x3b
  [<ffffffff8020bdee>] system_call+0x7e/0x83

rpc.statd     S 0000000000000000     0  3783      1
  ffff81012c5bda18 0000000000000086 0000000000000000 ffffffff80229af5
  ffff81012c5bdad8 ffff81012e373560 ffff81012fcc2000 ffff81012e373768
  0000000300000003 0000000000000000 00000000ffffffff 0000000000000000
Call Trace:
  [<ffffffff80229af5>] find_busiest_group+0x254/0x6d6
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff8040caaf>] tcp_poll+0x25/0x138
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
sage repeated 2 times
  [<ffffffff8042298d>] udp_recvmsg+0x193/0x1f3
  [<ffffffff803dd31e>] sock_common_recvmsg+0x30/0x45
  [<ffffffff803dbe0b>] sock_recvmsg+0xd5/0xed
  [<ffffffff80449c4f>] _spin_lock_bh+0x9/0x19
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff803ddac9>] lock_sock_nested+0xa2/0xad
  [<ffffffff80449c4f>] _spin_lock_bh+0x9/0x19
  [<ffffffff803dd99f>] release_sock+0x13/0x9b
  [<ffffffff80241321>] bit_waitqueue+0x16/0x82
  [<ffffffff802413cb>] wake_up_bit+0x11/0x22
  [<ffffffff802846bc>] sys_select+0xc1/0x183
  [<ffffffff80276816>] filp_close+0x5d/0x65
  [<ffffffff8020bdee>] system_call+0x7e/0x83

rpciod/0      S 0000000000000000     0  3790      2
  ffff81012c693ed0 0000000000000046 0000000000000000 ffffffff80228f46
  ffff810128534308 ffff81012d97b560 ffffffff805354c0 ffff81012d97b768
  00000000805dd920 0000000000000000 00000000ffffffff 0000000000000286
Call Trace:
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

rpciod/1      S 0000000000000000     0  3791      2
  ffff81012c695ed0 0000000000000046 0000000000000000 ffff81012cea4000
  ffff81012c846a10 ffff81012d97ae40 ffff81012fc52e40 ffff81012d97b048
  0000000100000282 ffffffff802413b5 00000000ffffffff ffff810100000004
Call Trace:
  [<ffffffff802413b5>] __wake_up_bit+0x28/0x2d
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

rpciod/2      S 0000000000000000     0  3792      2
  ffff81012c699ed0 0000000000000046 0000000000000000 ffffffff80228f46
  ffff810128534c08 ffff81012d97a000 ffff81012fc8a720 ffff81012d97a208
  00000002805dd920 0000000000000000 00000000ffffffff 0000000000000286
Call Trace:
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

rpciod/3      S 0000000000000000     0  3793      2
  ffff81012c69bed0 0000000000000046 0000000000000000 ffff81012cb29d00
  ffff81012cf8cd00 ffff81012e644000 ffff81012fcc2000 ffff81012e644208
  000000032ca80c00 0000000000000287 00000000ffffffff ffffffff881aad70
Call Trace:
  [<ffffffff8023e6c7>] worker_thread+0x74/0x9b
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8023e653>] worker_thread+0x0/0x9b
  [<ffffffff802412cc>] kthread+0x47/0x73
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff80241285>] kthread+0x0/0x73
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

rpc.idmapd    S 0000000000000000     0  3817      1
  ffff81012c165e98 0000000000000082 0000000000000000 ffff81012c165ec8
  7474697765680101 ffff81012fe14720 ffff81012fc8a720 ffff81012fe14928
  0000000200000000 ffff81012c165ea8 00000000ffffffff 000000010030ffb3
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff8029e84e>] sys_epoll_wait+0x17f/0x421
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8023ab09>] sys_rt_sigprocmask+0x50/0xce
  [<ffffffff8020bdee>] system_call+0x7e/0x83

nfsv4-svc     S 0000000000000000     0  3823      2
  ffff81012c585e20 0000000000000046 0000000000000000 000005a800000001
  ffff81012c41acc8 ffff81012ea46000 ffffffff805354c0 ffff81012ea46208
  000000002c41ac00 ffffffff803dd99f 00000000ffffffff 0000000000000282
Call Trace:
  [<ffffffff803dd99f>] release_sock+0x13/0x9b
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff8814d308>] :sunrpc:svc_recv+0x28e/0x40f
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff881a714f>] :nfs:nfs_callback_svc+0xa9/0x149
  [<ffffffff80232f85>] do_exit+0x7bc/0x7c0
  [<ffffffff8020cc08>] child_rip+0xa/0x12
  [<ffffffff881a70a6>] :nfs:nfs_callback_svc+0x0/0x149
  [<ffffffff8020cbfe>] child_rip+0x0/0x12

syslogd       R  running task        0  4037      1
klogd         S 0000000000000000     0  4043      1
  ffff81012c6e1bf8 0000000000000082 0000000000000000 ffffffff80227c57
  ffff810005260480 ffff81012c047560 ffff81012fc52e40 ffff81012c047768
  00000001805a3b40 ffffffff803ddc95 00000000ffffffff 0000000000000286
Call Trace:
  [<ffffffff80227c57>] enqueue_task+0x13/0x21
  [<ffffffff803ddc95>] sock_alloc_send_skb+0x77/0x1d2
  [<ffffffff803ddc95>] sock_alloc_send_skb+0x77/0x1d2
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff803e17b9>] __alloc_skb+0x76/0x121
  [<ffffffff8024149f>] prepare_to_wait_exclusive+0x15/0x5e
  [<ffffffff8043b9b4>] unix_wait_for_peer+0x90/0xac
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803e08cc>] skb_queue_tail+0x17/0x3e
  [<ffffffff803df401>] sock_def_readable+0x10/0x5f
  [<ffffffff8043bf67>] unix_dgram_sendmsg+0x3fb/0x491
  [<ffffffff803db31f>] sock_aio_write+0xd1/0xe0
  [<ffffffff80277ec8>] do_sync_write+0xc9/0x10c
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802b4818>] kmsg_read+0x3a/0x44
  [<ffffffff8027861b>] vfs_write+0xc0/0x136
  [<ffffffff80278b45>] sys_write+0x45/0x6e
  [<ffffffff8020bdee>] system_call+0x7e/0x83

ypbind        S 0000000000000000     0  4067      1
  ffff81012c3ddb28 0000000000000082 0000000000000000 ffffffff804081d6
  ffff81012c47e300 ffff81012e598720 ffff81012fc52e40 ffff81012e598928
  000000012c47e300 ffffffff804055e5 00000000ffffffff ffffffff80406d2f
Call Trace:
  [<ffffffff804081d6>] ip_output+0x2bb/0x301
  [<ffffffff804055e5>] ip_push_pending_frames+0x3bd/0x425
  [<ffffffff80406d2f>] ip_generic_getfrag+0x0/0x8b
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff8040caaf>] tcp_poll+0x25/0x138
  [<ffffffff80283b12>] do_sys_poll+0x278/0x360
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff803dd31e>] sock_common_recvmsg+0x30/0x45
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803dbe0b>] sock_recvmsg+0xd5/0xed
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff803db1f1>] move_addr_to_kernel+0x25/0x36
  [<ffffffff803e2b8e>] verify_iovec+0x46/0x84
  [<ffffffff803dc16a>] sys_sendmsg+0x264/0x287
  [<ffffffff803dc88e>] move_addr_to_user+0x3a/0x4f
  [<ffffffff803dcc1a>] sys_recvfrom+0x121/0x136
  [<ffffffff80228561>] update_curr+0xdf/0xfe
  [<ffffffff80448110>] thread_return+0x0/0xf9
  [<ffffffff80283c2c>] sys_poll+0x32/0x3b
  [<ffffffff8020bdee>] system_call+0x7e/0x83

ypbind        S 0000000000000002     0  4068      1
  ffff81012c3ffe58 0000000000000082 000000000050a000 ffff81012d799040
  ffff810100000766 ffff81012ea46720 ffff81012e598720 ffff81012ea46928
  0000000200000004 00000000ffffffda 0000000000000004 0000000000000002
Call Trace:
  [<ffffffff802473d3>] do_futex+0x30b/0x9e0
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff8023838b>] recalc_sigpending+0xe/0x25
  [<ffffffff80239f96>] dequeue_signal+0x8d/0x115
  [<ffffffff8023ae14>] sys_rt_sigtimedwait+0x17a/0x255
  [<ffffffff80247b8b>] sys_futex+0xe3/0x101
  [<ffffffff8027867c>] vfs_write+0x121/0x136
  [<ffffffff80278b60>] sys_write+0x60/0x6e
  [<ffffffff8020bdee>] system_call+0x7e/0x83

ypbind        S 0000000000000000     0  4069      1
  ffff81012bc01eb8 0000000000000082 0000000000000000 ffffffff80243b12
  ffff81012bc01ee8 ffff81012e644e40 ffff81012fcc2000 ffff81012e645048
  0000000300000000 ffffffff802433ae 00000000ffffffff ffffffff80243925
Call Trace:
  [<ffffffff80243b12>] ktime_get_ts+0x1a/0x4e
  [<ffffffff802433ae>] enqueue_hrtimer+0x5c/0x63
  [<ffffffff80243925>] hrtimer_start+0xf2/0x104
  [<ffffffff80448e56>] do_nanosleep+0x46/0x77
  [<ffffffff8024398f>] hrtimer_nanosleep+0x58/0x11e
  [<ffffffff802eb8c9>] _atomic_dec_and_lock+0x39/0x58
  [<ffffffff80243553>] hrtimer_wakeup+0x0/0x22
  [<ffffffff80243aa1>] sys_nanosleep+0x4c/0x62
  [<ffffffff8020bdee>] system_call+0x7e/0x83

acpid         S 0000000000000000     0  4143      1
  ffff81012bd0bb28 0000000000000086 0000000000000000 ffffffff80228f46
  ffffffff80466320 ffff81012c046000 ffff81012fc8a720 ffff81012c046208
  0000000200000000 ffff81012f3ee540 00000000ffffffff ffffffff802cafba
Call Trace:
  [<ffffffff80228f46>] __wake_up+0x38/0x4e
  [<ffffffff802cafba>] journal_stop+0x1e2/0x1ee
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802ada2d>] pde_users_dec+0x10/0x3f
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff80283b12>] do_sys_poll+0x278/0x360
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff80233df8>] current_fs_time+0x1e/0x24
  [<ffffffff8025885f>] file_read_actor+0xa0/0x118
  [<ffffffff8025897e>] find_get_page+0x21/0x50
  [<ffffffff8025a1f4>] __generic_file_aio_write_nolock+0x33e/0x3a8
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff802631f2>] __do_fault+0x370/0x3aa
  [<ffffffff80264b64>] handle_mm_fault+0x424/0x772
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff8026d4d5>] free_pages_and_swap_cache+0x73/0x8f
  [<ffffffff8023838b>] recalc_sigpending+0xe/0x25
  [<ffffffff80283c2c>] sys_poll+0x32/0x3b
  [<ffffffff8020bdee>] system_call+0x7e/0x83

courierlogger S 0000000000000000     0  4147      1
  ffff81012bd63ce8 0000000000000082 0000000000000000 ffff810100000000
  0000000000000000 ffff81012bd64e40 ffffffff805354c0 ffff81012bd65048
  00000000051f7bb8 00000000fffffff7 00000000ffffffff 0000000000004000
Call Trace:
  [<ffffffff8027db22>] pipe_wait+0x66/0x8d
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8027e39f>] pipe_read+0x318/0x39a
  [<ffffffff80277fd4>] do_sync_read+0xc9/0x10c
  [<ffffffff8027ada4>] cp_new_stat+0xe7/0xff
  [<ffffffff802413dc>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8027873b>] vfs_read+0xaa/0x132
  [<ffffffff80278ad7>] sys_read+0x45/0x6e
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000001     0  4148   4147
  ffff81012bc91a18 0000000000000086 ffff81012bc919e0 ffff81012bc91ab8
  0000000000000000 ffff81012bd65560 ffff81012bd64000 ffff81012bd65768
  000000012ebef800 ffff81012bc91a28 ffff81012c436500 0000000100314abf
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802c808a>] __ext3_journal_dirty_metadata+0x1e/0x46
  [<ffffffff80241321>] bit_waitqueue+0x16/0x82
  [<ffffffff80287d9d>] __d_lookup+0xb0/0x100
  [<ffffffff8027ec7d>] do_lookup+0x63/0x1ae
  [<ffffffff8025bb9c>] __rmqueue+0x79/0xe6
  [<ffffffff8025c84b>] get_page_from_freelist+0x2c3/0x35e
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff80264706>] __pte_alloc+0x78/0xb2
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80228561>] update_curr+0xdf/0xfe
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000000     0  4156   4148
  ffff81012c199a18 0000000000000082 0000000000000000 0000000000000000
  0000000000000000 ffff81012bd64000 ffff81012fc52e40 ffff81012bd64208
  0000000100000000 ffff81012c199a28 00000000ffffffff 0000000100314abf
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff802638ab>] do_wp_page+0x4d4/0x545
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000000     0  4157   4148
  ffff81012c6b5a18 0000000000000082 ffff81012c6b59e0 00000000000041ed
  ffff81012c6b5ce8 ffff81012bd64720 ffffffff805354c0 ffff81012bd64928
  000000002e0a4ca8 ffff81012c6b5a28 00000000ffffffff 0000000100314abf
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802810d4>] link_path_walk+0xce/0xe0
  [<ffffffff8025bb9c>] __rmqueue+0x79/0xe6
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000002     0  4158   4148
  ffff81012daf5a18 0000000000000086 0000000000000000 0000000000000000
  0000000000000000 ffff81012e48d560 ffff81012e372000 ffff81012e48d768
  0000000200000000 ffff81012daf5a28 ffff81012f628e00 0000000100314abf
Call Trace:
  [<ffffffff80237ddc>] __mod_timer+0xb6/0xc4
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff8025bb9c>] __rmqueue+0x79/0xe6
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000000     0  4159   4148
  ffff81012bdada18 0000000000000086 0000000000000000 0000000000000000
  0000000000000000 ffff81012e372e40 ffff81012fcc2000 ffff81012e373048
  0000000300000000 ffff81012bdada28 00000000ffffffff 0000000100314abf
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff8025bb9c>] __rmqueue+0x79/0xe6
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

authdaemond   S 0000000000000000     0  4160   4148
  ffff81012bdcda18 0000000000000086 0000000000000000 0000000000000000
  0000000000000000 ffff81012e372000 ffff81012fc8a720 ffff81012e372208
  0000000200000000 ffff81012bdcda28 00000000ffffffff 0000000100314abf
Call Trace:
  [<ffffffff80448a20>] schedule_timeout+0x8a/0xad
  [<ffffffff80237ba4>] process_timeout+0x0/0x5
  [<ffffffff80284031>] do_select+0x3fc/0x45f
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
  [<ffffffff8025c93f>] __alloc_pages+0x59/0x2ae
  [<ffffffff80258ab2>] find_lock_page+0x26/0xa1
  [<ffffffff8025a843>] filemap_fault+0x1e2/0x366
  [<ffffffff80284250>] core_sys_select+0x1bc/0x265
  [<ffffffff802eebcf>] __up_read+0x13/0x8a
  [<ffffffff8021f1d8>] do_page_fault+0x425/0x779
  [<ffffffff80284755>] sys_select+0x15a/0x183
  [<ffffffff8020bdee>] system_call+0x7e/0x83

dbus-daemon   S 0000000000000001     0  4162      1
  ffff81012be09b28 0000000000000082 ffff81012be09ad8 ffffffff80228561
  0000000000001e81 ffff81012c398e40 ffff81012e372720 ffff81012c399048
  000000012e372768 ffff8100052aec08 ffff8100052aeb80 ffffffff802287a3
Call Trace:
  [<ffffffff80228561>] update_curr+0xdf/0xfe
  [<ffffffff802287a3>] enqueue_entity+0x17c/0x1a2
  [<ffffffff804489b4>] schedule_timeout+0x1e/0xad
  [<ffffffff802415e1>] add_wait_queue+0x15/0x44
  [<ffffffff80283b12>] do_sys_poll+0x278/0x360
  [<ffffffff8028451a>] __pollwait+0x0/0xe1
  [<ffffffff802295f6>] default_wake_function+0x0/0xe
sage repeated 3 times
  [<ffffffff803db40b>] sock_aio_read+0xdd/0xec
  [<ffffffff80277fd4>] do_sync_read+0xc9/0x10c
  [<ffffffff802413dc>] autoremo
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
Messages in current thread:
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Neil Brown, (Sun Nov 4, 5:49 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Mon Nov 5, 4:36 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Chuck Ebbert, (Wed Nov 7, 12:39 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Wed Nov 7, 12:48 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Thu Nov 8, 7:42 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Thu Nov 8, 8:44 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Sun Nov 4, 5:51 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Dan Williams, (Mon Nov 5, 2:35 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Jeff Lessem, (Tue Nov 6, 7:18 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Mon Nov 5, 2:35 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Dan Williams, (Mon Nov 5, 8:19 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Tue Nov 6, 6:19 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Dan Williams, (Tue Nov 6, 9:25 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Wed Nov 7, 7:20 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Jeff Lessem, (Wed Nov 7, 1:00 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Carlos Carvalho, (Thu Nov 8, 5:40 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Fri Nov 9, 5:14 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Fabiano Silva, (Fri Nov 9, 10:09 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Bill Davidsen, (Thu Nov 8, 1:45 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Dan Williams, (Thu Nov 8, 2:02 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Jeff Lessem, (Fri Nov 9, 4:36 pm)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Tue Nov 6, 7:29 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Tue Nov 6, 7:39 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, Justin Piszcz, (Tue Nov 6, 7:42 am)
Re: 2.6.23.1: mdadm/raid5 hung/d-state, BERTRAND Joël, (Tue Nov 6, 8:20 am)