On 10/13/07, Andrew Morton <akpm@linux-foundation.org> wrote:
quoted text
> Please send /proc/meminfo and /proc/slabinfo after the leak has been
> happening for a while.
>
> Sometimes `echo m > /proc/sysrq_trigger ; dmesg -s 1000000' will
> provide useful info.

As I'm using SLUB there is no /proc/slabinfo.

I have attached any files that looked remotely related.
After I have seen ~2Gb leaked again, I took the first set of outputs.
atop showed at that point:
MEM | tot    3.9G | free  648.7M | cache 814.5M | buff    0.0M | slab 131.5M |
SWP | tot    9.3G | free    9.3G |              | vmcom  16.7M | vmlim 11.3G |

free showed:
             total       used       free     shared    buffers     cached
Mem:       4061808    3396852     664956          0         28     834012
-/+ buffers/cache:    2562812    1498996
Swap:      9775416         60    9775356

I then tried to build 2.6.23-mm1 with PAGE_OWNER enabled. During
modpost the system started swapping like mad, I needed to abort the
build, but was still able to shut down the system normal.
At that point I took the secound set of outputs (files called *.end)
free showed at that time:
             total       used       free     shared    buffers     cached
Mem:       4061808    3777084     284724          0          0       2032
-/+ buffers/cache:    3775052     286756
Swap:      9775416       4904    9770512

The slubinfo-files are the output of Documentation/vm/slabinfo.c

quoted text
> The page-owner code can pinpoint a leak source.  See
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.23/2.6.23-mm1/broken...
>
> Enable CONFIG_DEBUG_SLAB_LEAK, check out /proc/slab_allocators

Did that. The output of /proc/page_owner is ~350Mb, gzipped still ~7Mb.

Taking only the first line from each stackdump it shows the following counts:
     73 [0xffffffff8020a13e] __switch_to+430
      6 [0xffffffff8020a236] __switch_to+678
      8 [0xffffffff8020ac30] default_idle+0
      1 [0xffffffff8020ba3f] sys_rt_sigreturn+879
      3 [0xffffffff8020bbbe] system_call+126
    115 [0xffffffff802132e1] dma_alloc_pages+177
      1 [0xffffffff802191fb] __smp_call_function_mask+235
      1 [0xffffffff8021970b] flush_tlb_page+75
      8 [0xffffffff8021c1b7] mp_register_gsi+71
      3 [0xffffffff8021c25b] mp_register_gsi+235
     26 [0xffffffff8021ec6d] flush_gart+13
      2 [0xffffffff8021ec7f] flush_gart+31
     28 [0xffffffff8021ecda] gart_map_simple+58
      1 [0xffffffff8021f048] gart_map_sg+680
     15 [0xffffffff8021f2af] k8_flush_garts+191
      4 [0xffffffff8022078b] do_page_fault+683
      1 [0xffffffff80220cc8] ioremap_change_attr+88
      1 [0xffffffff80220db1] __ioremap+225
      2 [0xffffffff8022123d] __change_page_attr+525
      1 [0xffffffff80222d2e] twofish_dec_blk+1557
     20 [0xffffffff80227e15] load_elf32_binary+181
    123 [0xffffffff8022c223] enqueue_task+19
      2 [0xffffffff8022c2a2] activate_task+50
     31 [0xffffffff8022c3f4] source_load+52
     54 [0xffffffff8022c464] target_load+52
     25 [0xffffffff8022c5b6] move_tasks+118
      4 [0xffffffff8022c684] __wake_up_common+20
     23 [0xffffffff8022c6ca] __wake_up_common+90
     38 [0xffffffff8022cb31] update_curr+97
      5 [0xffffffff8022cb78] update_curr+168
      1 [0xffffffff8022cd35] balance_tasks+117
     12 [0xffffffff8022cde3] balance_tasks+291
      2 [0xffffffff8022cee8] load_balance_fair+88
      3 [0xffffffff8022cff0] load_balance_next_fair+0
    137 [0xffffffff8022d04d] __dequeue_entity+61
      4 [0xffffffff8022d083] set_next_entity+35
   1101 [0xffffffff8022d0f5] pick_next_task_fair+69
    319 [0xffffffff8022d3e3] __wake_up+67
      6 [0xffffffff8022d4bc] task_rq_lock+76
     17 [0xffffffff8022d693] try_to_wake_up+99
    211 [0xffffffff8022d910] default_wake_function+0
    575 [0xffffffff8022df58] find_busiest_group+472
      1 [0xffffffff80230844] finish_task_switch+164
    217 [0xffffffff802318f4] enqueue_task_fair+132
    335 [0xffffffff80231981] dequeue_task_fair+81
      8 [0xffffffff80231ca4] check_preempt_wakeup+212
      6 [0xffffffff80231cba] check_preempt_wakeup+234
      2 [0xffffffff8023252c] dup_fd+44
     31 [0xffffffff8023273b] dup_fd+571
      1 [0xffffffff80233393] copy_process+1811
      1 [0xffffffff8023549e] release_console_sem+494
      5 [0xffffffff80235b0e] printk+78
      1 [0xffffffff80237dd6] do_wait+1558
      1 [0xffffffff8023a2a6] on_each_cpu+38
      5 [0xffffffff8023e844] lock_timer_base+52
      1 [0xffffffff8023e8d2] try_to_del_timer_sync+82
      5 [0xffffffff8023e8f0] del_timer_sync+16
     10 [0xffffffff8023ea12] __mod_timer+194
     65 [0xffffffff8023ec4d] mod_timer+29
      1 [0xffffffff8023ecee] del_timer+110
      2 [0xffffffff8023ff98] __dequeue_signal+344
      1 [0xffffffff8024003e] signal_wake_up+30
      3 [0xffffffff80242090] sys_rt_sigprocmask+128
      2 [0xffffffff802457f9] call_usermodehelper_exec+153
    525 [0xffffffff802462c0] __queue_work+48
     28 [0xffffffff80246347] queue_work+55
      4 [0xffffffff802467b0] worker_thread+0
      1 [0xffffffff80246845] worker_thread+149
    132 [0xffffffff80247e99] __kernel_text_address+9
    151 [0xffffffff80247eaa] __kernel_text_address+26
     62 [0xffffffff8024a2c0] bit_waitqueue+16
     23 [0xffffffff8024a398] __wake_up_bit+40
     24 [0xffffffff8024a3b8] wake_up_bit+24
    254 [0xffffffff8024a3e0] autoremove_wake_function+0
    106 [0xffffffff8024a3e9] autoremove_wake_function+9
      1 [0xffffffff8024a6b9] remove_wait_queue+41
      2 [0xffffffff8024f245] sysctl_check_lookup+197
      4 [0xffffffff8024f554] sysctl_check_table+756
      1 [0xffffffff8024f5bc] sysctl_check_table+860
      1 [0xffffffff80254603] get_futex_key+83
     21 [0xffffffff8025595c] futex_wake+204
      6 [0xffffffff80257eeb] __module_text_address+91
   1221 [0xffffffff80263734] file_read_actor+324
     89 [0xffffffff80263821] find_get_page+33
      1 [0xffffffff80263850] find_get_page+80
   2009 [0xffffffff802638bf] __lock_page+95
     57 [0xffffffff802638e0] sync_page+0
      1 [0xffffffff80263a92] find_lock_page+2
    206 [0xffffffff80263ab9] find_lock_page+41
      1 [0xffffffff80263b03] find_lock_page+115
  13762 [0xffffffff80263cda] add_to_page_cache+186
  33836 [0xffffffff80263d93] __grab_cache_page+99
      4 [0xffffffff802640f9] do_generic_mapping_read+489
   7431 [0xffffffff802641b2] do_generic_mapping_read+674
      1 [0xffffffff8026421d] do_generic_mapping_read+781
    678 [0xffffffff802643a6] find_or_create_page+54
  12884 [0xffffffff802643b6] find_or_create_page+70
      1 [0xffffffff80265bbe] generic_file_aio_read+254
    122 [0xffffffff80265e13] filemap_fault+419
    227 [0xffffffff80266240] mempool_free_slab+0
 354042 [0xffffffff80266373] mempool_alloc+83
    113 [0xffffffff802689e8] __rmqueue_smallest+216
     76 [0xffffffff80268a39] __rmqueue_smallest+297
      3 [0xffffffff80269043] free_pages_bulk+611
      6 [0xffffffff80269253] __pagevec_free+35
    249 [0xffffffff802693fe] __get_free_pages+14
   2002 [0xffffffff80269457] get_zeroed_page+23
    221 [0xffffffff802694bf] __rmqueue+47
      1 [0xffffffff802695dc] __rmqueue+332
     40 [0xffffffff8026977e] rmqueue_bulk+110
     41 [0xffffffff80269b30] get_page_from_freelist+896
   7323 [0xffffffff80269b3b] get_page_from_freelist+907
      2 [0xffffffff80269d13] get_page_from_freelist+1379
     11 [0xffffffff80269d26] get_page_from_freelist+1398
      6 [0xffffffff80269eb2] __alloc_pages+2
      1 [0xffffffff80269eba] __alloc_pages+10
      1 [0xffffffff80269ec2] __alloc_pages+18
      1 [0xffffffff8026a0fe] __alloc_pages+590
     24 [0xffffffff8026a205] __alloc_pages+853
 105887 [0xffffffff8026c246] __do_page_cache_readahead+230
     15 [0xffffffff8026ca6f] release_pages+367
      6 [0xffffffff8026cb84] __pagevec_lru_add_active+212
    638 [0xffffffff8026cc79] __pagevec_lru_add+201
      1 [0xffffffff8026ccf4] activate_page+100
    165 [0xffffffff8026cd3f] activate_page+175
      2 [0xffffffff8026d07b] mark_page_accessed+43
     78 [0xffffffff8026d1d7] pagevec_lookup+23
     79 [0xffffffff8026d9a8] truncate_inode_pages_range+392
    128 [0xffffffff80270f62] zone_statistics+66
   2426 [0xffffffff80270f9d] zone_statistics+125
    236 [0xffffffff80272c60] __do_fault+480
      1 [0xffffffff80272c85] __do_fault+517
   5233 [0xffffffff80272cb8] __do_fault+568
     32 [0xffffffff80272d62] __do_fault+738
   4744 [0xffffffff80273233] do_wp_page+211
     97 [0xffffffff80273cd5] unmap_vmas+933
      1 [0xffffffff802743dc] unmap_mapping_range+140
     43 [0xffffffff80274978] handle_mm_fault+456
 103412 [0xffffffff80274ae4] handle_mm_fault+820
      4 [0xffffffff80274baa] handle_mm_fault+1018
     15 [0xffffffff80274d81] handle_mm_fault+1489
      2 [0xffffffff80277f6a] unmap_region+250
      5 [0xffffffff802783b4] find_extend_vma+36
      1 [0xffffffff8027886e] vma_adjust+606
      4 [0xffffffff80278ef6] do_munmap+662
    732 [0xffffffff80279292] vma_link+130
      1 [0xffffffff80279c8e] mmap_region+750
    343 [0xffffffff8027cb45] page_dup_rmap+21
   3613 [0xffffffff8027db83] __vmalloc_area_node+371
   4096 [0xffffffff8027dbc8] __vmalloc_area_node+440
      1 [0xffffffff80283348] zonelist_policy+8
      8 [0xffffffff8028339a] zonelist_policy+90
      1 [0xffffffff802833a2] zonelist_policy+98
      3 [0xffffffff802833c3] zonelist_policy+131
      1 [0xffffffff802833e3] zonelist_policy+163
     12 [0xffffffff802833ef] zonelist_policy+175
      1 [0xffffffff802833f8] zonelist_policy+184
   1040 [0xffffffff80284256] alloc_page_interleave+86
      9 [0xffffffff802842cd] alloc_page_interleave+205
      4 [0xffffffff80285411] alloc_pages_current+1
      2 [0xffffffff80285423] alloc_pages_current+19
     19 [0xffffffff8028542a] alloc_pages_current+26
     13 [0xffffffff8028542d] alloc_pages_current+29
      3 [0xffffffff80285452] alloc_pages_current+66
      4 [0xffffffff8028545b] alloc_pages_current+75
      2 [0xffffffff8028545f] alloc_pages_current+79
      1 [0xffffffff80285464] alloc_pages_current+84
  10948 [0xffffffff80287c7a] shmem_getpage+394
     37 [0xffffffff80287d30] shmem_getpage+576
     51 [0xffffffff8028a66b] add_partial+43
     26 [0xffffffff8028bddc] new_slab+172
     26 [0xffffffff8028befa] new_slab+458
  41349 [0xffffffff8028bf67] new_slab+567
     62 [0xffffffff8028c008] get_partial_node+104
   1332 [0xffffffff8028c217] __slab_free+487
    157 [0xffffffff8028c4f9] __slab_alloc+441
     34 [0xffffffff8028c561] __slab_alloc+545
      2 [0xffffffff8028c67f] __slab_alloc+831
    341 [0xffffffff8028c697] __slab_alloc+855
      4 [0xffffffff8028cbfb] kmem_cache_alloc+155
      1 [0xffffffff8028dfbe] sysfs_slab_alias+46
      1 [0xffffffff802905d4] filp_close+84
     19 [0xffffffff80290687] get_unused_fd_flags+119
      1 [0xffffffff80291ae0] vfs_statfs_native+64
      1 [0xffffffff80292449] do_sync_write+217
      2 [0xffffffff802958c0] chrdev_open+0
      2 [0xffffffff802961e7] sys_newlstat+39
     15 [0xffffffff802961f6] sys_newlstat+54
      1 [0xffffffff80296387] sys_newstat+39
     71 [0xffffffff80296396] sys_newstat+54
      2 [0xffffffff80298f70] pipe_ioctl+128
     11 [0xffffffff802998e6] pipe_write+742
     23 [0xffffffff80299e5f] pipe_read+687
   1225 [0xffffffff80299eae] pipe_read+766
     10 [0xffffffff8029a3f3] generic_permission+99
      3 [0xffffffff8029a869] __follow_mount+41
   8501 [0xffffffff8029aabf] do_lookup+143
     42 [0xffffffff8029c325] deny_write_access+21
     12 [0xffffffff8029c3bb] lookup_one_len+91
     10 [0xffffffff8029c5d6] __link_path_walk+134
      5 [0xffffffff8029c829] __link_path_walk+729
    508 [0xffffffff8029d056] __link_path_walk+2822
      3 [0xffffffff8029d269] __link_path_walk+3353
    486 [0xffffffff8029d460] link_path_walk+128
     29 [0xffffffff8029d739] do_path_lookup+137
     15 [0xffffffff8029ff94] do_ioctl+132
      7 [0xffffffff802a08e1] free_poll_entry+17
      1 [0xffffffff802a0c68] do_sys_poll+744
      1 [0xffffffff802a14f2] core_sys_select+562
     10 [0xffffffff802a1870] __pollwait+0
     77 [0xffffffff802a4b21] d_rehash+33
     48 [0xffffffff802a4db5] d_kill+53
     23 [0xffffffff802a588f] __d_lookup+175
      1 [0xffffffff802a5908] __d_lookup+296
    472 [0xffffffff802a5ef6] dput+38
      8 [0xffffffff802a725f] __iget+79
      1 [0xffffffff802a796e] alloc_inode+318
      2 [0xffffffff802a7ed4] iget_locked+68
      1 [0xffffffff802a863c] new_inode+92
      4 [0xffffffff802a88af] touch_atime+31
      3 [0xffffffff802a8978] touch_atime+232
      1 [0xffffffff802a8e52] notify_change+466
     29 [0xffffffff802aa80f] mntput_no_expire+31
     76 [0xffffffff802ac485] mnt_drop_write+37
      2 [0xffffffff802ac8cd] mnt_want_write+45
      7 [0xffffffff802afd0d] seq_open+77
      1 [0xffffffff802b33b5] __mark_inode_dirty+101
  68796 [0xffffffff802b7d7b] __set_page_dirty+219
      1 [0xffffffff802b7e00] mark_buffer_dirty+32
      1 [0xffffffff802b7edc] __block_commit_write+92
      1 [0xffffffff802b7f08] __block_commit_write+136
      1 [0xffffffff802b7f10] __block_commit_write+144
    311 [0xffffffff802b7f15] __block_commit_write+149
      1 [0xffffffff802b8346] unmap_underlying_metadata+6
      3 [0xffffffff802ba04e] generic_write_end+94
      2 [0xffffffff802bc347] check_disk_change+39
      5 [0xffffffff802bffb0] mpage_end_io_read+0
      3 [0xffffffff802d6ed2] proc_alloc_inode+18
      1 [0xffffffff802d6f40] pde_users_dec+16
      1 [0xffffffff802d7860] proc_reg_open+0
      1 [0xffffffff802d94fd] proc_flush_task_mnt+285
    132 [0xffffffff802d9b10] proc_fd_instantiate+0
      3 [0xffffffff802db798] create_proc_entry+88
      1 [0xffffffff802dc1a7] proc_lookup+247
      1 [0xffffffff802de5a8] show_stat+1064
      1 [0xffffffff802e289a] sysfs_add_file+106
      8 [0xffffffff802e2c30] sysfs_ilookup_test+0
      2 [0xffffffff802e2ed9] sysfs_find_dirent+41
      1 [0xffffffff802e327d] sysfs_addrm_finish+29
      1 [0xffffffff802e3641] sysfs_create_dir+49
    272 [0xffffffff802e8cc3] nfs_lookup_revalidate+579
      1 [0xffffffff802e9d41] nfs_do_access+241
   1615 [0xffffffff802ea2f0] nfs_readdir_filler+0
      1 [0xffffffff802f9ded] _nfs4_proc_lookupfh+125
     14 [0xffffffff802ff415] decode_op_hdr+21
      1 [0xffffffff8032190d] xfs_alloc_vextent+637
      1 [0xffffffff80321c15] xfs_alloc_log_block+69
      5 [0xffffffff80325622] xfs_attr_leaf_get+98
     28 [0xffffffff8032575b] xfs_attr_fetch+235
      8 [0xffffffff8032bec1] xfs_bmap_last_offset+273
      1 [0xffffffff8032c5a5] xfs_bmap_search_multi_extents+117
      8 [0xffffffff8032c5c6] xfs_bmap_search_multi_extents+150
     92 [0xffffffff8032c5df] xfs_bmap_search_multi_extents+175
    569 [0xffffffff8032c6bd] xfs_bmap_search_extents+109
      5 [0xffffffff8033358c] xfs_bmapi+540
      1 [0xffffffff80333a54] xfs_bmapi+1764
      1 [0xffffffff803393b5] xfs_btree_init_cursor+69
     49 [0xffffffff8033a084] xfs_buf_item_init+100
      1 [0xffffffff8033ab3f] xfs_da_brelse+111
   5066 [0xffffffff8033ad57] xfs_da_buf_make+231
      7 [0xffffffff8033ba64] xfs_da_do_buf+1444
      5 [0xffffffff8033e76b] xfs_dir_lookup+299
     12 [0xffffffff8033e79f] xfs_dir_lookup+351
      4 [0xffffffff8033eb9b] xfs_dir_createname+379
      2 [0xffffffff8033ec37] xfs_dir_init+103
      1 [0xffffffff8033ee1a] xfs_dir2_block_lookup_int+90
      1 [0xffffffff8033eeb9] xfs_dir2_block_lookup_int+249
      1 [0xffffffff803416d4] xfs_dir2_leaf_lookup_int+516
      1 [0xffffffff80343234] xfs_dir2_leafn_order+4
     63 [0xffffffff8034e2bc] xfs_iunlock+92
     24 [0xffffffff8034e31c] xfs_iunlock+188
      1 [0xffffffff8034ed57] xfs_iget_core+1063
    511 [0xffffffff8034f96b] xfs_iext_bno_to_ext+155
      5 [0xffffffff80350869] xfs_iext_add+361
     11 [0xffffffff803512cd] xfs_itobp+381
      5 [0xffffffff80351657] xfs_itruncate_finish+679
    810 [0xffffffff80351a22] xfs_iformat_local+258
    886 [0xffffffff803528d1] xfs_iformat_extents+177
      1 [0xffffffff803529c7] xfs_iformat_extents+423
      1 [0xffffffff80354a7e] xfs_inode_item_init+30
   2924 [0xffffffff803555b6] xfs_iomap+406
      3 [0xffffffff80355e55] xfs_iomap_write_delay+389
    140 [0xffffffff8035a4d6] xlog_state_release_iclog+38
    195 [0xffffffff8035a59b] xlog_state_get_iclog_space+59
      4 [0xffffffff8035aac8] xlog_write+920
    248 [0xffffffff8035aea0] xfs_log_release_iclog+16
      1 [0xffffffff803620a9] xfs_icsb_sync_counters_flags+121
      4 [0xffffffff80362931] xfs_icsb_modify_counters+113
      1 [0xffffffff80362b01] xfs_mod_incore_sb_batch+161
      3 [0xffffffff8036600f] xfs_trans_unreserve_and_mod_sb+207
     10 [0xffffffff80366967] _xfs_trans_commit+823
      2 [0xffffffff80366bbb] xfs_trans_unlocked_item+59
   1016 [0xffffffff80367654] xfs_trans_read_buf+100
      1 [0xffffffff80367657] xfs_trans_read_buf+103
     12 [0xffffffff80367786] xfs_trans_read_buf+406
      1 [0xffffffff80367c4d] xfs_trans_ijoin+61
      5 [0xffffffff80367e94] xfs_trans_unlock_chunk+132
      1 [0xffffffff8036b284] xfs_statvfs+292
      2 [0xffffffff80370db3] xfs_release+339
   1143 [0xffffffff80371592] xfs_readlink+130
    366 [0xffffffff80371f6c] kmem_zone_alloc+92
    128 [0xffffffff803720bd] kmem_alloc+93
   1812 [0xffffffff80374156] xfs_buf_iowait+70
      1 [0xffffffff80374a0e] xfs_buf_rele+46
      1 [0xffffffff80374cbe] xfs_buf_delwri_queue+94
   4252 [0xffffffff803751fd] _xfs_buf_lookup_pages+637
     31 [0xffffffff80376170] _xfs_buf_find+256
    666 [0xffffffff80376357] xfs_buf_get_flags+183
    118 [0xffffffff80376442] xfs_buf_read_flags+18
      2 [0xffffffff80376495] xfs_buf_read_flags+101
    278 [0xffffffff803790d5] xfs_vn_permission+21
     24 [0xffffffff80379872] xfs_vn_lookup+66
     21 [0xffffffff8037a9c5] xfs_read+405
     14 [0xffffffff80386c60] dummy_inode_alloc_security+0
      1 [0xffffffff80386d10] dummy_inode_readlink+0
     36 [0xffffffff80386d30] dummy_inode_permission+0
      1 [0xffffffff80388e60] cap_capable+0
      1 [0xffffffff80388e6f] cap_capable+15
      1 [0xffffffff80388fca] cap_syslog+42
     32 [0xffffffff8038e045] crypto_cbc_init_tfm+69
     49 [0xffffffff8039b14d] blk_recount_segments+61
      5 [0xffffffff8039c230] generic_make_request+464
      2 [0xffffffff8039c486] submit_bio+102
     23 [0xffffffff8039f108] generic_unplug_device+24
      2 [0xffffffff803a16ec] exact_lock+12
      1 [0xffffffff803a3e70] cfq_queue_empty+0
      5 [0xffffffff803a8908] _atomic_dec_and_lock+72
      5 [0xffffffff803a8f85] ida_get_new_above+165
     63 [0xffffffff803a90f9] idr_get_new_above_int+73
      1 [0xffffffff803a911b] idr_get_new+11
      1 [0xffffffff803a9a50] kobject_release+0
      2 [0xffffffff803a9ac2] kobject_get+18
      7 [0xffffffff803aa006] kobject_set_name+278
      2 [0xffffffff803aa73a] kobject_uevent_env+298
      2 [0xffffffff803ab2e3] prio_tree_insert+35
      5 [0xffffffff803ab33c] prio_tree_insert+124
      4 [0xffffffff803ab4be] prio_tree_insert+510
     18 [0xffffffff803abeb2] radix_tree_node_alloc+50
      1 [0xffffffff803ac6fa] radix_tree_delete+426
     14 [0xffffffff803ac94d] rb_insert_color+141
     14 [0xffffffff803ac99e] rb_insert_color+222
   2699 [0xffffffff803ad047] __up_read+39
      4 [0xffffffff803ad0e7] __up_write+39
     10 [0xffffffff803ae2ea] number+682
     20 [0xffffffff803aee3b] vsnprintf+811
      3 [0xffffffff803aee7b] vsnprintf+875
      4 [0xffffffff803af241] sprintf+81
      1 [0xffffffff803af304] snprintf+68
      1 [0xffffffff803af535] clear_page_c+5
      4 [0xffffffff803af537] clear_page_c+7
     11 [0xffffffff803c8218] vgacon_init+200
      1 [0xffffffff803c8981] vgacon_scroll+289
      3 [0xffffffff803c8e80] vgacon_cursor+0
      2 [0xffffffff803c9450] acpi_os_release_object+9
      1 [0xffffffff803ce3b5] acpi_ev_address_space_dispatch+360
      1 [0xffffffff803d26ba] acpi_ex_field_datum_io+231
      2 [0xffffffff803d282e] acpi_ex_extract_from_field+224
     11 [0xffffffff803d7412] acpi_ns_lookup+578
      1 [0xffffffff803d7858] acpi_ns_search_one_scope+28
     10 [0xffffffff803d7944] acpi_ns_search_and_enter+196
      1 [0xffffffff803ddddd] acpi_ut_evaluate_object+390
      1 [0xffffffff803de372] acpi_ut_valid_acpi_name+39
      1 [0xffffffff803df43c] acpi_ut_update_ref_count+82
      2 [0xffffffff803df564] acpi_ut_update_object_reference+219
      1 [0xffffffff803dfbcd] acpi_ut_allocate_object_desc_dbg+52
      1 [0xffffffff803dfe89] acpi_ut_create_generic_state+65
      1 [0xffffffff803e4250] acpi_pci_link_get_current+116
     28 [0xffffffff803f14c8] secure_ip_id+72
      2 [0xffffffff803f20e1] tty_ldisc_deref+81
      1 [0xffffffff803f30c3] init_dev+675
      1 [0xffffffff803f312d] init_dev+781
      1 [0xffffffff8040e6db] uart_console_write+43
      1 [0xffffffff8041224c] serial8250_start_tx+124
      1 [0xffffffff80431560] scsi_done+0
      1 [0xffffffff804352d6] scsi_error_handler+102
      1 [0xffffffff80439041] scsi_target_reap+49
      5 [0xffffffff80440cb3] ata_dev_init+83
      6 [0xffffffff804424f8] ata_qc_issue+312
      1 [0xffffffff80449376] ata_scsi_qc_complete+246
      2 [0xffffffff8044c79f] ata_exec_command+31
     22 [0xffffffff8044cc6c] ata_tf_load+236
      1 [0xffffffff80466b87] dma_trm_flush+359
      1 [0xffffffff8046a445] ether1394_data_handler+661
      1 [0xffffffff8047388b] usb_start_wait_urb+139
      1 [0xffffffff80473b4f] usb_control_msg+239
    669 [0xffffffff804b5c66] __release_stripe+198
     96 [0xffffffff804b7f49] grow_one_stripe+153
      5 [0xffffffff804c115d] md_import_device+125
      1 [0xffffffff804c18e4] mddev_find+372
     12 [0xffffffff804c9103] read_sb_page+35
    191 [0xffffffff804cb74d] __map_bio+61
    135 [0xffffffff804cc4ba] __split_bio+906
     13 [0xffffffff804cc8ad] dm_request+269
     44 [0xffffffff804ce33e] dm_table_any_congested+46
     23 [0xffffffff804d2589] crypt_map+105
      1 [0xffffffff804ddf85] snd_card_file_remove+101
      1 [0xffffffff804f6f4d] snd_hda_codec_write+109
      8 [0xffffffff8050ab1c] pci_conf1_read+204
     12 [0xffffffff8050d8e1] sockfd_lookup_light+65
     35 [0xffffffff8050e1bb] sock_sendmsg+203
     14 [0xffffffff8051044f] sk_reset_timer+15
    203 [0xffffffff80513f82] skb_dequeue+82
      1 [0xffffffff80515006] __alloc_skb+150
     18 [0xffffffff8051687d] memcpy_toiovec+77
      3 [0xffffffff805168cf] verify_iovec+47
      2 [0xffffffff80519630] dev_seq_open+96
      2 [0xffffffff8051cd1d] netdev_run_todo+333
      3 [0xffffffff8051d2fa] dev_queue_xmit+170
      1 [0xffffffff8052a2e8] __qdisc_run+152
      1 [0xffffffff8053c39f] ip_output+447
      5 [0xffffffff80543710] tcp_sendmsg+1904
      1 [0xffffffff80580318] unix_release_sock+408
      1 [0xffffffff805813e3] unix_stream_sendmsg+483
      2 [0xffffffff8058e069] xprt_end_transmit+57
      2 [0xffffffff80593207] rpc_wake_up_task+103
     17 [0xffffffff80593993] rpc_wake_up_next+147
      2 [0xffffffff805ac844] wait_for_common+292
     57 [0xffffffff805acb3f] __switch_to_end+24
     12 [0xffffffff805aced9] __switch_to_end+946
  11861 [0xffffffff805ad128] io_schedule+40
      8 [0xffffffff805ad235] schedule_timeout+149
      9 [0xffffffff805ad2d4] __wait_on_bit_lock+36
      2 [0xffffffff805ad315] __wait_on_bit_lock+101
     15 [0xffffffff805ad43b] __wait_on_bit+107
  46337 [0xffffffff805ae2c2] __down_read+18
    206 [0xffffffff805ae388] __down_failed+53
     17 [0xffffffff805ae725] __down+229
      2 [0xffffffff805ae8ec] _read_lock_irq+12
     25 [0xffffffff805aeaed] error_exit+0
      1 [0xffffffff807c2000] _sinittext+0
      1 [0xffffffff88047000] __videobuf_mmap_free+0
      1 [0xffffffff88096000] tda9855_volume+0
      1 [0xffffffff880bb000] w83792d_detach_client+0
      1 [0xffffffff880da570] ndisc_ifinfo_sysctl_change+0
      1 [0xffffffff8810e000] nfsd_vers+0
      1 [0xffffffff8814fa1a] drm_addmap_core+1098
      2 [0xffffffff88157650] drm_sg_alloc_ioctl+0


Torsten