> From:
linux-nfs-owner@vger.kernel.org
> [mailto:linux-nfs-owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> Sent: Friday, June 13, 2008 3:16 PM
> To: Weathers, Norman R.
> Cc: Jeff Layton; linux-kernel@vger.kernel.org;
> linux-nfs@vger.kernel.org; Neil Brown
> Subject: Re: CONFIG_DEBUG_SLAB_LEAK omits size-4096 and larger?
>
> On Thu, Jun 12, 2008 at 02:54:09PM -0500, Weathers, Norman R. wrote:
> >
> >
> > > -----Original Message-----
> > > From:
linux-nfs-owner@vger.kernel.org
> > > [mailto:linux-nfs-owner@vger.kernel.org] On Behalf Of J.
> Bruce Fields
> > > Sent: Wednesday, June 11, 2008 5:55 PM
> > > To: Weathers, Norman R.
> > > Cc: Jeff Layton; linux-kernel@vger.kernel.org;
> > >
linux-nfs@vger.kernel.org
> > > Subject: Re: CONFIG_DEBUG_SLAB_LEAK omits size-4096 and larger?
> > >
> > > On Wed, Jun 11, 2008 at 05:46:13PM -0500, Weathers,
> Norman R. wrote:
> > > > I will try and get it patched and retested, but it may be a
> > > day or two
> > > > before I can get back the information due to production jobs now
> > > > running. Once they finish up, I will get back with the info.
> > >
> > > Understood.
> > >
> >
> >
> > I was able to get my big user to cooperate and let me in to
> be able to
> > get the information that you were needing. The full output from the
> > /proc/slab_allocator file is at
> >
http://www.shashi-weathers.net/linux/cluster/NFS_DEBUG_2 . The 16
> > thread case is very interesting. Also, there is a small
> txt file in the
> > directory that has some rpc errors, but I imagine the way that I am
> > running the box (oversubscribed threads) has more to do with the rpc
> > errors than anything else. For those of you wanting the gist of the
> > story, the size-4096 slab has the following very large allocation:
> >
> > size-4096: 2 sys_init_module+0x140b/0x1980
> > size-4096: 1 __vmalloc_area_node+0x188/0x1b0
> > size-4096: 1 seq_read+0x1d9/0x2e0
> > size-4096: 1 slabstats_open+0x2b/0x80
> > size-4096: 5 vc_allocate+0x167/0x190
> > size-4096: 3 input_allocate_device+0x12/0x80
> > size-4096: 1 hid_add_field+0x122/0x290
> > size-4096: 9 reqsk_queue_alloc+0x5f/0xf0
> > size-4096: 1846825 __alloc_skb+0x7d/0x170
> > size-4096: 3 alloc_netdev+0x33/0xa0
> > size-4096: 10 neigh_sysctl_register+0x52/0x2b0
> > size-4096: 5 devinet_sysctl_register+0x28/0x110
> > size-4096: 1 pidmap_init+0x15/0x60
> > size-4096: 1 netlink_proto_init+0x44/0x190
> > size-4096: 1 ip_rt_init+0xfd/0x2f0
> > size-4096: 1 cipso_v4_init+0x13/0x70
> > size-4096: 3 journal_init_revoke+0xe7/0x270 [jbd]
> > size-4096: 3 journal_init_revoke+0x18a/0x270 [jbd]
> > size-4096: 2 journal_init_inode+0x84/0x150 [jbd]
> > size-4096: 2 bnx2_alloc_mem+0x18/0x1f0 [bnx2]
> > size-4096: 1 joydev_connect+0x53/0x390 [joydev]
> > size-4096: 13 kmem_alloc+0xb3/0x100 [xfs]
> > size-4096: 5 addrconf_sysctl_register+0x31/0x130 [ipv6]
> > size-4096: 7 rpc_clone_client+0x84/0x140 [sunrpc]
> > size-4096: 3 rpc_create+0x254/0x4d0 [sunrpc]
> > size-4096: 16 __svc_create_thread+0x53/0x1f0 [sunrpc]
> > size-4096: 16 __svc_create_thread+0x72/0x1f0 [sunrpc]
> > size-4096: 1 nfsd_racache_init+0x2e/0x140 [nfsd]
> >
> > The big one seems to be the __alloc_skb. (This is with 16
> threads, and
> > it says that we are using up somewhere between 12 and 14 GB
> of memory,
> > about 2 to 3 gig of that is disk cache). If I were to put anymore
> > threads out there, the server would become almost
> unresponsive (it was
> > bad enough as it was).
> >
> > At the same time, I also noticed this:
> >
> > skbuff_fclone_cache: 1842524 __alloc_skb+0x50/0x170
> >
> > Don't know for sure if that is meaningful or not....
>
> OK, so, starting at net/core/skbuff.c, this means that this memory was
> allocated by __alloc_skb() calls with something nonzero in the third
> ("fclone") argument. The only such caller is alloc_skb_fclone().
> Callers of alloc_skb_fclone() include:
>
> sk_stream_alloc_skb:
> do_tcp_sendpages
> tcp_sendmsg
> tcp_fragment
> tso_fragment
