Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?

Previous thread: [PATCH] i915: make vbl interrupts work properly on i965g/gm by Dave Airlie on Thursday, September 27, 2007 - 10:05 pm. (6 messages)

Next thread: Re: [patch 2/2] VFS: allow filesystem to override mknod capability checks by Neil Brown on Thursday, September 27, 2007 - 10:40 pm. (2 messages)
To: <linux-kernel@...>, <netdev@...>
Cc: <linux@...>
Date: Thursday, September 27, 2007 - 10:06 pm

Uniprocessor Althlon 64, 64-bit kernel, 2G ECC RAM,
2.6.23-rc8 + linuxpps (5.0.0) + ip1000a driver.
(patch from http://marc.info/?l=linux-netdev&m=118980588419882)

After a few hours of operation, ntp loses the ability to send packets.
sendto() returns -EAGAIN to everything, including the 24-byte UDP packet
that is a response to ntpq.

-EAGAIN on a sendto() makes me think of memory problems, so here's
meminfo at the time:

### FAILED state ###
# cat /proc/meminfo
MemTotal: 2059384 kB
MemFree: 15332 kB
Buffers: 665608 kB
Cached: 18212 kB
SwapCached: 0 kB
Active: 380384 kB
Inactive: 355020 kB
SwapTotal: 5855208 kB
SwapFree: 5854552 kB
Dirty: 28504 kB
Writeback: 0 kB
AnonPages: 51608 kB
Mapped: 11852 kB
Slab: 1285348 kB
SReclaimable: 152968 kB
SUnreclaim: 1132380 kB
PageTables: 3888 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 6884900 kB
Committed_AS: 590528 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 265628 kB
VmallocChunk: 34359472059 kB

Killing and restarting ntpd gets it running again for a few hours.
Here's after about two hours of successful operation. (I'll try to
remember to run slabinfo before killing ntpd next time.)

### WORKING state ###
# cat /proc/meminfo
MemTotal: 2059384 kB
MemFree: 20252 kB
Buffers: 242688 kB
Cached: 41556 kB
SwapCached: 200 kB
Active: 285012 kB
Inactive: 147348 kB
SwapTotal: 5855208 kB
SwapFree: 5854212 kB
Dirty: 36 kB
Writeback: 0 kB
AnonPages: 148052 kB
Mapped: 12756 kB
Slab: 1582512 kB
SReclaimable: 134348 kB
SUnreclaim: 1448164 kB
PageTables: 4500 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 6884900 kB
Committed_AS: 689956 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 265628 kB
VmallocChunk: 34359472059 kB
# /usr/sr...

To: <linux@...>
Cc: <linux-kernel@...>, <netdev@...>
Date: Friday, September 28, 2007 - 5:20 am

CONFIG_DEBUG_SLAB_LEAK?
-

To: <akpm@...>, <linux@...>
Cc: <linux-kernel@...>, <netdev@...>
Date: Sunday, September 30, 2007 - 3:59 am

> ntpd. Sounds like pps leaking to me.

That's what I'd think, except that pps does no allocation in the normal
running state, so there's nothing to leak. The interrupt path just
records the time in some preallocated, static buffers and wakes up
blocked readers. The read path copies the latest data out of those
static buffers. There's allocation when the PPS device is created,

Ah, thanks you; I've been using SLUB which doesn't support this option.
Here's what I've extracted. I've only presented the top few
slab_allocators and a small subset of the oom-killer messages, but I
have full copies if desired. Unfortunately, I've discovered that the
machine doesn't live in this unhappy state forever. Indeed, I'm not
sure if killing ntpd "fixes" anything; my previous observations
may have been optimistic ignorance.

(For my own personal reference looking for more oom-kill, I nuked ntpd
at 06:46:56. And the oom-kills are continuing, with the latest at
07:43:52.)

Anyway, I have a bunch of information from the slab_allocators file, but
I'm not quire sure how to make sense of it.

With a machine in the unhappy state and firing the OOM killer, the top
20 slab_allocators are:
$ sort -rnk2 /proc/slab_allocators | head -20
skbuff_head_cache: 1712746 __alloc_skb+0x31/0x121
size-512: 1706572 tcp_send_ack+0x23/0x102
skbuff_fclone_cache: 149113 __alloc_skb+0x31/0x121
size-2048: 148500 tcp_sendmsg+0x1b5/0xae1
sysfs_dir_cache: 5289 sysfs_new_dirent+0x4b/0xec
size-512: 2613 sock_alloc_send_skb+0x93/0x1dd
Acpi-Operand: 2014 acpi_ut_allocate_object_desc_dbg+0x34/0x6e
size-32: 1995 sysfs_new_dirent+0x29/0xec
vm_area_struct: 1679 mmap_region+0x18f/0x421
size-512: 1618 tcp_xmit_probe_skb+0x1f/0xcd
size-512: 1571 arp_create+0x4e/0x1cd
vm_area_struct: 1544 copy_process+0x9f1/0x1108
anon_vma: 1448 anon_vma_prepare+0x29/0x74
filp: 1201 get_empty_filp+0x44/0xcd
UDP: 1173 sk_alloc+0x25/0xaf
size-128: 1048 r1bio_pool_alloc+0x23/0x3b
size-128: 1024 nfsd_cache_init+0x2d/0xcf
Acpi-Namespace: 97...

To: <linux@...>
Cc: <linux-kernel@...>, <netdev@...>, Francois Romieu <romieu@...>
Date: Sunday, September 30, 2007 - 5:23 am

Absolutely. Normally a driver's transmit completion interrupt handler will
run dev_kfree_skb_irq() against the skbs which have been fully sent.

However it'd be darned odd if the driver was leaking only tcp acks.

I can find no occurrence of "dev_kfree_skb" in drivers/net/ipg.c, which is
suspicious.

I assume that meminfo was not captured when the system was ooming? There
isn't much slab there.

-

To: <akpm@...>, <linux@...>
Cc: <jesse@...>, <linux-kernel@...>, <netdev@...>, <romieu@...>, <s.l-h@...>
Date: Sunday, September 30, 2007 - 7:40 am

> OK. Did you try to reproduce it without the pps patch applied?

No. But I've yanked the ip1000a driver (using old crufy vendor-supplied

It's leaking lots of things... you can see ARP packets in there and
all sorts of stuff. But the big traffic hog is BackupPC doing inbound
rsyncs all night long, which generates a lot of acks. Those are the

Look for "IPG_DEV_KFREE_SKB", which is a wrapper macro. (Or just add
"-i" to your grep.) It should probably be deleted (it just expands to

As I wrote originally, I got it from
http://marc.info/?l=linux-netdev&m=118980588419882
which was a reuqest for mainline submission.

If there are other patches floating around, I'm happy to try them.

Oops, sorry. I captured slabinfo but not meminfo.

Thank you very much! Sorry to jump the gun and post a lot before I had
all the data, but if it WAS a problem in -rc8, I wanted to mention it
before -final.

Now, the rush is to get the ip1000a driver fixed before the merge
window opens. I've added all the ip1000a developers to the Cc: list in
an attempt to speed that up.
-

Previous thread: [PATCH] i915: make vbl interrupts work properly on i965g/gm by Dave Airlie on Thursday, September 27, 2007 - 10:05 pm. (6 messages)

Next thread: Re: [patch 2/2] VFS: allow filesystem to override mknod capability checks by Neil Brown on Thursday, September 27, 2007 - 10:40 pm. (2 messages)