Home » Mailing list archives » linux-kernel » 2008 » March » 17

Re: dst cache overflow

Previous thread: Re: Fw: [PACKET]: Fix /proc/net/packet crash due to bogus private pointer by Andrew Morton on Sunday, December 16, 2007 - 12:56 am. (1 message)

Next thread: Re: [PATCH] x86_64: fix problems due to use of "outb" to port 80 on some AMD64x2 laptops, etc. by Ingo Molnar on Sunday, December 16, 2007 - 2:27 am. (1 message)
[view original message]
From: Tobias Diedrich
Subject: Re: dst cache overflow
Date: Sunday, December 16, 2007 - 1:38 am

ranma@oni:~$ cat /proc/net/sockstat
sockets: used 143
TCP: inuse 16 orphan 0 tw 4 alloc 21 mem 1
UDP: inuse 8
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

HTH,

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de
このメールは十割再利用されたビットで作られています。
--


[ message continues ]
[view original message]
From: Tobias Diedrich
Subject: Re: dst cache overflow
Date: Sunday, March 16, 2008 - 6:25 pm

Hi all,

just a reminder, the server is now running 2.6.24 and the leak is
still there.  I noticed there is a slab leak debug option and turned
it on, results should be visible in a few days I guess.

Offenders: Slabs TCPv6, size_2048 and size_512 grow over time.

I also noticed that the machine stops responding to pings after some
time, which seems to coincide with a slight increase in leak speed.

Interstingly IPv4 pings give out first, IPv6 pings continue to work a bit
longer, but eventually fail too. (Because sixxs starts sending 'your
tunnel is down!' mails I actually wrote a userspace pingd as a
workaround...)

/proc/net/snmp seems to indicate the kernel thinks it is still
sending ping replies.

HTH,

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de
--


[ message continues ]
[view original message]
From: Tobias Diedrich
Subject: Re: dst cache overflow
Date: Tuesday, March 18, 2008 - 11:59 am

Current (20080318 19:42) biggest net-related numbers:

skbuff_head_cache: 1614 __alloc_skb+0x29/0x108
ip_dst_cache: 1579 dst_alloc+0x25/0x5f
skbuff_fclone_cache: 965 __alloc_skb+0x29/0x108
size-512: 895 sock_wmalloc+0x2b/0x4d
TCPv6: 856 sk_clone+0x14/0x16a
size-2048: 733 tcp_sendmsg+0x174/0x9fc
size-512: 279 sock_alloc_send_skb+0x7c/0x193
size-512: 238 tcp_send_ack+0x2a/0xec
size-512: 223 tcp_send_fin+0x60/0x12e
sock_inode_cache: 172 sock_alloc_inode+0x10/0x4e
size-1024: 147 sock_alloc_send_skb+0x7c/0x193


ranma@oni:~$ netstat -6 -an | wc -l
29

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de
このメールは十割再利用されたビットで作られています。
--


[ message continues ]
[view original message]
From: Tobias Diedrich
Subject: IPv6 slab leak (was: Re: dst cache overflow)
Date: Sunday, March 23, 2008 - 2:21 am

20080323 10:03
ip_dst_cache: 5556 dst_alloc+0x25/0x5f
skbuff_fclone_cache: 4668 __alloc_skb+0x29/0x108
TCPv6: 4113 sk_clone+0x14/0x16a
skbuff_head_cache: 3919 __alloc_skb+0x29/0x108
size-2048: 3662 tcp_sendmsg+0x174/0x9fc
size-512: 1126 sock_alloc_send_skb+0x7c/0x193
size-512: 990 tcp_send_ack+0x2a/0xec
size-512: 973 tcp_send_fin+0x60/0x12e
size-1024: 349 sock_alloc_send_skb+0x7c/0x193

http://tdiedrich.de/~ranma/slab_allocators.20080318
http://tdiedrich.de/~ranma/slab_allocators.20080319
http://tdiedrich.de/~ranma/slab_allocators.20080321
http://tdiedrich.de/~ranma/slab_allocators.20080323

http://tdiedrich.de/~ranma/slab-month-oni-20080323.png
http://tdiedrich.de/~ranma/route-month-oni-20080323.png

oni:~# ip route ls
217.172.187.0/24 dev eth0  proto kernel  scope link  src
217.172.187.75 
default via 217.172.187.1 dev eth0 
oni:~# ip -6 route ls
2001:6f8:900:aee::/64 via :: dev sixxs  metric 256  expires -152518sec mtu 1480 advmss 1420 hoplimit 4294967295
2001:6f8:1002::/64 via :: dev sixxs  metric 256  expires -152518sec mtu 1480 advmss 1420 hoplimit 4294967295
fe80::/64 dev eth0  metric 256  expires -552013sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 via :: dev sixxs  metric 256  expires -152517sec mtu 1480 advmss 1420 hoplimit 4294967295
ff00::/8 dev eth0  metric 256  expires -552013sec mtu 1500 advmss 1440 hoplimit 4294967295
ff00::/8 dev sixxs  metric 256  expires -152517sec mtu 1480 advmss 1420 hoplimit 4294967295
default via 2001:6f8:900:aee::1 dev sixxs  metric 1  expires -152517sec mtu 1480 advmss 1420 hoplimit 4294967295
oni:~# uptime
 10:09:07 up 6 days,  9:20,  1 user,  load average: 0.14, 0.10, 0.08
oni:~# ip route ls cache | fgrep cache | wc -l
81
oni:~# cat /proc/net/stat/rt_cache 
entries  in_hit in_slow_tot in_slow_mc in_no_route in_brd
in_martian_dst in_martian_src  out_hit out_slow_tot out_slow_mc
gc_total gc_ignored gc_goal_miss gc_dst_overflow in_hlist_search
out_hlist_search
000015b5  0049feed 0035b5a2 00000000 00000000 00000006 ...
[ message continues ]
[view original message]
From: Tobias Diedrich
Subject: Re: IPv6 slab leak (was: Re: dst cache overflow)
Date: Friday, May 16, 2008 - 12:46 pm

I'm upgrading to 2.6.25.4 now and I hope that "sit: Add missing
kfree_skb() on pskb_may_pull() failure." finally fixes the bug. :)
There also was some IPv6 related leak-fix in 2.6.24.something IIRC,
but that one did not help unfortunately.

Cheers,

--=20
Tobias						PGP: http://9ac7e0bc.uguu.de
=E3=81=93=E3=81=AE=E3=83=A1=E3=83=BC=E3=83=AB=E3=81=AF=E5=8D=81=E5=89=B2=E5=
=86=8D=E5=88=A9=E7=94=A8=E3=81=95=E3=82=8C=E3=81=9F=E3=83=93=E3=83=83=E3=83=
=88=E3=81=A7=E4=BD=9C=E3=82=89=E3=82=8C=E3=81=A6=E3=81=84=E3=81=BE=E3=81=99=
=E3=80=82

[ message continues ]
[view original message]
From: Tobias Diedrich
Subject: Re: IPv6 slab leak (was: Re: dst cache overflow)
Date: Sunday, May 18, 2008 - 9:49 am

Ok, it's only been running for ~2 days now, but it very much looks
fixed now:
http://www.tdiedrich.de/~ranma/oni_20080518_route-year.png
http://www.tdiedrich.de/~ranma/oni_20080518_route-week.png
http://www.tdiedrich.de/~ranma/oni_20080518_route-day.png

Cheers,

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de
このメールは十割再利用されたビットで作られています。
--


[ message continues ]
Previous thread: Re: Fw: [PACKET]: Fix /proc/net/packet crash due to bogus private pointer by Andrew Morton on Sunday, December 16, 2007 - 12:56 am. (1 message)

Next thread: Re: [PATCH] x86_64: fix problems due to use of "outb" to port 80 on some AMD64x2 laptops, etc. by Ingo Molnar on Sunday, December 16, 2007 - 2:27 am. (1 message)