Luigi Rizzo wrote:
quoted text
> On Mon, Oct 05, 2009 at 10:55:21AM +0800, Eugene Grosbein wrote:
>> On Sun, Oct 04, 2009 at 06:47:23PM +0500, rihad wrote:
>>
>>> sysctls:
>>> kern.ipc.nmbclusters=50000
>>> net.inet.ip.dummynet.io_fast=1
>> I guess you should also try to increase pipes length:
>>
>> net.inet.ip.dummynet.hash_size=65536
>> net.inet.ip.dummynet.pipe_slot_limit=1000
> 
> in fact, i forgot to ask, we'd need to know the output of
> "ipfw pipe show" just to get the idea if there is any
> known reason for the drop (e.g. queues too short, etc.)
> 
The pipes are fine, each normally having 100-120 concurrent consumers 
(i.e. active users). (The max number of consumers allowed per pipe is 
hash_size*max_chain_len, normally 64*16==1024.)

kern.ipc.nmbclusters=111111

And these are by default:
#net.inet.ip.dummynet.hash_size=64
#net.inet.ip.dummynet.max_chain_len=16



Yesterday I set up a cronjob logging the number of drops in the past 15 
minutes:
*/15 * * * *    (echo -n "$(date) "; netstat -z -s 2>/dev/null | fgrep 
-w "output packets dropped") >> /tmp/bufs.log

And here's bufs.log:
Sun Oct  4 21:45:00 AZST 2009 	418869 output packets dropped due to no 
bufs, etc.
Sun Oct  4 22:00:00 AZST 2009 	851693 output packets dropped due to no 
bufs, etc.
Sun Oct  4 22:15:01 AZST 2009 	932885 output packets dropped due to no 
bufs, etc.
Sun Oct  4 22:30:00 AZST 2009 	890522 output packets dropped due to no 
bufs, etc.
Sun Oct  4 22:45:00 AZST 2009 	1065931 output packets dropped due to no 
bufs, etc.
Sun Oct  4 23:00:00 AZST 2009 	937863 output packets dropped due to no 
bufs, etc.
Sun Oct  4 23:15:01 AZST 2009 	1018822 output packets dropped due to no 
bufs, etc.
Sun Oct  4 23:30:00 AZST 2009 	981922 output packets dropped due to no 
bufs, etc.
Sun Oct  4 23:45:00 AZST 2009 	1015124 output packets dropped due to no 
bufs, etc.
Mon Oct  5 00:00:01 AZST 2009 	1123926 output packets dropped due to no 
bufs, etc.
Mon Oct  5 00:15:01 AZST 2009 	948161 output packets dropped due to no 
bufs, etc.
Mon Oct  5 00:30:00 AZST 2009 	937277 output packets dropped due to no 
bufs, etc.
Mon Oct  5 00:45:00 AZST 2009 	875218 output packets dropped due to no 
bufs, etc.
Mon Oct  5 01:00:00 AZST 2009 	803527 output packets dropped due to no 
bufs, etc.
Mon Oct  5 01:15:00 AZST 2009 	728639 output packets dropped due to no 
bufs, etc.
Mon Oct  5 01:30:00 AZST 2009 	626154 output packets dropped due to no 
bufs, etc.
Mon Oct  5 01:45:00 AZST 2009 	519441 output packets dropped due to no 
bufs, etc.
Mon Oct  5 02:00:00 AZST 2009 	371098 output packets dropped due to no 
bufs, etc.
Mon Oct  5 02:15:00 AZST 2009 	681243 output packets dropped due to no 
bufs, etc.
Mon Oct  5 02:30:00 AZST 2009 	562909 output packets dropped due to no 
bufs, etc.
Mon Oct  5 02:45:00 AZST 2009 	426734 output packets dropped due to no 
bufs, etc.
Mon Oct  5 03:00:00 AZST 2009 	344619 output packets dropped due to no 
bufs, etc.
Mon Oct  5 03:15:00 AZST 2009 	90006 output packets dropped due to no 
bufs, etc.
Mon Oct  5 03:30:00 AZST 2009 	17064 output packets dropped due to no 
bufs, etc.
Mon Oct  5 03:45:00 AZST 2009 	3851 output packets dropped due to no 
bufs, etc.
Mon Oct  5 04:00:00 AZST 2009 	1323 output packets dropped due to no 
bufs, etc.
Mon Oct  5 04:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 04:30:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 04:45:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:30:01 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 05:45:01 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:15:01 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:30:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 06:45:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:30:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 07:45:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:30:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 08:45:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:30:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 09:45:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:00:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:15:00 AZST 2009 	0 output packets dropped due to no bufs, etc.
Mon Oct  5 10:30:00 AZST 2009 	177 output packets dropped due to no 
bufs, etc.
Mon Oct  5 10:45:00 AZST 2009 	1701 output packets dropped due to no 
bufs, etc.
Mon Oct  5 11:00:01 AZST 2009 	19933 output packets dropped due to no 
bufs, etc.
Mon Oct  5 11:15:00 AZST 2009 	30003 output packets dropped due to no 
bufs, etc.
Mon Oct  5 11:30:00 AZST 2009 	56712 output packets dropped due to no 
bufs, etc.
Mon Oct  5 11:45:00 AZST 2009 	78721 output packets dropped due to no 
bufs, etc.
Mon Oct  5 12:00:01 AZST 2009 	112518 output packets dropped due to no 
bufs, etc.
Mon Oct  5 12:15:00 AZST 2009 	7229 output packets dropped due to no 
bufs, etc.
Mon Oct  5 12:30:01 AZST 2009 	24965 output packets dropped due to no 
bufs, etc.
Mon Oct  5 12:45:00 AZST 2009 	75900 output packets dropped due to no 
bufs, etc.
Mon Oct  5 13:00:00 AZST 2009 	45002 output packets dropped due to no 
bufs, etc.
Mon Oct  5 13:15:00 AZST 2009 	67161 output packets dropped due to no 
bufs, etc.
Mon Oct  5 13:30:00 AZST 2009 	112591 output packets dropped due to no 
bufs, etc.


As you can see the drops gradually went away completely at about 4:00 
a.m., and started coming up at about 10:30 a.m., although at a lower 
rate, probably thanks to me bumping "ipfw ... queue NNN" up to 5000 at 
10a.m. this morning. The traffic flow between 4a.m. and 10:30a.m., the 
"quiet" times, is about 200-330 mbit/s 5 minute average, without a 
single drop. But after that, in come the drops, no matter how high I set 
the queue. Should I try 10000 slots? 20000? I'm really sure there are 
plenty of heavy downloaders between 4a.m. and 10a.m., but still without 
a single drop before approx. 330 mbit/s! Strange, isn't it? This makes 
me believe I'm hitting some other global memory limit, rather than 
per-user limit, at around 340-350 mbit/s, causing drops. top and netstat 
-m are OK right now, though:

Mem: 1870M Active, 1220M Inact, 481M Wired, 201M Cache, 214M Buf, 164M Free
15595/5385/20980/111112 mbuf clusters in use (current/cache/total/max)
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"