On Mon, 23 Nov 2009, Eric Dumazet wrote:

quoted text
> robert@herjulf.net a écrit :
> > Eric Dumazet writes:
> > 
> >  > Jesper Dangaard Brouer a écrit :
> >  > 
> >  > > How is your smp_affinity mask's set?
> >  > > 
> >  > > grep . /proc/irq/*/fiber1-*/../smp_affinity
> >  > 
> > 
> >  Weird... set clone_skb to 1 to be sure and vary dst or something so 
> >  the HW classifier selects different queues and with proper RX affinty. 
> >  
> >  You should see in /proc/net/softnet_stat something like:
> > 
> > 012a7bb9 00000000 000000ae 00000000 00000000 00000000 00000000 00000000 00000000
> > 01288d4c 00000000 00000049 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128fe28 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> > 01295387 00000000 00000047 00000000 00000000 00000000 00000000 00000000 00000000
> > 0129a722 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128c5e4 00000000 00000046 00000000 00000000 00000000 00000000 00000000 00000000
> > 0128f718 00000000 00000043 00000000 00000000 00000000 00000000 00000000 00000000
> > 012993e3 00000000 0000004a 00000000 00000000 00000000 00000000 00000000 00000000
> > 
> 
> slone_skb set to 1, this changes nothing but slows down pktgen (obviously)
> 
> Result: OK: 117614452(c117608705+d5746) nsec, 100000000 (60byte,0frags)
>   850235pps 408Mb/sec (408112800bps) errors: 0
> 
> All RX processing of 16 RX queues done by CPU 1 only.

Ok, I was confused earlier.  I thought you were saying that all packets 
were headed into a single Rx queue.  This is different.

Do you know what version of irqbalance you're running, or if it's running 
at all?  We've seen issues with irqbalance where it won't recognize the 
ethernet device if the driver has been reloaded.  In that case, it won't 
balance the interrupts at all.  If the default affinity was set to one 
CPU, then well, you're screwed.

My suggestion in this case is after you reload ixgbe and start your tests, 
see if it all goes to one CPU.  If it does, then restart irqbalance 
(service irqbalance restart - or just kill it and restart by hand).  Then 
start running your test, and in 10 seconds you should see the interrupts 
move and spread out.

Let me know if this helps,
-PJ