> You are saying that, 4 Intel 82571EB Gb NICs (2 pci cards x 2
> NICs/chip) using the e1000 driver.
> So, do you have 4 interrupt lines or only 2-lines?
>
> Based on this, if you have 4 interrupt lines, then you can assign each
> interrupt-line to each CPU-core, with NAPI enabled(for good
> performance).
>
> Regards,
> Madhukar.
> -----Original Message-----
> From:
linux-net-owner@vger.kernel.org
> [mailto:linux-net-owner@vger.kernel.org] On Behalf Of Don Porter
> Sent: Wednesday, October 15, 2008 12:36 AM
> To:
linux-net@vger.kernel.org
> Subject: e1000 softirq load balancing
>
> Hi,
>
> Background:
>
> I have a 16 core x86_64 machine (4 chips x 4 cores/chip) that has 4
> Intel 82571EB Gb NICs (2 pci cards x 2 NICs/chip) using the e1000
> driver.
>
> I have a simple client/server micro-benchmark that pounds a server on
> each NIC with requests to measure peak throughput. I am running Ubuntu
> 8.04.1, kernel version 2.6.24.
>
> Problem:
>
> What I am observing is that a single ksoftirqd thread is becoming a
> bottleneck for the system.
> More specifically, one cpu runs ksoftirqd at 100% cpu utilization, while
> 4 cpus each run their servers at about 25%. I carefully used
> sched_setaffinity() to map server threads to cpus and
> /proc/irq/<device>/smp_affinity to map hardware interrupts to cpus such
> that there should be exactly 1 cpu per server thread and 1 cpu for
> servicing hardware interrupts per device.
>
> I can observe (via /proc/interrupts) that the interrupts are being
> distributed properly, but despite this I only see 1 or 2 ksoftirqd
> running, and the server daemons bottlenecked behind them. (This is with
> NAPI disabled. With NAPI enabled, I can't get even 2 ksoftirqd threads
> to run). I have tried varous permutations such as assigning each
> hardware interrupt to a different physical chip.
>
> Desired Result:
>
> It seems to me that with 4 independent NICs and plenty of CPUs to spare,
> I ought to be able to assign one softirq daemon to each NIC rather than
> funnelling all of the traffic through 1 or 2.
>
> Any advice on this issue is greatly appreciated.
>
> Best regards,
> Don Porter
> --
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to
majordomo@vger.kernel.org More majordomo info
> at
http://vger.kernel.org/majordomo-info.html
>
