login
Search
Hi. I have something that looks like a regression in bonding between 2.6.26.8 and 2.6.27.6 (I'll try the mid-steps later). Setup: LACP bond(mode=4,mmimon=100) with 3 NIC's and dhcp on top (static ip didn't work either). Problem: The bond doesn't get up after bootup. Subsequence ifdown/ifup brings it up. I suspect it it timing related. The interface being configured before it's ready: root@quad01:~# dmesg | egrep '(dhc|bond)' [ 12.421963] bonding: MII link monitoring set to 100 ms [ 12.483370] bonding: bond0: enslaving eth0 as a backup interface with an up link. [ 12.523372] bonding: bond0: enslaving eth1 as a backup interface with an up link. [ 12.611731] bonding: bond0: enslaving eth2 as a backup interface with a down link. [ 12.780816] warning: `dhclient3' uses 32-bit capabilities (legacy support in use) [ 15.720491] bonding: bond0: link status definitely up for interface eth2. [ 87.800324] bond0: no IPv6 routers present The setup is a 3 NIC bond on a Sun X2200 dual-cpu Quad-core server. I have similar bond on a X4600 where they works with 2.6.27.6 so I suspect that the difference is that the X4600 has all NIC's from the same vendor where as the X2200 has 2 Broadcom NIC's and 2 NVidia nics. root@quad01:~# lspci | grep -i ethernet 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3) 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3) 06:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3) 06:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3) -- Jesper --
What exactly does "doesn't get up" mean? If you configure with a static IP, and it doesn't come up, what's in /proc/net/bonding/bond0? This looks like one of the slaves (eth2) took longer to assert carrier up (slower autoneg, perhaps) than the other two (eth0 and eth1). That wouldn't necessarily cause DHCP to fail; 802.3ad is allowed to aggregate eth0 and eth1 and use them independently of eth2. However, if eth0 and eth1 are incorrectly asserting carrier up (before autoneg is complete), then that could cause problems. If that's the case, then checking /proc/net/bonding/bond0 should show the actual aggregation status. If lacp is set to slow (the default), then it should try to reaggregate 30 seconds later, and that would clear up the aggregation. DHCP would still need to restart, though. What distro are you using? I just tried the bonding driver from the current net-next-2.6 mainline on recent SuSE and 802.3ad + DHCP Which flavor (Broadcom or Nvidia) are the 3 devices that are the same? -J --- -Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com --
There was something about that rc-27 could ruin my Intel NICs.. right?
Looks like this:
# ifconfig bond0
bond0 Link encap:Ethernet HWaddr 00:1e:68:57:82:b2
inet6 addr: fe80::21e:68ff:fe57:82b2/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:74 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5952 (5.8 KB) TX bytes:1900 (1.8 KB)
(usually this would have been assigned an ip-address using dhcp, does
that with 2.6.26.8, with the same configuration). Manually running
dhclient on the interface doesn't bring it up either.
# dhclient bond0
Internet Systems Consortium DHCP Client V3.0.6
Copyright 2004-2007 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
Listening on LPF/bond0/00:1e:68:57:82:b2
Sending on LPF/bond0/00:1e:68:57:82:b2
Sending on Socket/fallback
DHCPDISCOVER on bond0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on bond0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on bond0 to 255.255.255.255 port 67 interval 9
DHCPDISCOVER on bond0 to 255.255.255.255 port 67 interval 2
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
Booting up with static ip configuration it looks like this:
# ifconfig
bond0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:10.194.132.90 Bcast:10.194.133.255
Mask:255.255.254.0
UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Configured with a static ip. ifconfig claims that the interface is up
and configured with the ip-address.
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding ...I'll try that anyway.. I dont have any Intel NIC's in this server. (sorry about the confusion). -- Jesper --
For this case, when there's a non-zero HWaddr assigned, what is in /proc/net/bonding/bond0 and what bonding related messages are in dmesg or /var/log/messages? This is different, and not correct: there's no HWaddr. That probably means there are no slaves. This is most likely a totally separate problem; the first information suggests that the bond has slaves, but isn't working; this suggests that the bond has no slaves As with the prior information, the above indicates that there are no slaves attached to the bond. There's no HWaddr, and the /proc/net/bonding/bond0 lists no slaves. For this case (bonding HWaddr all zeros), what does dmesg or /var/log/messages list for bonding? This is most likely some kind of configuration problem causing no interfaces to be enslaved. -J --- -Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com --
Hi Jay. You're right. I misconfigured the static setup while moving it for testing and I moved the misconfiguration back to the dynamic setup. I'll make another response to the same message trying to fill in the questions while having it configured correctly. Jesper --
you can test 2.6.27-rcX without worry for e1000e device corruption IF you set CONFIG_FTRACE=n ftrace was the code that ran over the registers mapped by e1000e. just FYI, I know this thread is ancient. --
This time answered with a configuration, that I have tested that works
on 2.6.26.8. The setup is designed to run under dhcp. (small HPC-cluster).
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 3008
Partner Mac Address: 02:04:96:34:88:6a
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b2
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b3
Aggregator ID: 1
Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:68:57:82:b0
Aggregator ID: 2
# ifconfig bond0
bond0 Link encap:Ethernet HWaddr 00:1e:68:57:82:b2
inet addr:10.194.132.90 Bcast:10.194.133.255 Mask:255.255.254.0
inet6 addr: fe80::21e:68ff:fe57:82b2/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:5241 errors:0 dropped:0 overruns:0 frame:0
TX packets:1314 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:382392 (373.4 KB) TX bytes:126272 (123.3 KB)
doing ifdown bond0 && ifup bond0 brings it correctly up.
root@quad11:~# ping -c 1 -w 5 -W 5 sal
ping: unknown host sal
root@quad11:~# ifdown bond0 && ifup bond0
root@quad11:~# ping -c 1 -w 5 -W 5 sal
PING sal (10.194.133.13) 56(84) bytes of data.
64 bytes from sal (10.194.133.13): icmp_seq=1 ttl=64 time=0.106 ms
--- sal ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.106/0.106/0.106/0.000 ...Ok, update, Rome wasn't build in one day. A collegue of mine (on CC) found time to do a bisect of this on. The offending commit seems to be: bonding: refactor mii monitor Refactor mii monitor. As with the previous ARP monitor refactor, the motivation for this is to handle locking rationally (in this case, removing conditional locking) and generally clean up the code. This patch breaks up the monolithic mii monitor into two phases: an inspection phase, followed by an optional commit phase. The commit phase is the only portion that requires RTNL or makes changes to state, and is only called when inspection finds something to change. Signed-off-by: Jay Vosburgh <fubar@us.ibm.com> Signed-off-by: Jeff Garzik <jgarzik@redhat.com> A test with a fresh 2.6.29-rc6 revealed that the problem has been fixed subsequently.. but still exists in 2.6.27-newest. (havent tested 2.6.28-newest yet). Any ideas of what the "fixing" commit is .. or should that also be bisected? Dmesg of 2-6-29-rc6 working system here: http://krogh.cc/~jesper/dmesg-2.6.29-rc6.txt .. shows that when bonding is initialized it doesnt have link up on any of the interfaces. git bisect log # bad: [adee14b2e1557d0a8559f29681732d05a89dfc35] Linux 2.6.27-rc6 # good: [63e0e67b17dc233f93f709610971bbfadc97f75e] Linux 2.6.26.8 git-bisect start 'v2.6.27-rc6' 'v2.6.26.8' # good: [d536b1f86591fb081c7a56eab04e711eb4dab951] x86: fix crash due to missing debugctlmsr on AMD K6-3 git-bisect good d536b1f86591fb081c7a56eab04e711eb4dab951 # good: [d536b1f86591fb081c7a56eab04e711eb4dab951] x86: fix crash due to missing debugctlmsr on AMD K6-3 git-bisect good d536b1f86591fb081c7a56eab04e711eb4dab951 # good: [8d0b1c51eb8375f88c0886d2e9f71881e19d42a7] gbefb: cmap FIFO timeout git-bisect good 8d0b1c51eb8375f88c0886d2e9f71881e19d42a7 # bad: [ab1666c1364a209e6141d7c14e47a42b5f00eca2] USB: quirk PLL power down mode git-bisect bad ab1666c1364a209e6141d7c14e47a42b5f00eca2 # good: ...
Jesper Krogh <jesper@krogh.cc> wrote:
I went back and looked at your earlier mail. Since you're using
802.3ad mode, my first guess would be this commit:
commit fd989c83325cb34795bc4d4aa6b13c06f90eac99
Author: Jay Vosburgh <fubar@us.ibm.com>
Date: Tue Nov 4 17:51:16 2008 -0800
bonding: alternate agg selection policies for 802.3ad
This patch implements alternative aggregator selection policies
for 802.3ad. The existing policy, now termed "stable," selects the active
aggregator by greatest bandwidth, and only reselects a new aggregator
if the active aggregator is entirely disabled (no more ports or all ports
down).
This patch adds two new policies: bandwidth and count, selecting
the active aggregator by total bandwidth (like the stable policy) or by
the number of ports in the aggregator, respectively. These two policies
also differ from the stable policy in that they will reselect the active
aggregator when availability-related changes occur in the bond (e.g.,
link state change).
This permits "gang failover" within 802.3ad, allowing redundant
aggregators along parallel paths to always maintain the "best" aggregator
as the active aggregator (rather than having to wait for the active to
entirely fail).
This patch also updates the driver version to 3.5.0.
Signed-off-by: Jay Vosburgh <fubar@us.ibm.com>
Signed-off-by: Jeff Garzik <jgarzik@redhat.com>
This changed or refactored a great deal of the aggregator
selection logic, and it might be that it also fixed your problem by mere
happenstance.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
--
That didn't do it.. I applied it to 2.6.27.19 but it didnt make that work. dmesg | grep bond (2.6.27.19 + above patch). [ 13.643301] bonding: MII link monitoring set to 100 ms [ 13.730455] bonding: bond0: enslaving eth0 as a backup interface with an up link. [ 13.781934] bonding: bond0: enslaving eth1 as a backup interface with an up link. [ 13.904665] bonding: bond0: enslaving eth2 as a backup interface with a down link. [ 16.945264] bonding: bond0: link status definitely up for interface eth2. [ 75.040290] bond0: no IPv6 routers present dmesg | grep bond (2.6.29-rc6) $ ssh quad02 dmesg | grep bond [ 27.437877] bonding: MII link monitoring set to 100 ms [ 27.445246] ADDRCONF(NETDEV_UP): bond0: link is not ready [ 27.493260] bonding: bond0: enslaving eth0 as a backup interface with a down link. [ 27.521397] bonding: bond0: enslaving eth1 as a backup interface with a down link. [ 27.542332] bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond [ 27.611509] bonding: bond0: enslaving eth2 as a backup interface with a down link. [ 27.617017] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 27.642330] bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond [ 30.042501] bonding: bond0: link status definitely up for interface eth1. [ 30.142505] bonding: bond0: link status definitely up for interface eth0. [ 30.742547] bonding: bond0: link status definitely up for interface eth2. [ 37.875044] bond0: no IPv6 routers present I just tested 2.6.28.7.. it still broken. So the fix probably has to be somewhere in the post 2.6.28 sets. -- Jesper --
That was the only real functional change to 802.3ad, there are a It looks like the above two tests are on different machines, or were at least done with different network cards. Is that the case? I'm just wondering if what you're seeing is somehow tied to the network devices' respective autonegotiation speeds, or some difference in the device drivers. The first dmesg looks to have one slow (3 sec) and two fast ones; the second dmesg looks to have all slow devices. Have you tried the kernels the other way around (the first kernel on the second machine, and vice versa)? I'll compile 2.6.28.7 here and see if it works for me. -J --- -Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com --
There is 12 Sun Fire X2200 in the rack, they are fully identical (some with a small difference in memory configuration as the only difference. So yes, different machines, but same hardware (bought in the same Yes, I've randomly picked a machine in the set to do the test, they all Jesper -- Jesper --
I appreciate that you spend time on it, but my feeling is that it definately isn't reproducible in all environments (otherwise we would probably have seen a large cry by now). I'm trying to bisect the "fix" down and hope that'll tell us something more. If you do the test, remember, that it is not like "bonding isn't working". It just fails to initialize correctly at bootup and doesnt get the link state by itself. Subsequently doing a /etc/init.d/networking restart brigs it correct up. -- Jesper --
Ok, The fixing commit seems to be: cb52deba12f27af90a46d2f8667a64888118a888
Applying it to 2.6.28.7 and 2.6.27.19 makes them both work.
It also explains why my e1000 based bonds didnt break, allthough the
commitmessage doesnt mention anything about how it should effect
bonding. Wouldn't it make sense to propose this patch for 2.6.27 and
2.6.28 stable kernels?
commit cb52deba12f27af90a46d2f8667a64888118a888
Author: Ed Swierk <eswierk@arastra.com>
Date: Mon Dec 1 12:24:43 2008 +0000
forcedeth: power down phy when interface is down
Bring the physical link down when the interface is down by placing
the PHY
in power-down state, unless WOL is enabled. This mirrors the
behavior of
other drivers including e1000 and tg3.
Without the patch, ifconfig down leaves the physical link up, which
confuses
datacenter users who expect the link lights both on the NIC and the
switch to
go out when they bring an interface down.
Furthermore, even though the phy is powered on, autonegotiation
stops working,
so a normally gigabit link might suddenly become 100 Mbit
half-duplex when the
interface goes down, and become gigabit when it comes up again.
Ayaz said:
I would not include this patch until further testing is
performed. NVIDIA
MCP chips use 3rd party PHY vendors. By powering down the phy,
it could
have adverse affects on certain phys.
Arthur Jones said:
I just ran across this patch. Tested on a Marvell 88E1121R (GigE
PHY)
and works great. This is a very important feature for me.
Signed-off-by: Ed Swierk <eswierk@arastra.com>
Tested-by: Arthur Jones <ajones@riverbed.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/drivers/net/forcedeth.c b/drivers/net/forcedeth.c
index 0d7e575..12384df 100644
--- a/drivers/net/forcedeth.c
+++ ...Perhaps.
I don't have a forcedeth to test with, and as you surmised, I
was unable to reproduce the problem with other chipsets (tg3 or e1000).
However, I did find another bug I introduced during the "mii
refactor" patch that you mentioned as being the original source of the
problem. That bug will cause 802.3ad to not notice speed changes.
Could you test the patch below on your 2.6.68.7 and/or 2.6.27.19
and see if it resolves your problem (without the forcedeth patch)?
-J
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 2c96b93..ad81474 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3545,11 +3545,27 @@ static int bond_slave_netdev_event(unsigned long event, struct net_device *slave
}
break;
case NETDEV_CHANGE:
- /*
- * TODO: is this what we get if somebody
- * sets up a hierarchical bond, then rmmod's
- * one of the slave bonding devices?
- */
+ if (bond->params.mode == BOND_MODE_8023AD ||
+ bond_is_lb(bond)) {
+ struct slave *slave;
+
+ slave = bond_get_slave_by_dev(bond, slave_dev);
+ if (slave) {
+ u16 old_speed = slave->speed;
+ u16 old_duplex = slave->duplex;
+
+ bond_update_speed_duplex(slave);
+
+ if (bond_is_lb(bond))
+ break;
+
+ if (old_speed != slave->speed)
+ bond_3ad_adapter_speed_changed(slave);
+ if (old_duplex != slave->duplex)
+ bond_3ad_adapter_duplex_changed(slave);
+ }
+ }
+
break;
case NETDEV_DOWN:
/*
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
--
There was something missing from the header to make it compile.. I found
that in a later version. Patch below fixed the problem (without the
forcedeth patch).
diff --git a/drivers/net/bonding/bond_main.c
b/drivers/net/bonding/bond_main.c
index 1b9c4dc..fd61dfb 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3516,11 +3516,27 @@ static int bond_slave_netdev_event(unsigned long
event, struct net_device *slave
}
break;
case NETDEV_CHANGE:
- /*
- * TODO: is this what we get if somebody
- * sets up a hierarchical bond, then rmmod's
- * one of the slave bonding devices?
- */
+ if (bond->params.mode == BOND_MODE_8023AD ||
+ bond_is_lb(bond)) {
+ struct slave *slave;
+
+ slave = bond_get_slave_by_dev(bond, slave_dev);
+ if (slave) {
+ u16 old_speed = slave->speed;
+ u16 old_duplex = slave->duplex;
+
+ bond_update_speed_duplex(slave);
+
+ if (bond_is_lb(bond))
+ break;
+
+ if (old_speed != slave->speed)
+
bond_3ad_adapter_speed_changed(slave);
+ if (old_duplex != slave->duplex)
+
bond_3ad_adapter_duplex_changed(slave);
+ }
+ }
+
break;
case NETDEV_DOWN:
/*
diff --git a/drivers/net/bonding/bonding.h b/drivers/net/bonding/bonding.h
index fb730ec..b1315e4 100644
--- a/drivers/net/bonding/bonding.h
+++ b/drivers/net/bonding/bonding.h
@@ -248,6 +248,14 @@ static inline struct bonding
*bond_get_bond_by_slave(struct slave *slave)
return (struct bonding *)slave->dev->master->priv;
}
+static inline bool ...From: Jesper Krogh <jesper@krogh.cc> Jay please resend this with proper signoffs etc. if you want me to apply it. --
I posted it again with the usual stuff a day or two after I
posted the test patch; I'll append it to the end of this email. Note
that the below patch has a minor cosmetic change from the test patch.
I believe this fix should go to -stable for 2.6.26 and 2.6.27,
but it'll need the change Jesper added to pick up a macro that was added
to mainline:
diff --git a/drivers/net/bonding/bonding.h b/drivers/net/bonding/bonding.h
index fb730ec..b1315e4 100644
--- a/drivers/net/bonding/bonding.h
+++ b/drivers/net/bonding/bonding.h
@@ -248,6 +248,14 @@ static inline struct bonding
*bond_get_bond_by_slave(struct slave *slave)
return (struct bonding *)slave->dev->master->priv;
}
+static inline bool bond_is_lb(const struct bonding *bond)
+{
+ return bond->params.mode == BOND_MODE_TLB
+ || bond->params.mode == BOND_MODE_ALB;
+}
+
+
+
#define BOND_FOM_NONE 0
#define BOND_FOM_ACTIVE 1
#define BOND_FOM_FOLLOW 2
The above fragment isn't needed for mainline, only for -stable.
-J
From: Jay Vosburgh <fubar@us.ibm.com>
To: netdev@vger.kernel.org
Cc: "David S. Miller" <davem@davemloft.net>, stable@kernel.org
Subject: [PATCH net-next-2.6] bonding: Fix updating of speed/duplex changes
Date: Fri, 06 Mar 2009 15:27:33 -0800
This patch corrects an omission from the following commit:
commit f0c76d61779b153dbfb955db3f144c62d02173c2
Author: Jay Vosburgh <fubar@us.ibm.com>
Date: Wed Jul 2 18:21:58 2008 -0700
bonding: refactor mii monitor
The un-refactored code checked the link speed and duplex of
every slave on every pass; the refactored code did not do so.
The 802.3ad and balance-alb/tlb modes utilize the speed and
duplex information, and require it to be kept up to date. This patch
adds a notifier check to perform the appropriate updating when the slave
device speed changes.
Signed-off-by: Jay Vosburgh <fubar@us.ibm.com>
---
drivers/net/bonding/bond_main.c | ...Did you sent it off to the stable kernel maintainers? -- Jesper --
From: Jesper Krogh <jesper@krogh.cc> The fix has to go into Linus's tree first. I haven't integrated Jay's changes yet. --
Excellent. I was just trying to make sure that it wasn't lost somewhere in the process. -- Jesper --
From: Jay Vosburgh <fubar@us.ibm.com> Applied and I'll queue it up for -stable too, thanks! --
