login
Search
Hello again I dug a little more and found out some new info. Unsolved bugreport with same symptoms: http://marc.info/?l=linux-kernel&m=119613299024398&w=2 Problems appear if I start br0 interface, as context switch rate increases 200 times. If I start eth0 instead everything looks ok. The bugreport above had bridging enabled too. When using eth0 I get: vmstat -n 1 10 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 17720 162288 25024 576852 0 0 21 22 6 9 10 4 85 0 0 0 17720 162196 25024 576880 0 0 0 0 13 473 0 0 99 0 0 0 17720 162196 25028 576880 0 0 0 4 36 1122 1 0 99 0 0 0 17720 162196 25028 576880 0 0 0 0 83 844 0 0 99 0 0 0 17720 162196 25028 576880 0 0 0 0 55 691 1 0 99 0 0 0 17720 162556 25028 576880 0 0 0 0 13 490 1 0 100 0 0 0 17720 162100 25028 576880 0 0 0 0 39 561 6 0 94 0 1 0 17720 162028 25028 576880 0 0 0 0 16 1030 4 0 96 0 0 0 17720 162028 25028 576880 0 0 0 0 40 597 1 0 99 0 0 0 17720 162028 25028 576880 0 0 0 0 12 512 2 0 97 0 top top - 03:30:07 up 6 days, 5:40, 4 users, load average: 0.02, 0.11, 0.28 Tasks: 149 total, 2 running, 147 sleeping, 0 stopped, 0 zombie Cpu(s): 12.1%us, 3.3%sy, 0.0%ni, 84.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1033388k total, 871336k used, 162052k free, 25016k buffers Swap: 610460k total, 17720k used, 592740k free, 576852k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4 root 15 -5 0 0 0 S 0 0.0 554:23.44 ksoftirqd/0 If I use br0 I get: vmstat -n 1 10 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo ...
Hello I found the culprit. When using powertop I get: Top causes for wakeups: 35,2% (251,0) ip : br_stp_enable_bridge (br_hello_timer_expired So I tried to turn them off with: brctl sethello br0 0 but the problem persisted. If I do brctl sethello br0 5 context switch rate drops 200 times and problem is gone. I think that the command brctl sethello br0 0 doesn't turn off hello messages, but sends them 250 times per second. Thanks for your time Dushan --
On Sun, 31 Aug 2008 10:51:46 +0200 You can't turn off the hello timer, it is needed for Spanning Tree to work. The kernel should reject requests to set hello timer < 1sec. Most routers allow 1 - 10sec. I am going to do a new patch to add tighter range checking for STP timer settings and another to default fowarding delay of zero if STP is disabled. --
The Spanning Tree Protocol timers need to be set within certain boundaries
to keep the internal protocol engine working, and to be interoperable.
This patch restricts changes to those timers to the values defined in IEEE 802.1D
specification.
The only exception to the standards are:
* if STP is disabled allow forwarding delay to be turned off
* allow wider range of ageing timer since this isn't directly part of
STP, and setting it to zero allows for non-remembering bridge.
Warning: this may cause user backlash since apparently working but standards
conforming configurations will get configuration errors that they didn't
see before.
--- a/net/bridge/br_ioctl.c 2008-08-31 10:00:44.000000000 -0700
+++ b/net/bridge/br_ioctl.c 2008-08-31 10:34:00.000000000 -0700
@@ -177,38 +177,63 @@ static int old_dev_ioctl(struct net_devi
}
case BRCTL_SET_BRIDGE_FORWARD_DELAY:
+ {
+ unsigned long t = clock_t_to_jiffies(args[1]);
if (!capable(CAP_NET_ADMIN))
return -EPERM;
+ /* enforce range checking per IEEE 802.1D 17.14 */
+ if (br->stp_enabled != BR_NO_STP &&
+ (t < 4*HZ || t > 30 * HZ))
+ return -EINVAL;
+
spin_lock_bh(&br->lock);
- br->bridge_forward_delay = clock_t_to_jiffies(args[1]);
+ br->bridge_forward_delay = t;
if (br_is_root_bridge(br))
br->forward_delay = br->bridge_forward_delay;
spin_unlock_bh(&br->lock);
return 0;
-
+ }
case BRCTL_SET_BRIDGE_HELLO_TIME:
+ {
+ unsigned long t = clock_t_to_jiffies(args[1]);
+
if (!capable(CAP_NET_ADMIN))
return -EPERM;
+ if (t < HZ || t > 15 * HZ)
+ return -EINVAL;
+
spin_lock_bh(&br->lock);
- br->bridge_hello_time = clock_t_to_jiffies(args[1]);
+ br->bridge_hello_time = t;
if (br_is_root_bridge(br))
br->hello_time = br->bridge_hello_time;
spin_unlock_bh(&br->lock);
return 0;
-
+ }
case BRCTL_SET_BRIDGE_MAX_AGE:
+ {
+ unsigned long t = clock_t_to_jiffies(args[1]);
if (!capable(CAP_NET_ADMIN))
return -EPERM;
+ /* ...On Sun, 31 Aug 2008 10:43:09 -0700 Why do we care ? You have to be the network administrator to set values, there are cases you may want to be out of the spec and you are privileged. The kernel does need to stop things being done which are fatal but running around restricting privileged administrators who have the ability to bring the network down anyway isn't its job. Seems bogus extra code to me - stops things working that should be allowed too. --
The timer configuration is propagated in network protocol, so misconfigured Linux box could survive but effect other devices on the network that are less robust. Maybe the small values would cause some other bridge to crash, go infinite loop, ... More likely robust devices might ignore our packets (because values out of range), leading to routing loops and other disasters. The kernel does need to stop administrative settings from taking out a network. If someone has a custom device or other non-standard usage, they can always rebuild the kernel and remove the range check. --
That would be irrelevant. CAP_NET_ADMIN lets you make that size mess If you have CAP_NET_ADMIN you can trivially take out the network unless it is properly switched. Now you might want your pretty little GUI and/or config tools to warn people that their configuration is outside 802 specs but that is a different matter altogether Alan --
Can one change the TCP maximum RTO to be smaller than specified in the specs? rick jones --
From: Rick Jones <rick.jones2@hp.com> We always min-clamp the RTO at RTO calculation time in order to be compatible with BSD's coarse grained times. --
But tuning TCP_RTO_MAX isn't permitted right? I'm drawing (perhaps flawed) parallels/distinctions between what is/isn't permitted to tweak for timers for one protocol versus another and wondering which may be a case of sauce for the goose/gander. rick jones --
Did you mean "apparently working but *non*-standards conforming"? Other than that, seems to be a sane application of "Be conservative in what you send". Our network is some 30K cat-5 ports, 1100 switches, 1300 wireless access points, and we appreciate it every time somebody makes things more bulletproof. And yes, we prefer things to out-and-out *fail* rather than run in a wonky configuration - hard failures usually get fixed in a few minutes, wonkiness can drag on for months of mystifying symptoms...
From: Stephen Hemminger <shemminger@vyatta.com> I don't think we can really add these kinds of restrictions wholesale like this. And the user is reporting that using brctl to turn off STP doesn't appear to actually turn off STP and thus fix all of the crazy ksoftirqd high cpu load problems. So what we need to do is resolve the user configuration issue that is causing this problem to begin with. --
The bridge hello time can't be safely set to values less than 1 second,
otherwise it is possible to end up with a runaway timer.
Signed-off-by: Stephen Hemminger <shemminger@vyatta.com>
--- a/net/bridge/br_ioctl.c 2008-09-04 15:25:41.000000000 -0700
+++ b/net/bridge/br_ioctl.c 2008-09-04 15:44:33.000000000 -0700
@@ -188,15 +188,21 @@ static int old_dev_ioctl(struct net_devi
return 0;
case BRCTL_SET_BRIDGE_HELLO_TIME:
+ {
+ unsigned long t = clock_t_to_jiffies(args[1]);
if (!capable(CAP_NET_ADMIN))
return -EPERM;
+ if (t < HZ)
+ return -EINVAL;
+
spin_lock_bh(&br->lock);
- br->bridge_hello_time = clock_t_to_jiffies(args[1]);
+ br->bridge_hello_time = t;
if (br_is_root_bridge(br))
br->hello_time = br->bridge_hello_time;
spin_unlock_bh(&br->lock);
return 0;
+ }
case BRCTL_SET_BRIDGE_MAX_AGE:
if (!capable(CAP_NET_ADMIN))
--- a/net/bridge/br_sysfs_br.c 2008-09-04 15:27:20.000000000 -0700
+++ b/net/bridge/br_sysfs_br.c 2008-09-04 15:33:31.000000000 -0700
@@ -29,11 +29,12 @@
*/
static ssize_t store_bridge_parm(struct device *d,
const char *buf, size_t len,
- void (*set)(struct net_bridge *, unsigned long))
+ int (*set)(struct net_bridge *, unsigned long))
{
struct net_bridge *br = to_bridge(d);
char *endp;
unsigned long val;
+ int err;
if (!capable(CAP_NET_ADMIN))
return -EPERM;
@@ -43,9 +44,9 @@ static ssize_t store_bridge_parm(struct
return -EINVAL;
spin_lock_bh(&br->lock);
- (*set)(br, val);
+ err = (*set)(br, val);
spin_unlock_bh(&br->lock);
- return len;
+ return err ? err : len;
}
@@ -56,12 +57,13 @@ static ssize_t show_forward_delay(struct
return sprintf(buf, "%lu\n", jiffies_to_clock_t(br->forward_delay));
}
-static void set_forward_delay(struct net_bridge *br, unsigned long val)
+static int set_forward_delay(struct net_bridge *br, unsigned long val)
{
unsigned long delay = clock_t_to_jiffies(val);
br->forward_delay = delay;
if ...From: Stephen Hemminger <shemminger@vyatta.com> Applied, thanks Stephen. I added more information to the commit message so that Dushan's incredibly contribution to this bug getting fixed are mentioned. I don't see how we would have figured out Bridging as even the cause without his detective work. So it's definitely wrong not to give him at least some mention in the commit message :-/ bridge: don't allow setting hello time to zero Dushan Tcholich reports that on his system ksoftirqd can consume between %6 to %10 of cpu time, and cause ~200 context switches per second. He then correlated this with a report by bdupree@techfinesse.com: http://marc.info/?l=linux-kernel&m=119613299024398&w=2 and the culprit cause seems to be starting the bridge interface. In particular, when starting the bridge interface, his scripts are specifying a hello timer interval of "0". The bridge hello time can't be safely set to values less than 1 second, otherwise it is possible to end up with a runaway timer. Signed-off-by: Stephen Hemminger <shemminger@vyatta.com> Signed-off-by: David S. Miller <davem@davemloft.net> --- net/bridge/br_ioctl.c | 8 +++++++- net/bridge/br_sysfs_br.c | 26 ++++++++++++++++++-------- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/net/bridge/br_ioctl.c b/net/bridge/br_ioctl.c index eeee218..5bbf073 100644 --- a/net/bridge/br_ioctl.c +++ b/net/bridge/br_ioctl.c @@ -188,15 +188,21 @@ static int old_dev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) return 0; case BRCTL_SET_BRIDGE_HELLO_TIME: + { + unsigned long t = clock_t_to_jiffies(args[1]); if (!capable(CAP_NET_ADMIN)) return -EPERM; + if (t < HZ) + return -EINVAL; + spin_lock_bh(&br->lock); - br->bridge_hello_time = clock_t_to_jiffies(args[1]); + br->bridge_hello_time = t; if (br_is_root_bridge(br)) br->hello_time = br->bridge_hello_time; spin_unlock_bh(&br->lock); return 0; + } case ...
I don't know what to say :) A little nitpick: 200 times greater context switch rate :), like Btw. is there a way to make the command to turn STP off work too? brctl stp br0 off Because AFAIK if I shut down STP the hello timer should shut down too, but it still continues to work. Thank you for your time and effort --
On Mon, 8 Sep 2008 23:35:19 +0200 The basics: * Hello timer is always enabled * STP defaults to off unless you turn it on * Turn STP on/off with brctl. In the existing design, the hello timer always runs, even when STP is not turned on. If STP is not enabled, the packet is just never created. Fixing it would not be hard (or gain much), but would have to deal with complex lock ordering and timer problems, so it isn't worth fixing for current releases. --
Hi On Sun, Aug 31, 2008 at 7:05 PM, Stephen Hemminger Well I try to turn stp off but it doesn't want to :) This is in my /etc/conf.d/net brctl_br0=( "setfd 0" "sethello 10" "stp off" ) I had problems with sethello 0 so now I'm using 10. Tried with brctl stp br0 off but still had same troubles. Have a nice day Dushan --
