Home » Mailing list archives » freebsd-current » 2008 » December » 9

Re: "interrupt storm..."; seems associated with an0 NIC

Previous thread: Re: i give up by Matt Dawson on Friday, December 5, 2008 - 7:36 am. (1 message)

Next thread: Labeling disks by Stefan Bethke on Friday, December 5, 2008 - 1:45 pm. (2 messages)
[view original message]
From: David Wolfskill
Subject: "interrupt storm..."; seems associated with an0 NIC
Date: Friday, December 5, 2008 - 10:48 am

After updating my laptop to CURRENT as of this morning, I now see

  interrupt storm detected on "irq11:"; throttling interrupt source

repeated indefinitely if I have inserted a Cisco/Aironet 350 PCCard.

Once the situation has been detected, the only way I've found to escape
is by power-cycling -- I can't even do anything with a serial console
(unless I had logged in to that serial console ahead of time -- in
that case, I was able to reboot gracefully).

Here's a list of the files that saw updates today:

U sys/cddl/contrib/opensolaris/uts/common/sys/isa_defs.h
U sys/dev/cxgb/common/cxgb_ael1002.c
U sys/dev/pccbb/pccbb.c
U sys/dev/pccbb/pccbb_pci.c
U sys/dev/pccbb/pccbbvar.h

And here are some other related bits:

FreeBSD g1-37.catwhisker.org 8.0-CURRENT FreeBSD 8.0-CURRENT #881: Fri Dec =
 5 06:38:46 PST 2008     root@g1-37.catwhisker.org:/common/S4/obj/usr/src/s=
ys/CANARY  i386

Here's what "vmstat -i" says when an0 hasn't been connected since the
last boot:

interrupt                          total       rate
irq0: clk                         637184        998
irq1: atkbd0                          46          0
irq4: uart0                         2261          3
irq6: fdc1                             1          0
irq7: ppc0                             6          0
irq8: rtc                          81638        127
irq11: cbb0 cbb1+*                  3112          4
irq14: ata0                        65357        102
Total                             789605       1237

And here's output from the same command shortly after I inserted the
NIC:

interrupt                          total       rate
irq0: clk                         671999        998
irq1: atkbd0                          46          0
irq4: uart0                         2300          3
irq6: fdc1                             1          0
irq7: ppc0                             6          0
irq8: rtc                          86095        127
irq11: cbb0 cbb1+*                  7907         ...
[ message continues ]
[view original message]
From: M. Warner Losh
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Friday, December 5, 2008 - 12:00 pm

Thanks.  Grump.  Will have to back out and try again.

Warner
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ilya Orehov
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 7:19 am

+------- M. Warner Losh, 2008-12-05 -------
| Thanks.  Grump.  Will have to back out and try again.

Hello!

I see storm too, but with 32-bit cards.

Thinkpad 600X, two cardbus cards: xl0 and ath0.

Since pccbb.c 1.176 and pccbb_pci.c 1.30
after rebooting with card(s) inserted or inserting any card
I see "interrupt storm...throttling..." messages about 1 per second.
Laptop remains usable, cards working.
Storm don't stop even if I eject all cards.
During storm, vmstat -i shows rate ~500 on cbb.
No messages appeared if laptop rebooted without cards
(until any card inserted).

Later revisions ( pccbb.c 1.178 and pccbb_pci.c 1.31) 
didn't bring any visible changes.

But this hack helped.
No storm detected, vmstat -i shows rate 0 or 1 on cbb.

diff -up xxx/pccbb_pci.c ./pccbb_pci.c
--- xxx/pccbb_pci.c	2008-12-06 11:56:00.000000000 +0300
+++ ./pccbb_pci.c	2008-12-09 14:08:03.000000000 +0300
@@ -689,6 +689,7 @@ cbb_pci_filt(void *arg)
 	struct cbb_softc *sc = arg;
 	uint32_t sockevent;
 	int retval = FILTER_STRAY;
+	int ack = 0;
 
 	/*
 	 * Read the socket event.  Sometimes, the theory goes, the PCI
@@ -722,6 +723,7 @@ cbb_pci_filt(void *arg)
 			sc->cardok = 0;
 			cbb_disable_func_intr(sc);
 			wakeup(&sc->intrhand);
+			ack = 1;
 		}
 		/*
 		 * If we get a power interrupt, wakeup anybody that might
@@ -732,7 +734,10 @@ cbb_pci_filt(void *arg)
 			cbb_set(sc, CBB_SOCKET_EVENT, CBB_SOCKET_EVENT_POWER);
 			sc->powerintr++;
 			wakeup((void *)&sc->powerintr);
+			ack = 1;
 		}
+		if (!ack) 
+			cbb_set(sc, CBB_SOCKET_EVENT, sockevent);
 		retval = FILTER_HANDLED;
 	}
 	/*

Do you need dmesg or some other info?

regards,
Ilya.

| 
| Warner
| _______________________________________________
| freebsd-current@freebsd.org mailing list
| http://lists.freebsd.org/mailman/listinfo/freebsd-current
| To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
| ...
[ message continues ]
[view original message]
From: M. Warner Losh
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 9:27 am

In message: <20081209141908.GA15845@artx.ru>

Can you please do the following:

+		if (!ack) {
+			printf("Need to ack %#x\n", sockevent);
+			cbb_set(sc, CBB_SOCKET_EVENT, sockevent);
+		}

And let me know what it says?

Warner
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ilya Orehov
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 12:24 pm

+------- M. Warner Losh, 2008-12-09 -------
| In message: <20081209141908.GA15845@artx.ru>
|             Ilya Orehov <elias@artx.ru> writes:
| : +------- M. Warner Losh, 2008-12-05 -------
| : | Thanks.  Grump.  Will have to back out and try again.
| : 
| : Hello!
| : 
| : I see storm too, but with 32-bit cards.
| : 
| : Thinkpad 600X, two cardbus cards: xl0 and ath0.
| : 
| : Since pccbb.c 1.176 and pccbb_pci.c 1.30
| : after rebooting with card(s) inserted or inserting any card
| : I see "interrupt storm...throttling..." messages about 1 per second.
| : Laptop remains usable, cards working.
| : Storm don't stop even if I eject all cards.
| : During storm, vmstat -i shows rate ~500 on cbb.
| : No messages appeared if laptop rebooted without cards
| : (until any card inserted).
| : 
| : Later revisions ( pccbb.c 1.178 and pccbb_pci.c 1.31) 
| : didn't bring any visible changes.
| : 
| : But this hack helped.
| : No storm detected, vmstat -i shows rate 0 or 1 on cbb.
| : 
| : diff -up xxx/pccbb_pci.c ./pccbb_pci.c
| : --- xxx/pccbb_pci.c	2008-12-06 11:56:00.000000000 +0300
| : +++ ./pccbb_pci.c	2008-12-09 14:08:03.000000000 +0300
| : @@ -689,6 +689,7 @@ cbb_pci_filt(void *arg)
| :  	struct cbb_softc *sc = arg;
| :  	uint32_t sockevent;
| :  	int retval = FILTER_STRAY;
| : +	int ack = 0;
| :  
| :  	/*
| :  	 * Read the socket event.  Sometimes, the theory goes, the PCI
| : @@ -722,6 +723,7 @@ cbb_pci_filt(void *arg)
| :  			sc->cardok = 0;
| :  			cbb_disable_func_intr(sc);
| :  			wakeup(&sc->intrhand);
| : +			ack = 1;
| :  		}
| :  		/*
| :  		 * If we get a power interrupt, wakeup anybody that might
| : @@ -732,7 +734,10 @@ cbb_pci_filt(void *arg)
| :  			cbb_set(sc, CBB_SOCKET_EVENT, CBB_SOCKET_EVENT_POWER);
| :  			sc->powerintr++;
| :  			wakeup((void *)&sc->powerintr);
| : +			ack = 1;
| :  		}
| : +		if (!ack) 
| : +			cbb_set(sc, CBB_SOCKET_EVENT, sockevent);
| :  		retval = FILTER_HANDLED;
| :  	}
| :  	/*
| : 
| : Do you need dmesg or some ...
[ message continues ]
[view original message]
From: M. Warner Losh
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 12:47 pm

In message: <20081209192439.GA16703@artx.ru>

What happens if you also print the current mask register?  CBB_SOCKET_MASK?

Warner
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ilya Orehov
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 1:44 pm

+------- M. Warner Losh, 2008-12-09 -------
| In message: <20081209192439.GA16703@artx.ru>
|             Ilya Orehov <elias@artx.ru> writes:
| : Need to ack 0x1
| 
| What happens if you also print the current mask register?  CBB_SOCKET_MASK?

Rebooted with xl0 card inserted,
first time (after initialization) mask=7,
after eject/insert xl0 mask=1.

...
xl0: Ethernet address: 00:60:08:d2:38:56
xl0: [ITHREAD]
Need to ack 0x1, mask=00000007
acd0: CDROM <TOSHIBA CD-ROM XM-1902B/1717> at ata1-master PIO4
Trying to mount root from ufs:/dev/ad0s2a
WARNING: attempt to net_add_domain(bluetooth) after domainfinalize()
xl0: reset didn't complete
xl0: command never completed!
xl0: command never completed!
xl0: command never completed!
tdkphy0: detached
miibus0: detached
xl0: detached
Need to ack 0x1, mask=00000001
xl0: <3Com 3c575TX Fast Etherlink XL> port 0x1000-0x103f irq 11 at device 0.0 on cardbus1
miibus0: <MII bus> on xl0
...

Rebooted once more, without card.
After card (xl0) inserted, mask=1.

Rebooted with same card inserted in second slot.
First time (after initialization) mask=7,
after eject/insert into same slot xl0 mask=1,
after eject card was inserted into first slot, mask=1,

code was:
if (!ack) {
	mask = cbb_get(sc, CBB_SOCKET_MASK);
	printf("Need to ack %#x, mask=%08x\n", sockevent, mask);
	cbb_set(sc, CBB_SOCKET_EVENT, sockevent);
}

regards,
Ilya.


| 
| Warner
| 
+-----------------------------
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: M. Warner Losh
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Tuesday, December 9, 2008 - 2:16 pm

Thanks!  This helps a lot.  The fact that xl works also is an
important hint for me for another problem I'm chasing...

does this patch cause the printfs we've added to not be hit?

Warner

Index: pccbb.c
===================================================================
--- pccbb.c	(revision 185750)
+++ pccbb.c	(working copy)
@@ -514,7 +514,7 @@
 		 * a chance to run.
 		 */
 		mtx_lock(&sc->mtx);
-		cbb_setb(sc, CBB_SOCKET_MASK, CBB_SOCKET_MASK_CD | CBB_SOCKET_MASK_CSTS);
+		cbb_setb(sc, CBB_SOCKET_MASK, CBB_SOCKET_MASK_CD);
 		msleep(&sc->intrhand, &sc->mtx, 0, "-", 0);
 		err = 0;
 		while (err != EWOULDBLOCK &&


In message: <20081209204404.GA17018@artx.ru>
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"

[ message continues ]
[view original message]
From: Ilya Orehov
Subject: Re: "interrupt storm..."; seems associated with an0 NIC
Date: Wednesday, December 10, 2008 - 3:19 am

+------- M. Warner Losh, 2008-12-09 -------
| Thanks!  This helps a lot.  The fact that xl works also is an
| important hint for me for another problem I'm chasing...
| 
| does this patch cause the printfs we've added to not be hit?

With patch printf still hit, but in different places,
and mask is different now: 6 during startup (later then without patch), 
and 0 or 6 after card insertion.
Tried (insert/eject) several times, sequence hardly repeatable,
but maybe it depends on timings :
if card inserted quickly after eject, mask=0.

That puzzled me, so I went to single user.

In single user, if rebooted without card:
card inserted -> no message,
card ejected -> "Need to ack 0x1, mask=00000006"

In single user too:
card inserted -> no message,
IP set with ifconfig xl0 -> "Need to ack 0x1, mask=00000006"
card ejected -> no message

Then i reverted patch to check in single user:

Reboot without card, single user.
Card inserted -> "Need to ack 0x1, mask=00000001"
IP set with ifconfig -> no message
card ejected -> no message

Reboot with card, 
Card initialized -> "Need to ack 0x1, mask=00000007"

regards,
Ilya.

| 
| Warner
| 
| Index: pccbb.c
| ===================================================================
| --- pccbb.c	(revision 185750)
| +++ pccbb.c	(working copy)
| @@ -514,7 +514,7 @@
|  		 * a chance to run.
|  		 */
|  		mtx_lock(&sc->mtx);
| -		cbb_setb(sc, CBB_SOCKET_MASK, CBB_SOCKET_MASK_CD | CBB_SOCKET_MASK_CSTS);
| +		cbb_setb(sc, CBB_SOCKET_MASK, CBB_SOCKET_MASK_CD);
|  		msleep(&sc->intrhand, &sc->mtx, 0, "-", 0);
|  		err = 0;
|  		while (err != EWOULDBLOCK &&
| 
| 
| In message: <20081209204404.GA17018@artx.ru>
|             Ilya Orehov <elias@artx.ru> writes:
| : +------- M. Warner Losh, 2008-12-09 -------
| : | In message: <20081209192439.GA16703@artx.ru>
| : |             Ilya Orehov <elias@artx.ru> writes:
| : | : Need to ack 0x1
| : | 
| : | What happens if you also print the current mask register?  CBB_SOCKET_MASK?
| : 
| : ...
[ message continues ]
Previous thread: Re: i give up by Matt Dawson on Friday, December 5, 2008 - 7:36 am. (1 message)

Next thread: Labeling disks by Stefan Bethke on Friday, December 5, 2008 - 1:45 pm. (2 messages)