login
Search
Actually, I think there is a bug in this place, but of course this might be unconnected. Anyway, Michael, could you try this patch? BTW, did you try with CONFIG_PACKET_MMAP disabled? Thanks, Jarek P. -----------------> Changing an skb after dev_queue_xmit() is illegal. And since it's inconsistent to treat specially net_xmit_errno() non-zero return, while ignoring other dev_queue_xmit() errors, there is no reason to break the loop in tpacket_snd() in this case. With debugging by: Stephen Hemminger <shemminger@linux-foundation.org> Reported-by: Michael Breuer <mbreuer@majjas.com> Signed-off-by: Jarek Poplawski <jarkao2@gmail.com> --- net/packet/af_packet.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-) diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index e0516a2..984a1fa 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -1021,8 +1021,9 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) status = TP_STATUS_SEND_REQUEST; err = dev_queue_xmit(skb); - if (unlikely(err > 0 && (err = net_xmit_errno(err)) != 0)) - goto out_xmit; + if (unlikely(err > 0)) + err = net_xmit_errno(err); + packet_increment_head(&po->tx_ring); len_sum += tp_len; } while (likely((ph != NULL) || @@ -1033,9 +1034,6 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) err = len_sum; goto out_put; -out_xmit: - skb->destructor = sock_wfree; - atomic_dec(&po->tx_ring.pending); out_status: __packet_set_status(po, ph, status); kfree_skb(skb); --
I did not try with CONFIG_PACKET_MMAP disabled. Please let me know which permutations of the following would be most valuable to test: Davids patch in/out The attached patch in/out CONFIG_PACKET_MMAP on/off I'm thinking your patch in, david's out, with and without --
I'm not sure about David's (except do it first ;-), but try my patch with CONFIG_PACKET_MMAP on; if there is no change, CONFIG_PACKET_MMAP off, without any patches. Thanks, Jarek P. --
This patch solves the original reported oops - as did Steven's patch of 12/26: [PATCH] sky2: make sure ethernet header is in transmit skb (I ran without Steven's patch and with this patch). Oddly, with this patch vs. Steven's - I'm getting software interrupt errors sporadically while not under load - with Steven's I get the frequently while under load (as per nethogs). For example: Jan 5 21:29:00 mail kernel: sky2 0000:06:00.0: error interrupt status=0x40000008 Jan 5 21:29:00 mail kernel: sky2 software interrupt status 0x40000008 Jan 5 21:29:00 mail kernel: sky2 Tx ring pending=124...125 report=125 done=125 Jan 5 21:29:00 mail kernel: 124: 0xb38de0be(5374) Jan 5 21:29:00 mail kernel: sky2 0000:06:00.0: error interrupt status=0x8 Jan 5 21:29:00 mail kernel: sky2 software interrupt status 0x8 Jan 5 21:29:00 mail kernel: sky2 Tx ring pending=126...127 report=126 done=127 Jan 5 21:29:00 mail kernel: 126: 0xb38d80be(9014) I also believe (can't prove yet) that my load test is slower with this patch vs. the sky2 patch. Is it possible that this patch is causing the transmission to momentarily halt such that the overall utilization appears low at the time I see the interrupt errors, vs. the other patch which perhaps doesn't interrupt the traffic flow quite so much? I haven't run yet with CONFIG_PACKET_MMAP disabled. --
Yes, without this patch xmit could be stopped earlier on some kind of errors, with retransmit of the last message possible. On the other hand, other dev_queue_xmit() (negative) errors, are ignored. So this place could be still improved by adding proper err handling (or removing getting err return from dev_queue_xmit() at all). Anyway, I think this patch should be a safe proposal for stable. If This should behave similarly as MMAP with this patch or maybe even better in case of errors. Thanks, Jarek P. --
On the other hand, returning just this one net_xmit_errno() would be consistent with non-MMAP sendmsg, so here is an alternative version for testing. Thanks, Jarek P. -----------------> (alternative #2) tpacket_snd() can change and kfree an skb after dev_queue_xmit(), which is illegal. With debugging by: Stephen Hemminger <shemminger@linux-foundation.org> Reported-by: Michael Breuer <mbreuer@majjas.com> Signed-off-by: Jarek Poplawski <jarkao2@gmail.com> --- net/packet/af_packet.c | 8 +++----- 1 files changed, 3 insertions(+), 5 deletions(-) diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index e0516a2..aba2049 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -1021,9 +1021,10 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) status = TP_STATUS_SEND_REQUEST; err = dev_queue_xmit(skb); - if (unlikely(err > 0 && (err = net_xmit_errno(err)) != 0)) - goto out_xmit; packet_increment_head(&po->tx_ring); + if (unlikely(err > 0 && (err = net_xmit_errno(err)) != 0)) + goto out_put; + len_sum += tp_len; } while (likely((ph != NULL) || ((!(msg->msg_flags & MSG_DONTWAIT)) && @@ -1033,9 +1034,6 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) err = len_sum; goto out_put; -out_xmit: - skb->destructor = sock_wfree; - atomic_dec(&po->tx_ring.pending); out_status: __packet_set_status(po, ph, status); kfree_skb(skb); --
On Wed, 6 Jan 2010 09:15:32 +0000 Good catch. Acked-by: Stephen Hemminger <shemminger@vyatta.com> -- --
First of all, we're incredibly lucky that such a helpful guy as Michael had misfortune to hit so helpfully broken piece of hardware ;-) Thanks, Jarek P. --
This patch at first behaved similarly to the previous one - seemed to be running a bit better... until the adapter went down :( This is the syslog output at the time the network failed: Jan 6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt status=0x40000008 Jan 6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008 Jan 6 14:11:01 mail kernel: sky2 Tx ring pending=125...126 report=125 done=126 Jan 6 14:11:01 mail kernel: 125: 0xc9c340be(8310) <network dropped as seen by clients here> Jan 6 14:12:51 mail kernel: DRHD: handling fault status reg 2 Jan 6 14:12:51 mail kernel: DMAR:[DMA Read] Request device [06:00.0] fault addr fffa9ebfe000 Jan 6 14:12:51 mail kernel: DMAR:[fault reason 06] PTE Read access is not set Jan 6 14:12:51 mail kernel: sky2 0000:06:00.0: error interrupt status=0x80000000 Jan 6 14:12:51 mail kernel: sky2 software interrupt status 0x80000000 Jan 6 14:12:51 mail kernel: sky2 Tx ring pending=68...69 report=68 done=68 Jan 6 14:12:51 mail kernel: 68: 0x9ebfe4ca(105) Jan 6 14:12:51 mail kernel: sky2 hardware interrupt status 0xd000000 Jan 6 14:12:51 mail kernel: sky2 Tx ring pending=68...69 report=68 done=68 Jan 6 14:12:51 mail kernel: 68: 0x9ebfe4ca(105) Jan 6 14:12:51 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010) Jan 6 14:12:51 mail smbd[6281]: [2010/01/06 14:12:51, 0] lib/util_sock.c:539(read_fd_with_timeout) Jan 6 14:12:51 mail smbd[6281]: [2010/01/06 14:12:51, 0] lib/util_sock.c:1491(get_peer_addr_internal) Jan 6 14:12:51 mail smbd[6281]: getpeername failed. Error was Transport endpoint is not connected Jan 6 14:12:51 mail smbd[6281]: read_fd_with_timeout: client 0.0.0.0 read error = Connection timed out. Much of the same... followed by sky2's unsuccessful attempts to reset the adapter - starting with NETDEV WATCHDOG oops: Jan 6 14:13:41 mail kernel: ------------[ cut here ]------------ Jan 6 14:13:41 mail kernel: WARNING: at net/sched/sch_generic.c:261 ...
I'm not sure: do you mean this patch above vs previous one by Stephen, or did you manage to try my "alernative #2" patch already? BTW, I forgot to mention, and maybe it doesn't matter here, but it would be better to (always) use my sky2 patch from Berck Nash's thread. Jarek P. --
This was using "alternative #2" patch. I didn't get the hang with alternative #1. Your sky2 patch from Berck Nash's thread was included in both cases; Stephen's was not. --
OK, so I guess "alternative #1" (above) seems safer to recommend for now (as I assumed earlier). On the other hand, we really don't know if it's only because it's because it's nicer for your hardware (or still some other bug around), so as before: let David choose ;-) BTW, I think you could still use Stephen's patch too (there might be still something more like this). There was also mentioned this network manager again. I might be wrong, but IMHO there could be some interaction even if it doesn't use this device; so could/did you try to disable it entirely? Thanks for testing! Jarek P. --
Just reran without the network manager - no change. Going to rerun with Stephen's new patch, alternative #1, and the patch from Berck Nash's thread. --
On Wed, 06 Jan 2010 14:49:38 -0500
Could you go back to baseline sky2 driver. The display code might be buggy.
These bits indicate an error in the MAC. The interrupt source enabled
is Transmit FIFO underrun.
Looking at how vendor driver handles this.
It looks like the Yukon EC_U chip doesn't really do Jumbo frames correctly.
Maybe not enough internal buffering to ensure that the whole packet
is in the chip. Of course, none of this is in the chip manual.
Does this help
--------------
--- a/drivers/net/sky2.c 2010-01-06 12:48:43.012318966 -0800
+++ b/drivers/net/sky2.c 2010-01-06 13:05:31.273987255 -0800
@@ -792,33 +792,21 @@ static void sky2_set_tx_stfwd(struct sky
{
struct net_device *dev = hw->dev[port];
- if ( (hw->chip_id == CHIP_ID_YUKON_EX &&
- hw->chip_rev != CHIP_REV_YU_EX_A0) ||
- hw->chip_id >= CHIP_ID_YUKON_FE_P) {
- /* Yukon-Extreme B0 and further Extreme devices */
- /* enable Store & Forward mode for TX */
-
- if (dev->mtu <= ETH_DATA_LEN)
- sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
- TX_JUMBO_DIS | TX_STFW_ENA);
-
- else
- sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
- TX_JUMBO_ENA| TX_STFW_ENA);
- } else {
- if (dev->mtu <= ETH_DATA_LEN)
- sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
- else {
- /* set Tx GMAC FIFO Almost Empty Threshold */
- sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
- (ECU_JUMBO_WM << 16) | ECU_AE_THR);
-
- sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
-
- /* Can't do offload because of lack of store/forward */
- dev->features &= ~(NETIF_F_TSO | NETIF_F_SG | NETIF_F_ALL_CSUM);
- }
- }
+ if ( (hw->chip_id == CHIP_ID_YUKON_EX && hw->chip_rev != CHIP_REV_YU_EX_A0) ||
+ hw->chip_id >= CHIP_ID_YUKON_FE_P) {
+ /* Yukon-Extreme B0 and further Extreme devices */
+ /* enable Store & Forward mode for TX */
+ sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
+ } else if (dev->mtu > ...I'll try this a bit later today. However, early on, I saw the same issues with MTU=1500. Also, maybe I'm missing something, but I can only recreate the issue with a high receive rate. Given the interaction with DHCP, for example, I'm thinking that there is some precondition that is as yet unknown. May be buggy hardware, or perhaps a race condition resulting in a corrupt i/o buffer somewhere. I'm wondering whether there's some useful place to insert some diagnostics on the RX side - at least we can see if there are any consistent events on the RX side preceding the TX error. --
Ok ... results - and maybe some more clues... Running with this patch; Jarek's "alternative 1", and the patch from the other thread. Not so good. No reported errors (sky2, etc.) - however with mtu=9000, lots of stuff broke: XDMCP; http via MASQ/netfilter, ssh connections intermittently (when large frames involved perhaps), etc. Tried to change mtu to 1500 on the fly, got a bunch of errors - and network watchdog kicked in. Have now rebooted with the same patches and mtu=1500. ... with mtu=1500, Everything is again working (i.e., XDMCP, netfilter, etc.) Load test with mtu=1500 went well for a while - high throughput sustained for a few minutes - then similar crash as before... but no interrup error messages this time until after the oops: <nothing of note before this> Jan 6 18:17:54 mail kernel: DRHD: handling fault status reg 2 Jan 6 18:17:54 mail kernel: DMAR:[DMA Read] Request device [06:00.0] fault addr 1bbfe000 Jan 6 18:17:54 mail kernel: DMAR:[fault reason 06] PTE Read access is not set Jan 6 18:17:54 mail kernel: sky2 0000:06:00.0: error interrupt status=0x80000000 Jan 6 18:17:54 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010) Jan 6 18:18:04 mail kernel: ------------[ cut here ]------------ Jan 6 18:18:04 mail kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0xf3/0x164() Jan 6 18:18:04 mail kernel: Hardware name: System Product Name Jan 6 18:18:04 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit queue 0 timed out Jan 6 18:18:04 mail kernel: Modules linked in: ip6table_filter ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec snd_hda_intel snd_hda_codec ...
Walked through the code based on Jarek's patches... came upon NET_CLS_ACT. At least in some cases (sch_cbq.c for example), the net transmit error could be returned from here... after releasing the skb. A quick scan of the various files in net/sched suggests that with NET_CLS_ACT the skb may or may not have been freed in the event of an error. If I have time later I'll see if I can bypass NET_CLS_ACT and see whether this is even relevant. --
Ok - so rerunning with Jarek's alternative #2 patch (the one that doesn't re-free the skb after the net_dev_enqueue error) and all else as above (your mtu patch on a clean 2.6.32.2 sky2.c, plus the devtpts inode patch) with an MTU of 1500 I can run without errors (including the interrupt errors previously reported). Changing MTU to 9000, everything basically breaks - Can't use X11 (local or remote - get X11 screen after gdm login locally, but then goes back to greeter; remote gets no greeter); ssh sessions hang; etc. This time I was able to reset the MTU back to 1500 without a reboot - but I did have to ifconfig eth0 down and then up. Looking at the sk98lin code, it looks to me like they do a bit more work with existing buffers before completing the MTU switch. Note that even doing this, X11 did not work (it did with the old mtu change code). Tried changing to mtu 4500 - same effect as 9000... but when I switched back to 1500, ksoftirqd started spinning using 100% of one core. Running with these patches and 1500 MTU seems stable, but the network is running rather slow. Latency is OK, but peak transfer rates seem to be running about 20% of what I saw while receiving the interrupt errors with earlier patches. I'll leave this running overnight to confirm that at least the errors and hangs are resolved with this patch set. --
On Wed, 06 Jan 2010 23:00:34 -0500 The problem is that patch was enabling scatter-gather and checksum offload that won't work on EC_U hardware with 9K MTU. At least, it never worked for me when I tested it. So because of that it really doesn't change anything for the better on that chip version. What version chip is on that motherboard? Mine is: Yukon-2 EC Ultra chip revision 3 which corresponds to B0 step. Another possibility is the PHY register which controls number of ticks of buffering. The default is zero, which gives the most buffering (good), but the firmware could be reprogramming it (bad). In general, the driver doesn't fiddle with bits that are already set correctly, because sometimes vendors need to tweak PCI timing in firmware/BIOS. It seems the firmware on this chip is just a bunch of register setups done on power on. --
So at this point, things are working as mentioned - but really slow...
at least an order of magnitude slower than with the other set of
patches. The other set also generated errors and was not stable :( But,
that set worked with mtu=9000, this set doesn't seem to work with
anything over 1500. The slowdown may also be (based on earlier testing)
attributable to Jarek's alternative #2 patch. As to the chip, I *think*
we have the same chip - I'm including lspci -vv - perhaps there is
something different.
06:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E
Gigabit Ethernet Controller (rev 14)
Subsystem: Giga-byte Technology Device e000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 58
Region 0: Memory at fbdfc000 (64-bit, non-prefetchable) [size=16K]
Region 2: I/O ports at d800 [size=256]
Expansion ROM at fbdc0000 [disabled] [size=128K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: Marvell Yukon 88E8056 Gigabit Ethernet
Controller
Read-only fields:
[PN] Part number: Yukon 88E8056
[EC] Engineering changes: Rev. 1.4
[MN] Manufacture ID: 4d 61 72 76 65 6c 6c
[SN] Serial number: AbCdEfG001C3B
[CP] Extended capability: 01 10 cc 03
[RV] Reserved: checksum good, 9 byte(s) reserved
Read/write fields:
[RW] ...Also - I'm seeing a huge number of dropped packets (RX) 200-300/second.
Probably why this is so slow.
Current ifconfig:
eth0 Link encap:Ethernet HWaddr 00:26:18:00:1C:3B
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:18ff:fe00:1c3b/64 Scope:Link
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:26647536 errors:0 dropped:517884 overruns:0 frame:0
TX packets:12112780 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:38960063319 (36.2 GiB) TX bytes:1889879762 (1.7 GiB)
Interrupt:18
--
Never mind... spoke too soon. Crashed again. Just took longer: Jan 7 00:37:39 mail kernel: DRHD: handling fault status reg 2 Jan 7 00:37:39 mail kernel: DMAR:[DMA Read] Request device [06:00.0] fault addr fff1401fe000 Jan 7 00:37:39 mail kernel: DMAR:[fault reason 06] PTE Read access is not set Jan 7 00:37:39 mail kernel: sky2 0000:06:00.0: error interrupt status=0x80000000 Jan 7 00:37:39 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010) Jan 7 00:37:40 mail smbd[4729]: [2010/01/07 00:37:40, 0] lib/util_sock.c:539(read_fd_with_timeout) Jan 7 00:37:40 mail smbd[4729]: [2010/01/07 00:37:40, 0] lib/util_sock.c:1491(get_peer_addr_internal) Jan 7 00:37:40 mail smbd[4729]: getpeername failed. Error was Transport endpoint is not connected Jan 7 00:37:40 mail smbd[4729]: read_fd_with_timeout: client 0.0.0.0 read error = Connection timed out. Jan 7 00:37:40 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:40 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:41 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:41 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:44 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:44 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 7 00:37:47 mail kernel: ------------[ cut here ]------------ Jan 7 00:37:47 mail kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0xf3/0x164() Jan 7 00:37:47 mail kernel: Hardware name: System Product Name Jan 7 00:37:47 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit queue 0 timed out Jan 7 00:37:47 mail kernel: Modules linked in: ip6table_filter ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd nfs_acl auth_rpcgss exportfs hwmon_vid coretemp ...
... Reapplied a couple of earlier patches - still can't do jumbo frames, but the rx errors are gone and speed has improved. Too early to assure that it's stable. Patches that seem to fix the rx drops (all from Stephen): 1) Patch change to tx_init 2) Patch to lock netif_device_detach 3) Patch to sky2_tx_complete to add netif_device_present test Also in the mix: Jarek's alternative 2 With this set and mtu=1500 all seems good - decent if not stellar throughput; no logged errors; no reported packet loss. As before, will leave running and see if anything falls apart. --
BTW, the main difference between alt. 1 and 2 is error notification: alternative 2 doesn't hide some (most) of drops, so, dependending on app, there might be more and faster retransmits. (I don't know what Good news! Jarek P. --
Unless I misread the code, I think that in some cases e skb is actually freed if the cfq (among others perhaps) scheduler returns an error on enqueue (flow control perhaps). Thus with alternative 1, it is possible that the skb is acted upon after being freed - this would be consistent --
I can't see your point: could you give some scenario? Jarek P. --
With NET_CLS_ACT set, net_dev_enqueue can return an error after freeing the skb. Alternative 1 disregards the error and assumes the skb is still valid. The original code and alternative 2 exit the loop assuming the skb has been freed. --
Not exactly: alternative 1 disregards the error, and tries to send next skbs if the message was longer. After consuming all the message it returns without err code (at least wrt. dev_queue_xmit). This is quite often practice to skip dev_queue_xmit() return (try to grep in net\). It should never touch any part of an earlier sent skb. Jarek P. --
Did I say "net\"?! Extremely sorry! ;-) Jarek P. --
Bad news - crashed about an hour after I wrote this email - under load - same crash as before. Network watchdog... lots of attempts to reset the adapter... then hw watchdog rebooted the system. --
It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off if you find time. Jarek P. --
Ok - any particular patch set to try with? I'm going to start with a clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but can't get a usable console... will look at that later.) --
Never tried earlier than 2.6.31 - new machine. I had a bunch of issues on 2.6.31, including sky2 DMAR errors... so probably not a 2.6.32 regression. --
My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW, could you remind if it worked any better with 2.6.31 or earlier?) Jarek P. --
I'm not sure my crash-and-burn runs yesterday included the pskb_may_pull patch :( Going to rerun with these patches and with and without MMAP. Will also retry both with jumbo frames if possible. --
If MMAP then some "alternative" too. But first no MMAP. Jarek P. --
Another things IMHO worth to try: a sky2 module parameter "disable_msi=1", and CONFIG_DMAR off. Jarek P. --
Ok - that'd be with or without MMAP enabled? (note that so-far, without MMAP I'm not seeing any errors - throughput is running about half what I was seeing with MMAP enabled (before crashing that is). CPU is also way busier (also to be expected). One other observation - I had been seeing lots of DNS errors - IPV6 related format errors I really didn't think much of it as they were mostly .ru and seemed spam-related, but now I don't see any. Haven't updated bind; doubt the world has changed - so perhaps this is related to the network issue. --
MMAP enabled (with some "alternative" patch - to avoid known bugs) should give as earlier the answer if these changes matter. But first let's try longer (if possible) if this "no MMAP" could really heal your hardware. Jarek P. --
Results: * no MMAP, mtu=1500, neither alternative patch loaded: adapter crashed: Jan 7 15:44:23 mail kernel: DRHD: handling fault status reg 2 Jan 7 15:44:23 mail kernel: DMAR:[DMA Read] Request device [06:00.0] fault addr fffb7bffe000 Jan 7 15:44:23 mail kernel: DMAR:[fault reason 06] PTE Read access is not set Jan 7 15:44:23 mail kernel: sky2 0000:06:00.0: error interrupt status=0x80000000 Jan 7 15:44:23 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010) Jan 7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24, 0] lib/util_sock.c:539(read_fd_with_timeout) Jan 7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24, 0] lib/util_sock.c:1491(get_peer_addr_internal) Jan 7 15:44:24 mail smbd[6572]: getpeername failed. Error was Transport endpoint is not connected Jan 7 15:44:24 mail smbd[6572]: read_fd_with_timeout: client 0.0.0.0 read error = Connection timed out. Jan 7 15:44:44 mail kernel: ------------[ cut here ]------------ Jan 7 15:44:44 mail kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0xf3/0x164() Jan 7 15:44:44 mail kernel: Hardware name: System Product Name Jan 7 15:44:44 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit queue 0 timed out Jan 7 15:44:44 mail kernel: Modules linked in: ip6table_filter ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq snd_seq_device snd_pcm gspca_spca505 gspca_main firewire_ohci videodev v4l1_compat firewire_core pcspkr v4l2_compat_ioctl32 snd_timer iTCO_wdt i2c_i801 crc_itu_t iTCO_vendor_support snd soundcore snd_page_alloc sky2 wmi asus_atk0110 hwmon fbcon tileblit ...
BTW, was there any other oops saved before this one? ??? Hmm... Alternative 1 or 2 doesn't even compile into when no MMAP, Very interesting. It would be nice to give it a really long try, and OK, for now let's make sure this MMAP + NO DMAR + disable_msi is really really working. Thanks, Jarek P. --
I see your point. I'm pretty sure that run failed miserably. Perhaps something else is going on - some sort if intermittent thing that just Still up - no kernel errors reported. There was a large dropped packet rate (RX) which seems to actually correlate with DNS format error messages (ipv6 only). I spent some time tracking those down. Interestingly, most pointed back to one netblock & one ISP (0qf.ru). I blocked that domain and the errors expectedly dropped - as did the RX dropped packet rate. Since booting this version yesterday eth0 shows 1752944 dropped packets. 1752939 of those happened before I blocked the domain about 8 hours ago. I have run load tests since as well. I think this dns activity is sendmail attempting to validate spam - but not 100% sure yet as I can't correlate the .ru domain with anything sendmail has reported. I'm running a sniffer now hoping to catch the next one. I *think* but can't prove, that these are coming in via sendmail - i.e., bad email, not even spam really - just enough to get a system configured to do dns lookups as part of spam filtering to connect to the server in question. What comes back would seem to be corrupt ipv6 packets. This gets us back somewhat to Berck Nash's reported problem. His report of sky2 failure was due to external attack. Could this be related? Is it possible that absent some set of the patches & config settings in this version that ipv6 bind activity involving corrupt (perhaps intentionally) packets is breaking something? Will try rerunning without disable_msi later (after I catch the dns --
It looks like very timing dependent and without MMAP it can work just below limit, which could trigger those dmar or msi bugs. Anyway, it seems to show there is no another serious bug in the MMAP part (i.e. Berck Nash reported oopses during sky2 TX timeout recovery, which are generally hardware/driver problems, and shouldn't be triggered by ip BTW, don't hurry with that yet, but in the next test, please try This patch is very helpful for debugging, but I doubt appropriate for the mainline if it isn't triggered any more after 1) fix. But, please keep it yourself for some time in all these tests (and of Very nice :) Jarek P. --
My thought was that his crash was secondary to the netdev watchdog & Will do - still up from yesterday... no more dropped packets... none of the dns errors either. To be expected I suppose as long as I'm trying to sniff it. Assuming no immediate erorrs with alt2, no DMAR + disable_msi I'll report back after it's been up for a while. --
Yes, the netdev watchdog which triggers on TX timeout. Jarek P. --
Ok - ran alt1 and alt2, both with MMAP, no DMAR and disable_msi. Both seem to behave similarly. No logged errors; large numbers of dropped RX packets. One odd thing: when driving every sort of traffic through, I was able to hose the client adapter (win7) repeatedly by runnnig the win7 backup and connecting Windows Media Player to a Mediatomb stream while also running a remote X11 session. Looks like the SSDP traffic that occurs at the same time as the SMB traffic and X11 traffic takes out the adapter on Win7 - Nforce. Reran with msi enabled, MMAP and no DMAR. Also no errors; much faster, and the Win7 side survives the same conditions that don't work when msi is disabled. Doesn't make sense to me, but it is what it is. --
Tracked it down - this appears to be result of Win7 enabling "EnableDeadGWDetect" by default. Long story short, if this is set and a packet sent to the gw is retransmitted more than, 1/2 the value of maxdataretransmissions (default of 5), then the gateway is declared dead and the next one selected. If there is only one gateway, then the default gateway is effectively removed. So under load where the server is the default gateway, this is not a surprising result. With msi enabled, the system responds better, fewer retransmissions, no dropping --
Ok - since alt2 introduces less changes and is acked by Stephen already, I'll resend it with your "Tested-by" in a new thread to clear You might try some tweaking with another sky2 parameter "copybreak" It looks like DMAR is the main candidate for a new linux-kernel@ or bugzilla bug report. You should also consider reporting these ipv6 problems, especially if you think they can trigger TX timeouts. (In both cases it would be good to try current mainline first, when you get it workable.) Thanks, Jarek P. --
I'll play around with this after I figure out what's actually being dropped and why. Looking at ethtool, over 99% of RX packets are in the --
Just an FYI - 2.6.32.3 with alt 3 af_packet patch & sky2 pskb_may_pull --
Hmm... What a pity! It was such a useful debugging tool for networking ;-) BTW, I'm not sure if "runs OK" means with or without those DHCP drops & large packets you described. Thanks, Jarek P. --
As of now, no errors even when blasting traffic & forcing dhcp packets as before. I haven't tried putting mtu back to 9k yet. OK means that there are no obvious differences in behavior with or without DMAR all else being equal. There were some updates made to stable that could have fixed this - I'd guess intel_iommu fixes helped. If it helps, I'm still getting one error without DMAR enabled - at startup, there's a DMA sync oops - mismatch of 72 bytes coming from sky2. That oops was posted previously - with DMAR (re) enabled, there's no related oops. --
I hope re-posting this oops with this information should be helpful. Jarek P. --
Update: after leaving the system up for a few days, I hit the DMAR error again. This happened during a scheduled backup from my win7 box. A reboot was required to re-enable eth0. After the error, eth0 was receiving, but was unable to transmit. For example, the log reported arp bogons; DHCPINFORM/ACK sequences (where the ACK that was logged was not transmitted), etc. The log was filled with sky2 eth0: tx timeout messages; as well as disable/enable of eth0. I attempted to get things up again without a reboot, but failed. Even rmmod & insmod did not fix whatever was broken on the TX side. Note that this is similar to the earlier sky2 errors I had under load with the variety of patches, and with or without DMAR enabled. Just took way longer this time. Note that eth1 remained functional. Unfortunately, with the latest set of patches installed, this is no longer reproducible at will. I'd guess therefore that the patches narrowed some hole, but didn't close it. Relevant log portions: Jan 17 05:29:49 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 17 05:29:49 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 (mbitouch) via eth0 Jan 17 05:36:49 mail kernel: DRHD: handling fault status reg 2 Jan 17 05:36:49 mail kernel: DMAR:[DMA Read] Request device [06:00.0] fault addr ffe7957fe000 Jan 17 05:36:49 mail kernel: DMAR:[fault reason 06] PTE Read access is not set Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: error interrupt status=0xc0000000 Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010) Jan 17 05:36:49 mail smbd[14840]: [2010/01/17 05:36:49, 0] lib/util_sock.c:539(read_fd_with_timeout) Jan 17 05:36:49 mail smbd[14840]: [2010/01/17 05:36:49, 0] lib/util_sock.c:1491(get_peer_addr_internal) Jan 17 05:36:49 mail smbd[14840]: getpeername failed. Error was Transport endpoint is not connected Jan 17 05:36:49 mail smbd[14840]: read_fd_with_timeout: client 0.0.0.0 read error = Connection ...
My proposal is to send some summary as a new thread, with dmar in the It would be nice to name those patches each time. Anyway, try this again without DMAR. Thanks, ... --
Not sure I agree. The symptoms are identical to those I hit without DMAR earlier on. Also, as this issue only happens when there is high receive load, I'm thinking there's some sort of race between TX and RX within the sky2 driver, or hardware. I think that DMAR is correctly catching My bad: was running with the af_packet.c version 3 patch; and Stephen's v4 patch from last week. Both on 2.6.32 from git (so 2.6.32.4). Can't move back to head as I've hit two unrelated issues. --
Hmm... OK, then let's wait with this report and go back to testing it "really really long" ;-) without DMAR, and maybe without the last Stephen's patch either? (So only the two things in the current linux-2.6.) Jarek P. --
Ok - but absent the last patch, I think I still need the pskb_may_pull patch... so it'd be pskb_may_pull and afpacket v3 and no DMAR. Also - not sure if related, but there's still the odd tx side behavior when RX is under load. That I CAN reproduce at will (yesterday's report - no crash, but I confirmed that DHCPOFFER packets are being dropped somewhere after wireshark sees them and before hitting the wire. I am also wondering whether or not that testing I did yesterday set up today's hang - perhaps those lost TX packets are corrupting something that manifests worse later. --
Exactly. Or if it's working for you already, the mainline (2.6.33-rc4) I'm not sure either, but until there is no crash it might be some minor bug or/and missing stat. Btw, you could probably try alternative Maybe, but you wrote earlier they had to fix something around this DMAR in the meantime, because it triggered much faster during your previous tests. So, I don't know why you assume this DMAR has to be correct this time. Jarek P. --
Ok - up on the two patches, no DMAR. Some early observations: 1. There's an early on MMAP oops (see below). This happens once, at the completion of the transition to runlevel 5 (I've seen it entering runlevel 3 as well). This does not recur when runlevels are subsequently changed. I do not see this when running with DMAR enabled. 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it still happens. Interestingly, I initially saw no dropped packets with ping - but after I went the DCHP route and eventually reconnected, I could then cause dropped tx packets with ping. To clarify: a) start throughput b) ping device - no packet loss - this was true for the entire test run. c) start throughput again d) ping - no loss. e) drop wifi on the device & restart - first attempt worked. Repeat attempt yielded the dropped DHCPOFFER packets. After about 6 tries, the device reconnected to wifi. f) ping again (after the reconnection) - packet loss rate about 80%. g) simultaneously ping the wifi router - no loss. h) After a while, packets are no longer dropped during ping. If I manage to cause the dhcp drop again, and then ping after the device finally reconnects, packet loss is significant for a while (maybe 30 sec to a minute). Then things return to normal. Note that the packet loss continues even if the reported throughput drops to nil. i) I can't cause the initial packet loss at RX rates below about 30,000KBPS (as reported by nethogs). At rates over 40 I can reproduce this on this set of patches & config about 60% of the time. The initial sky2 oops: Jan 18 10:42:43 mail kernel: ------------[ cut here ]------------ Jan 18 10:42:43 mail kernel: WARNING: at lib/dma-debug.c:898 check_sync+0xbd/0x426() Jan 18 10:42:43 mail kernel: Hardware name: System Product Name Jan 18 10:42:43 mail kernel: sky2 0000:06:00.0: DMA-API: device driver tries to sync DMA memory it has not allocated [device address=0x00000003249b4022] [size=98 bytes] Jan 18 10:42:43 mail ...
OK, you mentioned this oops (actually a warning only) happened during Btw, I guess you improved the test because you didn't mention it here, even after my explicit question?: I forgot to mention, but did you try to check if these lost ping packets are "being dropped somewhere after wireshark sees them and before hitting the wire" like DHCPOFFER? Aren't there any sky2 warnings/resets while this happens? Jarek P. --
Yes - dk if it's significant or not. Only obvious difference between I had been focusing on the hangs - dhcp causing the initial crash from December. After things stabilized with the af patch & skb may pull I started noticing the dropped tx packets. I reported the TX loss on the Yes. There are no errors, and no statistics anywhere that I know to look reflect the loss. Nothing in netstat; ethtool -S; etc. The only loss reported is RX. The recent TX warnings/resets happened while the machine was up for several days and while unattended and under high RX load. --
On Mon, 18 Jan 2010 15:56:45 -0500 You might want to use tc filter rule to set priority of DHCP packets higher. This would cause them to be in a separate queue and eliminate the problem. -- --
But it looks like they are dropped in TX after passing a qdisc (and wireshark). Jarek P. --
Ok - for fun, tried that - no change. Not sure I see why this might be a factor. The packet loss happens when TX load is low and RX high. Also, packets only being dropped if traversing a router vs.to the router itself. Keep in mind that pings to the router did not lose packets, pings through the router lost packets. The router was not under load (traffic is being generated from a device connected via the 1Gb switch, not the wifi router), and tcpdump on the router input port shows the pings to the router, but not the ones through the router. One added note, when I just tried this, the test data ended while the packet loss was occurring. The DHCPOFFER packet loss did not clear until about a minute after the throughput abated. I really think something is getting hosed, and I'd but some weird interaction with the arp logic high on the list of suspects. Not sure what else would be a factor when looking at the extra hop on the same subnet. --
Good point! Actually, IIRC, your setup might be a problem: you seem to have two switches on the path (I guess the router is a bridge for these wireless), so I wonder if it's not something between them. Jarek P. --
OK, let's try (as long as possible) if it can break so hard as with OK, but we need to establish some status quo after these patches before any new things (including DMAR), so I'd suggest trying this Please check "tc -s qdisc" each time as well. Jarek P. --
Some output from tc -s qdisc: Before test: qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 35279532 bytes 291080 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 377308 bytes 3107 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 During test (after initial observed packet loss): qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 123389424 bytes 1781403 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 400862 bytes 3250 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 During test - while packet loss occuring: qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 150518974 bytes 2138312 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 422003 bytes 3432 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 After the conclusion of the test: qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 244900497 bytes 3416350 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 564380 bytes 4708 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 During the test, 8.9GB received; 232.9MB sent). I also connected a second device through the wifi router. I was able to ping that device w/o loss while DHCP packets were being dropped to ...
Could you remind us if the problem is always with this first device? Btw, I wonder if you could test it skipping the (HP?) switch? Jarek P. --
If so, then of course don't forget to try tcpdump on the router. Jarek P. --
Well - no.... but I'm not sure that would show anything.
Setup diagram:
Server->gb switch-> (100mb) wifi router -> devices
|
Win7 PC (gb)
The problem does not occur (at least I haven't been able to recreate it)
at 100mb, and the wifi router doesn't do 1Gb. I drive the traffic from
the win7 PC to the server. I've seen the loss when the only traffic
going through the wifi router was ping & dhcp. I've also never seen any
loss on a device directly attached to the 1GB switch. I can drive load
through the wifi router while driving load from the Win7 box, but don't
see TX packet loss at all when not doing DHCP RELEASE/RENEW.
As there is no packet loss to devices not involved in the DHCP sequence
through the same path, I'm not really sure that the GB switch is implicated.
As I don't have a standalone sniffer, I'm thinking that it might be
easier to instrument places where the TX packet could be dropped and see
at least whether it's getting to the card.
Given the circumstances of the TX drop, and that it was DHCP traffic
while under load that caused the oops rectified with the two patches,
I'm thinking that the packet loss is the current manifestation of
whatever the underlying problem is. Given the extra hop required to
break things, and given that a dhcp release/renew seems to trigger
things, I keep coming back to arp logic as being somehow implicated.
If arp is somehow involved, then I'd expect to see manifestations under
similar circumstances with other drivers. As the pskb_may_pull patch
stopped the crash, perhaps other drivers do suffer packet loss and it's
just not been widely noticed or attributed to the kernel - especially if
the network topology is a factor. I do know people at large enterprises
who have been complaining of what *could* be this same issue, however
they're currently blaming their switch vendors. As most traffic is TCP,
this is really only noticed by those few places deeply concerned with ...Ok - one last update for a while ...not sure what's next... I put some printk's into sky2.c xmit logic - the packets are being sent to the card, and the i/o's are completing successfully. So it would seem either the switch is dropping the packets, or else the wifi router is. As tcpdump doesn't show the packets arriving on the wifi router, I'm leaning towards the switch. I ran wireshark on the win7 box to see what is coming off the switch. I did notice one thing that's visible to the win7 box but is not showing up on the linux wireshark - before every successful dhcpoffer, there's an XID message broadcast from the device. I'm wondering why I don't see this on the linux side: The packet is from the mac of the device, dst ff:ff:ff:ff:ff:ff; protocol eth:llc... hex packet: ffffffffffff001cccf39ff600060001af810100. Now I guess I've got some reading to do... I've got no idea what the correct application of llc messages would be given my topology :(. I do suspect that the llc stuff (or lack thereof under some conditions) is causing the switch to fail to forward the dhcpoffer message. As the dhcpoffer message is not broadcast, but directed to the remote mac address and as that address is not connected directly to the switch, I'm guessing that under some conditions whatever tells the switch how to find the mac is missing. I'd guess that the wifi router should be letting the switch know around the time it forwards the first arp and/or DHCP broadcast message from the client... or maybe the linux box should be doing something before the offer. So net-net, as far as my TX packet loss issue, sky2 is in the clear. If something on the linux side should be informing the switch about something then there may still be an issue. If the wifi router should be doing something differently, then it's unfortunately likely a 2.4.37 kernel issue (That's what dd-wrt is using). --
IMHO until there is no proof from a sniffer or some regs dumps the switch and the router are more suspicious than your NIC or linux box. Then debugging these other things isn't so much interesting from my POV ;-) Anyway, if you only want to get it working (instead of debugging), it seems you might try moving the dhcp server to the router or maybe even using two separate servers with their pools - unless I missed something in your config. Jarek P. --
I'm going to take a stab at debugging before creating a new subnet. Thanks. --
Postscript: HP has confirmed that the 1400-8g switch doesn't work, "well," in a mixed 100Mb/1000Mb environment, and is dropping DHCP packets. --
... Btw, could you try if this patch can makes difference in triggering the "lib/dma-debug.c:898" warning? Jarek P. --- drivers/net/sky2.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c index 7650f73..e02e9e9 100644 --- a/drivers/net/sky2.c +++ b/drivers/net/sky2.c @@ -2479,6 +2479,9 @@ static int sky2_status_intr(struct sky2_hw *hw, int to_do, u16 idx) port = le->css & CSS_LINK_BIT; dev = hw->dev[port]; + if (!netif_running(dev)) + continue; + sky2 = netdev_priv(dev); length = le16_to_cpu(le->length); status = le32_to_cpu(le->status); --
Still get the warning... but now 60 bytes. Jan 19 10:43:50 mail kernel: ------------[ cut here ]------------ Jan 19 10:43:50 mail kernel: WARNING: at lib/dma-debug.c:902 check_sync+0xc1/0x43f() Jan 19 10:43:50 mail kernel: Hardware name: System Product Name Jan 19 10:43:50 mail kernel: sky2 0000:04:00.0: DMA-API: device driver tries to sync DMA memory it has not allocated [device address=0x0000000320a0b022] [size=60 bytes] Jan 19 10:43:50 mail kernel: Modules linked in: microcode(+) ip6table_filter ip6table_mangle ip6_tables iptable_raw iptable_mangle ipt_MASQUERADE iptable_nat nf_nat appletalk psnap llc nfsd lockd nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec snd_hda_intel ac97_bus snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm gspca_spca505 snd_timer gspca_main firewire_ohci snd soundcore videodev i2c_i801 firewire_core v4l1_compat snd_page_alloc v4l2_compat_ioctl32 pcspkr wmi crc_itu_t iTCO_wdt iTCO_vendor_support asus_atk0110 sky2 hwmon fbcon tileblit font bitblit softcursor raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbf Jan 19 10:43:50 mail kernel: illrect [last unloaded: ip6_tables] Jan 19 10:43:50 mail kernel: Pid: 0, comm: swapper Not tainted 2.6.32.4MMAPNODMARAF3SKY2PSKBMAYPULL-00912-g914160d-dirty #3 Jan 19 10:43:50 mail kernel: Call Trace: Jan 19 10:43:50 mail kernel: <IRQ> [<ffffffff810536ee>] warn_slowpath_common+0x7c/0x94 Jan 19 10:43:50 mail kernel: [<ffffffff8105375d>] warn_slowpath_fmt+0x41/0x43 Jan 19 10:43:50 mail kernel: [<ffffffff8127b871>] check_sync+0xc1/0x43f Jan 19 10:43:50 mail kernel: [<ffffffff8107a683>] ? ...
On Tue, Jan 19, 2010 at 10:47:27AM -0500, Michael Breuer wrote: Thanks for trying this. Jarek P. --
NP. Let me know what else I can do to help with this one. Note that it seems innocuous. I'll continue with the older set and no dmar for now and see if I get another crash. --
I'll try to look at it more. Since, as you predicted, this might be connected with dmar problems, it needs some ideas for debugging/fixing and until this time you could give it a rest from crashing. Jarek P. --
Maybe one more idea to try (not compiled).
Jarek P.
---
drivers/net/sky2.c | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7650f73..cdebdd3 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2252,12 +2252,14 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2,
skb = netdev_alloc_skb_ip_align(sky2->netdev, length);
if (likely(skb)) {
pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
- length, PCI_DMA_FROMDEVICE);
+ pci_unmap_len(re, data_size),
+ PCI_DMA_FROMDEVICE);
skb_copy_from_linear_data(re->skb, skb->data, length);
skb->ip_summed = re->skb->ip_summed;
skb->csum = re->skb->csum;
pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
- length, PCI_DMA_FROMDEVICE);
+ pci_unmap_len(re, data_size),
+ PCI_DMA_FROMDEVICE);
re->skb->ip_summed = CHECKSUM_NONE;
skb_put(skb, length);
}
--
That not only compiled, but it cleared the error as well. Additionally, I used to see a bit of a delay receiving the login prompt when first connecting to the box by ssh. That delay is gone with this patch. I'd guess that the warning wasn't quite as innocuous as I thought. Note: tested on 2.6.32.4. I'll leave this up for a bit before attempting to move back to head. --
On Tue, 19 Jan 2010 20:01:10 -0500 Seems like an underlying bug in the DMA api. Maybe it just can't handle operations on partial mapping. Other drivers with same problem: bnx2, cassini, pcnet32, r8169, rrunner, skge, sungem, tg3, -- --
I didn't read the start of this thread and may misunderstand the issue, but I comment anyway...: Documentation/DMA-API.txt says the following about (pci_)dma_sync_single and friends: "All the parameters must be the same as those passed into the single mapping API." http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentatio... So, it might be considered an API bug --- but it is a documented one. :-) -- Stefan Richter -=====-==-=- ---= =-=-= http://arcgraph.de/sr/ --
Disregard my comment. I just noticed that there is more qualified discussion of this in the thread "[PATCH] sky2: Fix WARNING: at lib/dma-debug.c:902 check_sync". -- Stefan Richter -=====-==-=- ---= =-=-= http://arcgraph.de/sr/ --
No - can be any device connected via the wifi router - wired or wireless. --
Anyway, if it can't be repeated with this Win7 box or even the router getting dhcp itself, then this router's interaction seems most suspected. Jarek P. --
