login
Search
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb
failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
drivers/net/igb/igb_main.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
--- linux-2.6.orig/drivers/net/igb/igb_main.c
+++ linux-2.6/drivers/net/igb/igb_main.c
@@ -4358,14 +4358,22 @@ static int igb_suspend(struct pci_dev *p
wr32(E1000_WUFC, 0);
}
+ /*
+ * Apparently it is not possible to reinitialise from D3 hot,
+ * only put the device into D3 if we really go for poweroff.
+ */
/* make sure adapter isn't asleep if manageability/wol is enabled */
if (wufc || adapter->en_mng_pt) {
- pci_enable_wake(pdev, PCI_D3hot, 1);
- pci_enable_wake(pdev, PCI_D3cold, 1);
+ if (system_state == SYSTEM_POWER_OFF) {
+ pci_enable_wake(pdev, PCI_D3hot, 1);
+ pci_enable_wake(pdev, PCI_D3cold, 1);
+ }
} else {
igb_shutdown_fiber_serdes_link_82575(hw);
- pci_enable_wake(pdev, PCI_D3hot, 0);
- pci_enable_wake(pdev, PCI_D3cold, 0);
+ if (system_state == SYSTEM_POWER_OFF) {
+ pci_enable_wake(pdev, PCI_D3hot, 0);
+ pci_enable_wake(pdev, PCI_D3cold, 0);
+ }
}
/* Release control of h/w to f/w. If f/w is AMT enabled, this
@@ -4374,7 +4382,8 @@ static int igb_suspend(struct pci_dev *p
pci_disable_device(pdev);
- pci_set_power_state(pdev, pci_choose_state(pdev, state));
+ if (system_state == SYSTEM_POWER_OFF)
+ pci_set_power_state(pdev, pci_choose_state(pdev, state));
return 0;
}
--
I see the point of the patch, but I know for a fact that ixgbe when enabled for MSI-X also doesn't work with kexec. so my questions are: are you going to change every driver? why can't this be fixed in core kernel code instead? Shouldn't pci_enable_device take it out of D3? Or maybe it should be taken out of D3 immediately if someone tries to ioremap any of the BARx registers? Jesse --
On Fri, Mar 6, 2009 at 11:18 PM, Jesse Brandeburg looks like second kernel can not detect the state any more. YH --
I know this has historically been a problem with the e1000 NICs. Placing the hardware in a state they can not get them out of on the reboot path. Last I heard (a couple of weeks ago?) we had code to bring devices out of a low power state that was working for the e1000 driver. YH can you look and see if you can find that code and if it works? <rant> Frankly I don't understand why anyone would want to power down a device when they are rebooting or shutting down a computer. That is a system state change. But it seems to be bleed over from the confusion that has been the power management code. </rant> If we can teach the kernel to handle this case with the proper enables and disables that would be great. Otherwise let's look at getting the responsibilities of the various methods sorted out so we can at least say with certainty what the various methods are supposed to do and not do. Eric --
On Sat, Mar 7, 2009 at 10:20 AM, Eric W. Biederman root cause could be some BIOS/acpi lying to Kernel about device state querying. for igb, it only has problem on one platform. but other one platform is ok. for forcedeth, it seems all platform has the problem, aka you can not put it in D3 in kexec path. YH --
Well, well. IMHO, the confusion is that ->shutdown() is used for too many _different_ things, because kexec is sufficiently different from power off, for example, to be handled by a separate driver callback. I'm totally unsure what power management has to do with it, though. Thanks, Rafael --
In fact pci_enable_device() calls pci_set_power_state(dev, PCI_D0) as the first thing. The question is why it doesn't work as expected. What kernel(s) have you tested? Thanks, Rafael --
tip/master also RHEL 5.3 with igb + kexec have the same problem. YH --
not sure... please check the version for forcedeth that you made.
commit 3cb5599a84c557c0dd9a19feb63a3788268cf249
Author: Rafael J. Wysocki <rjw@sisk.pl>
Date: Fri Sep 5 14:00:19 2008 -0700
forcedeth: fix kexec regression
Fix regression tracked as http://bugzilla.kernel.org/show_bug.cgi?id=11361
and caused by commit f735a2a1a4f2a0f5cd823ce323e82675990469e2 ("[netdrvr]
forcedeth: setup wake-on-lan before shutting down") that makes network
adapters integrated into the NVidia MCP55 chipsets fail to work in kexeced
kernels. The problem appears to be that if the adapter is put into D3_hot
during ->shutdown(), it cannot be brought back into D0 after kexec (ref.
http://marc.info/?l=linux-kernel&m=121900062814967&w=4). Therefore, only
put forcedeth into D3 during ->shutdown() if the system is to be powered
off.
Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
Tested-by: Yinghai Lu <yhlu.kernel@gmail.com>
Cc: Ayaz Abdulla <aabdulla@nvidia.com>
Acked-by: Jeff Garzik <jeff@garzik.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
diff --git a/drivers/net/forcedeth.c b/drivers/net/forcedeth.c
index 331b86b..0b6ecef 100644
--- a/drivers/net/forcedeth.c
+++ b/drivers/net/forcedeth.c
@@ -5975,10 +5975,12 @@ static void nv_shutdown(struct pci_dev *pdev)
if (netif_running(dev))
nv_close(dev);
- pci_enable_wake(pdev, PCI_D3hot, np->wolenabled);
- pci_enable_wake(pdev, PCI_D3cold, np->wolenabled);
pci_disable_device(pdev);
- pci_set_power_state(pdev, PCI_D3hot);
+ if (system_state == SYSTEM_POWER_OFF) {
+ if (pci_enable_wake(pdev, PCI_D3cold, np->wolenabled))
+ pci_enable_wake(pdev, PCI_D3hot, np->wolenabled);
+ pci_set_power_state(pdev, PCI_D3hot);
+ }
}
#else
#define nv_suspend NULL
--
Thanks, I remember now. This appears to be quirky hardware, doesn't it? Rafael --
all my systems with ck804 and mcp55 need this trick. for igb: one system need this trick, and other doesn't need this trick. could BIOS/ACPI difference or the nic option rom difference. but for kexec path, we really don't need put pci devices to D3. YH --
Agreed, but pci_set_power_state() may be called for too many different reasons to be a suitable place for a fix IMO. Thanks, Rafael --
In which case you need to rework igb_shutdown() rather than igb_suspend().
Something like the patch below, perhaps (totally untested).
Thanks,
Rafael
---
drivers/net/igb/igb_main.c | 24 +++++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
--- linux-2.6.orig/drivers/net/igb/igb_main.c
+++ linux-2.6/drivers/net/igb/igb_main.c
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
}
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
{
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
}
/* make sure adapter isn't asleep if manageability/wol is enabled */
- if (wufc || adapter->en_mng_pt) {
+ if ((wufc || adapter->en_mng_pt) && enable_wake) {
pci_enable_wake(pdev, PCI_D3hot, 1);
pci_enable_wake(pdev, PCI_D3cold, 1);
} else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
pci_disable_device(pdev);
- pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
return 0;
}
#ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+ int retval;
+
+ retval = __igb_shutdown(pdev, true);
+ if (!retval)
+ pci_set_power_state(pdev, PCI_D3hot);
+
+ return retval;
+}
+
static int igb_resume(struct pci_dev *pdev)
{
struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
static void igb_shutdown(struct pci_dev *pdev)
{
- igb_suspend(pdev, PMSG_SUSPEND);
+ if (system_state == SYSTEM_POWER_OFF) {
+ __igb_shutdown(pdev, true);
+ pci_set_power_state(pdev, PCI_D3hot);
+ } else {
+ __igb_shutdown(pdev, false);
+ }
}
#ifdef CONFIG_NET_POLL_CONTROLLER
--
Yes, because we've already decided we'd put the device into D3_hot by calling
pci_enable_wake(pdev, PCI_D3hot, 1) in __igb_shutdown().
In fact we should first determine the target state and _then_ call
pci_enable_wake() and pci_set_power_state() with that as the argument, but this
is a separate issue. For now, IMO, it's better to use D3_hot in both places
directly.
On a slightly related note, the sequence
pci_enable_wake(pdev, PCI_D3hot, 1);
pci_enable_wake(pdev, PCI_D3cold, 1);
is (slightly) incorrect by itself, because ACPI doesn't distinguish bewteen
D3_hot and D3_cold, so this just causes the same platform code to run twice,
which may or may not work (it theory it should always work, but still).
This is yet another problem, though.
Thanks,
Rafael
--
OK, I will, but it's going to take some time. Thanks, Rafael --
it works, David, can you picked it up
---
From: Rafael J. Wysocki <rjw@sisk.pl>
[PATCH] igb: fix kexec with igb -v2
Impact: could probe igb
Found one system with 82575EB, in the kernel that is kexeced, probe igb
failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
v2: Rafael J. Wysocki seperate igb_shutdown and igb_suspend code
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
drivers/net/igb/igb_main.c | 24 +++++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
--- linux-2.6.orig/drivers/net/igb/igb_main.c
+++ linux-2.6/drivers/net/igb/igb_main.c
@@ -4299,7 +4299,7 @@ int igb_set_spd_dplx(struct igb_adapter
}
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool enable_wake)
{
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4359,7 +4359,7 @@ static int igb_suspend(struct pci_dev *p
}
/* make sure adapter isn't asleep if manageability/wol is enabled */
- if (wufc || adapter->en_mng_pt) {
+ if ((wufc || adapter->en_mng_pt) && enable_wake) {
pci_enable_wake(pdev, PCI_D3hot, 1);
pci_enable_wake(pdev, PCI_D3cold, 1);
} else {
@@ -4374,12 +4374,21 @@ static int igb_suspend(struct pci_dev *p
pci_disable_device(pdev);
- pci_set_power_state(pdev, pci_choose_state(pdev, state));
-
return 0;
}
#ifdef CONFIG_PM
+static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+ int retval;
+
+ retval = __igb_shutdown(pdev, true);
+ if (!retval)
+ pci_set_power_state(pdev, PCI_D3hot);
+
+ return retval;
+}
+
static int igb_resume(struct pci_dev *pdev)
{
struct net_device *netdev = pci_get_drvdata(pdev);
@@ -4434,7 +4443,12 @@ static int igb_resume(struct pci_dev *pd
...Still, Yinghai, can you please also test the patch below? It fixes all
shortcomings in the driver's suspend and shutdown methods I was talking about
in one of the previous messages. If it works, IMO it will be a preferable fix
(in particular, it would be good to check if WoL still works with it,
but I don't have the hardware).
Thanks,
Rafael
---
From: Rafael J. Wysocki <rjw@sisk.pl>
Subject: net/igb: Fix kexec with igb (rev. 3)
Impact: Fix
Yinghai Lu found one system with 82575EB where, in the kernel that is
kexeced, probe igb failed with -2, the reason being that the adapter
could not be brought back from D3 by the kexec kernel, most probably
due to quirky hardware (it looks like the same behavior happened on
forcedeth).
Prevent igb from putting the adapter into D3 during shutdown except
when we going to power off the system. For this purpose, seperate
igb_shutdown() from igb_suspend() and use the appropriate PCI PM
callbacks in both of them.
Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
Reported-by: Yinghai Lu <yinghai@kernel.org>
---
drivers/net/igb/igb_main.c | 42 ++++++++++++++++++++++++++++++------------
1 file changed, 30 insertions(+), 12 deletions(-)
Index: linux-2.6/drivers/net/igb/igb_main.c
===================================================================
--- linux-2.6.orig/drivers/net/igb/igb_main.c
+++ linux-2.6/drivers/net/igb/igb_main.c
@@ -4277,7 +4277,7 @@ int igb_set_spd_dplx(struct igb_adapter
}
-static int igb_suspend(struct pci_dev *pdev, pm_message_t state)
+static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
{
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
@@ -4336,15 +4336,9 @@ static int igb_suspend(struct pci_dev *p
wr32(E1000_WUFC, 0);
}
- /* make sure adapter isn't asleep if manageability/wol is enabled */
- if (wufc || adapter->en_mng_pt) {
- pci_enable_wake(pdev, PCI_D3hot, 1);
- pci_enable_wake(pdev, PCI_D3cold, 1);
- } ...Great, thanks for testing. Jeff, Jesse, is the patch fine with you? Rafael --
Let me talk with Jesse on Monday about it. I know that Jesse's concern's were that there were more than one driver afflicted with the same or similar issue and that it made more sense to fix this in the core for all drivers. For instance ixgbe, IIRC, but because Yinghai did not have any other hardware, it was unclear if this was an issue on other drivers. After discussing this with Jesse, one of us will respond. I will apply this patch to my tree anyway and if we are alright with the patch, I will push it out with my other patches to Dave. -- Cheers, Jeff --
I think it was, at least theoretically. We should generally avoid powering off things on shutdown when it's not really necessary. Also, the use of pci_enable_wake() in these drivers is not really correct. And in fact this is why I'm asking, because I have analogouns patches for some other drivers in the works too. If you are fine with the approach, I'll post Thanks! Best, Rafael --
After talking with Jesse, we are fine with it if our testers buy off on it. I have the patch in my tree and so Emil will run a barrage of regression/stress tests on the patch overnight. If everything looks good, I will send it out with the other igb patches I have. Thanks. -- Cheers, Jeff --
The patch checks out. I tested suspend/resume (including WOL) and kexec. There is only a small issue with warning on compile when CONFIG_PM is disabled: diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c index 13fe162..0a4f8f4 100644 --- a/drivers/net/igb/igb_main.c +++ b/drivers/net/igb/igb_main.c @@ -135,8 +135,8 @@ static inline int igb_set_vf_rlpml(struct igb_adapter *, int, int); static int igb_set_vf_mac(struct igb_adapter *adapter, int, unsigned char *); static void igb_restore_vf_multicasts(struct igb_adapter *adapter); -static int igb_suspend(struct pci_dev *, pm_message_t); #ifdef CONFIG_PM +static int igb_suspend(struct pci_dev *, pm_message_t); static int igb_resume(struct pci_dev *); #endif static void igb_shutdown(struct pci_dev *); Thanks, Emil
On Tue, Mar 31, 2009 at 12:14 PM, Tantilov, Emil S Rafael - Based on Emil's testing, I will modify the patch with Emil's suggestion and push to Dave, ok? -- Cheers, Jeff --
Impact: second kernel by kexec will have some pci devices working
Found one system with 82575EB, in the kernel that is kexeced, probe igb
failed with -2.
it looks like the same behavior happened on forcedeth.
try to check system_state to make sure if put it on D3
Jesse Brandeburg said that we should do that check in core code instead of
every device driver.
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
---
drivers/pci/pci.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
Index: linux-2.6/drivers/pci/pci.c
===================================================================
--- linux-2.6.orig/drivers/pci/pci.c
+++ linux-2.6/drivers/pci/pci.c
@@ -593,6 +593,14 @@ int pci_set_power_state(struct pci_dev *
if (state == PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))
return 0;
+ /*
+ * Apparently it is not possible to reinitialise from D3 hot,
+ * only put the device into D3 if we really go for poweroff.
+ */
+ if (system_state != SYSTEM_POWER_OFF &&
+ (state == PCI_D3hot || state == PCI_D3cold))
+ return 0;
+
error = pci_raw_set_power_state(dev, state, true);
if (state > PCI_D0 && platform_pci_power_manageable(dev)) {
@@ -1124,6 +1132,15 @@ int pci_enable_wake(struct pci_dev *dev,
int error = 0;
bool pme_done = false;
+ /*
+ * Apparently it is not possible to reinitialise from D3 hot,
+ * only put the device into D3 if we really go for poweroff.
+ * we only need to enable wake when we are going to power off
+ */
+ if (enable && system_state != SYSTEM_POWER_OFF &&
+ (state == PCI_D3hot || state == PCI_D3cold))
+ return 0;
+
if (enable && !device_may_wakeup(&dev->dev))
return -EINVAL;
--
This is not enough, because the PM code doesn't change system_state and This breaks suspend/hibernation, doesn't it? Surely, we want to put devices into D3 when going for suspend, for example. I don't like this at all, sorry. I thought we were supposed to avoid using system_state in such a way, apart from the other things. Thanks, Rafael --
So Yinghai has now written a driver fix and a generic code fix as well and both are being rejected pointing to the other side? This really sucks for users (who'd be happy with any of the patches) and the disagreement needs to be resolved ASAP. Ingo --
I hope you read my other comment (that you deleted from the reply)? If we're going to fix this at the core level along the Yinghai's patch lines, we'll have to introduce a few additional values for system_state and make suspend and hibernation code use them. Do you think it's worth the effort, given that only a couple of drivers have been found to be affected so far? Rafael --
And, quite frankly, I'm not sure if users will be happy with the $subject patch, because it _really_ breaks things (well, the kexec users who don't use suspend might be, but surely suspend users who don't use kexec won't). Thanks, Rafael --
Please note that i havent reviewed the patches and i did not take any sides in the discussion - i just flagged the maintainer ping-pong. As long as we pick one of the patches (or a third one) within a bound amount of time we should be fine :) Ingo --
I'll defer to Rafael here; he's been working the most in this area. The changelog wasn't very complete for the original patch, but it sounds like in the kexec case the newly booted kernel will get an igb device in D3 which it can't handle? That really does sound like a driver bug, not something we should mess with in the core. -- Jesse Barnes, Intel Open Source Technology Center --
I have already posted an alternative patch for igb that's been reported to fix the problem. Thanks, Rafael --
