We have about 100 servers based on Intel S5000PSL-SATA motherboards. 
They have been running for anywhere between 1 and 10 months. For the 
past few months, after updating them all to the 2.6.20.15 kernel 
(because of a bug in the 2.6.18 kernel), we are seeing some strange NMI 
errors. For example:

Aug 29 09:02:10 master kernel: Uhhuh. NMI received for unknown reason 30.
Aug 29 09:02:10 master kernel: Do you have a strange power saving mode 
enabled?
Aug 29 09:02:10 master kernel: Dazed and confused, but trying to continue

Sometimes these errors cause a total system freeze. Most of the time the 
systems keep running.

We have determined these errors come most frequently on machines that 
have an Intel PCI-e Quad Port Gigabit Adapter. On machines that HAVE 
these cards (it doesn't matter what slot they are in), the NMI errors 
can occur as frequently as every 3-5 minutes. On machines that do NOT 
have these Quad Port Adapters, the NMI errors occur about once per month 
on average. (we have tried the "in-kernel" e1000 drivers, as well as 
Intel's latest - 7.6.5).

We have also determined (through a chance discovery) that running 
“scanpci” can 100 percent reliably reproduce the NMI error on any 
machine that has the Quad Port NICS. Our various motherboards have 
different Intel BIOS versions – some have Rev 70, others 74, 79 or 81. 
They all exhibit the same behavior regardless of BIOS version.

We have reproduced this problem with:

Mandriva 2008 RC2 (2.6.22 kernel)
Mandriva 2007 with custom 2.6.20.15 kernel
Mandriva 2007 with custom 2.6.19.8 kernel
Ubuntu “Feisty” with 2.6.20 kernel
Fedora Core 7 with 2.6.22 kernel

The problem does NOT occur with any distribution running a 2.6.18 kernel 
or lower. I.E., CentOS or SUSE 10 and also Mandriva 2007 with included 
2.6.17 kernel or custom-compiled 2.6.18 kernel.

We have been in contact with Intel. Their high level tech support people 
have basically said,

    “the errors we have logged so far are pointing to a kernel ...
[ message continues ]

product info (for others):


Please provide some basic info, like:

- how much RAM
- what CPUs (be precise: use 'cat /proc/cpuinfo')
- output of 'lspci -v'
- what kind(s) of SATA drives
- are you using 32-bit or 64-bit kernel(s)

Can you test kernels from kernel.org (i.e., not vendor kernels,
  no other [unkwown] patches applied to them)?

Does tracing 'scanpci' produce any helpful information?
# strace -o scanpci.trace scanpci


---
~Randy
Phaedrus says that Quality is about caring.
-

[ message continues ]

Answer #2:  if a kernel change was responsible for this problem,
the direct way to find that change is to clone the kernel 'git' tree
and then use git bisect to find the culprit.  If you are certain
that 2.6.18 is good and 2.6.19 is bad, then use those git tree tags
instead of the ones that are used in the example at:
  http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html

git wiki is here:  http://git.or.cz/
and git docs are here:  http://www.kernel.org/pub/software/scm/git/docs/

If you want to use this tool, say so and I think that we (the royal
"we") will try to work you thru it.

---
~Randy
Phaedrus says that Quality is about caring.
-

[ message continues ]

What would be useful is to know under what situations that board can

The first thing to find out is to find out in which kernel the behaviour
is introduced. It might also be worth disabling msi in case Intel screwed

See if 2.6.20.* with the 2.6.18 driver compiles and how that behaves.
Also see if pci=nomsi and/or pci=nommconf make a difference.

Alan
-

[ message continues ]

I'm also working with Andrew and Samson.  It seems that the cause of
the problem is CONFIG_PCIEAER, which was introduced after 2.6.18 and
defaults to y.

With CONFIG_PCIEAER=n, scanpci works fine with no errors.  This is the
workaround that they'll likely use for now.

With CONFIG_PCIEAER=y, scanpci always triggers the NMI error.  The
option aerdriver.forceload=1 has no effect.

The related dmesg output at boot is:

  Evaluate _OSC Set fails. Status = 0x0005
  Evaluate _OSC Set fails. Status = 0x0005
  aer_init: AER service init fails - Run ACPI _OSC fails
  aer: probe of 0000:00:02.0:pcie01 failed with error 2
  aer_init: AER service init fails - No ACPI _OSC support
  aer: probe of 0000:00:03.0:pcie01 failed with error 1
  Evaluate _OSC Set fails. Status = 0x0005
  Evaluate _OSC Set fails. Status = 0x0005
  aer_init: AER service init fails - Run ACPI _OSC fails
  aer: probe of 0000:00:04.0:pcie01 failed with error 2
  Evaluate _OSC Set fails. Status = 0x0005
  Evaluate _OSC Set fails. Status = 0x0005
  aer_init: AER service init fails - Run ACPI _OSC fails
  aer: probe of 0000:00:05.0:pcie01 failed with error 2
  Evaluate _OSC Set fails. Status = 0x0005
  Evaluate _OSC Set fails. Status = 0x0005
  aer_init: AER service init fails - Run ACPI _OSC fails
  aer: probe of 0000:00:06.0:pcie01 failed with error 2
  aer_init: AER service init fails - No ACPI _OSC support
  aer: probe of 0000:00:07.0:pcie01 failed with error 1

Full dmesg, lspci, and ACPI DSDT are available here:
  http://jim.sh/~jim/tmp/nmi/

-jim
-

[ message continues ]

The 'forceload' option only forces the driver to load even when the
ACPI hardware initialization routine fails.

It would be nice to be able to disable PCIEAER at boot time though.


---
~Randy
Phaedrus says that Quality is about caring.
-

[ message continues ]

So, looking for some closure here, what do we think is the "root cause"? 
Is it:

1)  a defect with Intel's S5000PSL motherboards that is exposed by an 
otherwise fine new (since 2.6.19) Linux kernel feature? (in which case 
we and others should probably press Intel to recognize they have a 
problem, seeing as they only "officially support" distributions running 
on 2.6.16 or below so maybe they don't even know about this issue).

2)  a problem with PCIEAER? And maybe "CONFIG_PCIEAER=y"  should NOT be 
the default setting? (in which case the kernel maybe needs fixing)

3)  just a bad interaction between a good motherboard and a good Linux 
feature that don't play well together? (in which case this is a kernel 
"feature" that anybody compiling a kernel to run on the Intel S5000PSL 
motherboard should know not to enable -- maybe a note is warranted so 
that when configuring the kernel, people with S5000PSL motherboards 

-

[ message continues ]

So, looking for some closure here, what do we think is the "root cause"? 
Is it:

1)  a defect with Intel's S5000PSL motherboards that is exposed by an 
otherwise fine new (since 2.6.19) Linux kernel feature? (in which case 
we and others should probably press Intel to recognize they have a 
problem, seeing as they only "officially support" distributions running 
on 2.6.16 or below so maybe they don't even know about this issue).

2)  a problem with PCIEAER? And maybe "CONFIG_PCIEAER=y"  should NOT be 
the default setting? (in which case the kernel maybe needs fixing)

3)  just a bad interaction between a good motherboard and a good Linux 
feature that don't play well together? (in which case this is a kernel 
"feature" that anybody compiling a kernel to run on the Intel S5000PSL 
motherboard should know not to enable -- maybe a note is warranted so 
that when configuring the kernel, people with S5000PSL motherboards 

-

[ message continues ]

This is a slightly edited repost of a note sent on Friday September 28, 
as we haven't heard back from anyone yet. (I know it was the weekend!) 
Sorry to post again but this issue caused great problems for us and I 
want to be sure we're choosing a decent solution.

Perhaps one of the people who so helpfully commented on this issue 
earlier last week can now give their opinion on the what should be 
concluded from our discovery that "CONFIG_PCIEAER=y" -- introduced in 
the 2.6.19 kernel and set as the default -- leads to NMI errors on the 
Intel S5000PSL motherboard.

I'm told Intel people were closely involved in the development of this 
PCIEAER feature -- so it seems even weirder that it causes problems for 
this Intel motherboard. But we have confirmed the problem with multiple 
Linux distributions.

We are hoping to get some insights into the real cause. Please see below 
Although running "scanpci" provoked the NMI errors 100 percent on 
demand, the NMI errors would also occur randomly every few weeks on a 
given system without doing anything special. I don't want anybody to 
think we are just trying to prevent a problem from occurring because we 
like running "scanpci".  "Scanpci" just turned out to be a reliable way 


So, looking for some closure here, what do you think is the "root 
cause"? Is it:

1)  a defect with Intel's S5000PSL motherboards that is not seen when 
running 2.6.18 and earlier kernels but that is exposed by this feature 
added in 2.6.19? In which case, shouldn't we work to get Intel to 
investigate?

2)  a problem with the PCIEAER feature? And maybe "CONFIG_PCIEAER=y"  
should NOT be the default setting?

3)  just a bad interaction between a good motherboard and a good Linux 
feature that don't play well together? (in which case isn't this a 
"feature" that anybody compiling a kernel to run on the Intel S5000PSL 
motherboard should know not to enable?/

And in general is it a bad idea to set "CONFIG_PCIEAER to "no"". Or is 
it something that ...
[ message continues ]