On Wed, Nov 21, 2007 at 09:49:03AM +0000, Jan Beulich wrote:
Thanks. I think I understand now.
A kernel without the patch always forced creation of a prefetch
window for expansion ROMs which was incorrect on systems where
insufficient memory resources are available for both non-prefetch
and prefetch windows. On the systems I was dealing with, the
BIOS assumes (correctly, I believe) that expansion ROM memory
resource needs will be satisfied from the non-prefetch window.
On those same systems, the BIOS (via _CRS) does not provide
enough additional prefetch or non-prefetch memory for even a
minimum size prefetch window. These systems limit the memory
resources provided for already installed PCI devices to the
bare minimum because of the large number of PCI devices in a
multi-node configuration that are competing for the available
resources.
A kernel with the patch forces expansion ROM resources to be
allocated from a non-prefetch window which, based on your
input, is apparently problematic on at least one system due to
the BIOS (via e820 hole?) not providing enough non-prefetch memory
to construct a large enough non-prefetch window to accommodate
the underlying expansion ROMs.
I think it would still be useful to know what system/PCI adapter
combination you are seeing the problem with so that I can try
to put together a setup that will reproduce the problem here.
Alternatively, it would good if you wouldn't mind providing
more information, testing possible fixes, etc. As a start,
could you send me the following taken with and without the
patch?
- /proc/iomem
- /proc/ioports
- `dmesg` output
- `lspci -vt` output
- `lspci -vvv` output
It is quite new. It was added with a separate patch that I
submitted at the same time as the "Avoid creating P2P prefetch
window for expansion ROMs" patch.
Prior to constraining PCI memory resource allocations to what
_CRS returned, the BIOS unassigned mem resources (including
that required for expansion ROMs) were being allocated from
often incorrect ranges in the e820 hole. After adding the
_CRS constraint we encountered an expansion ROM allocation
problem that motivated the "Avoid creating P2P prefetch window
for expansion ROMs" change. If I recall correctly, we ran
into the expansion ROM allocation problem when using a PCIe
adapter which has an on-board P2P bridge (PCIe-to-PCI/PCI-X)
above a device with an expansion ROM.
Interesting. I actually anticipated that this sort of thing
might happen on some systems with BIOSes that export a _CRS
that does not correctly return all of the resources available
under the associated root bridge. This is one reason why the
use _CRS feature is "off" by default. You might want to make
sure that the BIOS on that system is up to date and, if so,
let the h/w vendor know about this.
Greg,
Without a closer look at the code and the information I
requested above I am not 100% certain that my expansion ROM
allocation fix is incomplete but it seems pretty likely that
it is. If you feel that the possible regression that Jan
reported is more important than the issue I was trying to solve
feel free to remove the change until I can come up with
something better.
Thanks,
Gary
--
Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503 IBM T/L: 775-4503
garyhade@us.ibm.comhttp://www.ibm.com/linux/ltc
-
