Home » Mailing list archives » linux-kernel » 2008 » January » 18

Re: [PATCH] x86_64: checking aperture report for node instead of?cpu v2

Previous thread: Re: Fwd: Fwd: laptop / computer hardlocks during execution of 32bit applications(binaries) on 64bit system (Gentoo) by Ed Tomlinson on Thursday, January 10, 2008 - 10:22 pm. (2 messages)

Next thread: Splice(): exports for module programmers by Chris Smowton on Thursday, January 10, 2008 - 10:43 pm. (2 messages)
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Andrew Morton <akpm@...>, Ingo Molnar <mingo@...>, Thomas Gleixner <tglx@...>, Christoph Lameter <clameter@...>, <ebiederm@...>
Cc: LKML <linux-kernel@...>
Subject: [PATCH] x86-64: disable the GART early
Date: Thursday, January 10, 2008 - 11:14 pm

this one is against to linus tree.

in case some one need to use kexec to new kernel from old kernel with gart_shutdown.


YH


[PATCH] x86-64: disable the GART early


For K8 system: 4G RAM with memory hole remapping enabled, or more than 4G RAM

installed.


when try to use kexec second kernel, and the first doesn't include gart_shutdown.

the second kernel could have different aper position than the first kernel. and second

kernel could use that hole as RAM that is still used by GART set by the first kernel.

esp. when try to kexec 2.6.24 with sparse mem enable from previous kernel (from RHEL 5

or SLES 10). the new kernel will use aper by GART (set by first kernel) for vmemmap.

and after new kernel setting one new GART. the position will be real RAM. the _mapcount

set is lost.


Bad page state in process 'swapper'

page:ffffe2000e600020 flags:0x0000000000000000 mapping:0000000000000000 mapcount:1 count:0

Trying to fix it up, but a reboot is needed

Backtrace:

Pid: 0, comm: swapper Not tainted 2.6.24-rc7-smp-gcdf71a10-dirty #13


Call Trace:

 [<ffffffff8026401f>] bad_page+0x63/0x8d

 [<ffffffff80264169>] __free_pages_ok+0x7c/0x2a5

 [<ffffffff80ba75d1>] free_all_bootmem_core+0xd0/0x198

 [<ffffffff80ba3a42>] numa_free_all_bootmem+0x3b/0x76

 [<ffffffff80ba3461>] mem_init+0x3b/0x152

 [<ffffffff80b959d3>] start_kernel+0x236/0x2c2

 [<ffffffff80b9511a>] _sinittext+0x11a/0x121


and

 [ffffe2000e600000-ffffe2000e7fffff] PMD ->ffff81001c200000 on node 0

phys addr is : 0x1c200000


RHEL 5.1 kernel -53 said:

PCI-DMA: aperture base @ 1c000000 size 65536 KB


new kernel said:

Mapping aperture over 65536 KB of RAM @ 3c000000


So try to disable that GART as early as possible.


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


Index: linux-2.6/arch/x86/kernel/aperture_64.c

===================================================================

--- linux-2.6.orig/arch/x86/kernel/aperture_64.c

+++ linux-2.6/arch/x86/kernel/aperture_64.c

@@ -204,6 ...
[ message continues ]
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Ingo Molnar <mingo@...>, Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>
Cc: LKML <linux-kernel@...>
Subject: [PATCH] x86-64: disable the GART early v2
Date: Friday, January 11, 2008 - 3:54 am

please check the one that can be applied to x86.git -mm


YH


[PATCH] x86-64: disable the GART early v2


For K8 system: 4G RAM with memory hole remapping enabled, or more than 4G RAM

installed.


when try to use kexec second kernel, and the first doesn't include gart_shutdown.

the second kernel could have different aper position than the first kernel. and second

kernel could use that hole as RAM that is still used by GART set by the first kernel.

esp. when try to kexec 2.6.24 with sparse mem enable from previous kernel (from RHEL 5

or SLES 10). the new kernel will use aper by GART (set by first kernel) for vmemmap.

and after new kernel setting one new GART. the position will be real RAM. the _mapcount

set is lost.


Bad page state in process 'swapper'

page:ffffe2000e600020 flags:0x0000000000000000 mapping:0000000000000000 mapcount:1 count:0

Trying to fix it up, but a reboot is needed

Backtrace:

Pid: 0, comm: swapper Not tainted 2.6.24-rc7-smp-gcdf71a10-dirty #13


Call Trace:

 [<ffffffff8026401f>] bad_page+0x63/0x8d

 [<ffffffff80264169>] __free_pages_ok+0x7c/0x2a5

 [<ffffffff80ba75d1>] free_all_bootmem_core+0xd0/0x198

 [<ffffffff80ba3a42>] numa_free_all_bootmem+0x3b/0x76

 [<ffffffff80ba3461>] mem_init+0x3b/0x152

 [<ffffffff80b959d3>] start_kernel+0x236/0x2c2

 [<ffffffff80b9511a>] _sinittext+0x11a/0x121


and

 [ffffe2000e600000-ffffe2000e7fffff] PMD ->ffff81001c200000 on node 0

phys addr is : 0x1c200000


RHEL 5.1 kernel -53 said:

PCI-DMA: aperture base @ 1c000000 size 65536 KB


new kernel said:

Mapping aperture over 65536 KB of RAM @ 3c000000


So try to disable that GART as early as possible.


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c

index 52d2bea..6e7af45 100644

--- a/arch/x86/kernel/aperture_64.c

+++ b/arch/x86/kernel/aperture_64.c

@@ -218,6 +218,73 @@ static __u32 __init search_agp_bridge(u32 *order, int *valid_agp)

 	return 0;

...
[ message continues ]
[view original message]
From: Ingo Molnar <mingo@...>
To: Yinghai Lu <Yinghai.Lu@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86-64: disable the GART early v2
Date: Friday, January 11, 2008 - 4:14 am

hm, i'm wondering, instead of modifying the GART, why dont we simply

_detect_ whatever GART settings we have inherited, and propagate that

into our e820 maps? I.e. if there's inconsistency, then punch that out

from the memory maps and just dont use that memory.


that way it would not matter whether the GART settings came from a [old

or crashing] Linux kernel that has not called gart_iommu_shutdown(), or

whether it's a BIOS that has set up an aperture hole inconsistent with

the memory map it passed. (or the memory map we _think_ i tried to pass

us)


it would also be more robust to only read and do a memory map quirk

based on that, than actively trying to change the GART so early in the

bootup. Later on we have to re-enable the GART _anyway_ and have to

punch a hole for it.


and as a bonus, we would have shored up our defenses against crappy

BIOSes as well.


	Ingo

--
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Ingo Molnar <mingo@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86-64: disable the GART early v2
Date: Saturday, January 12, 2008 - 6:07 am

please check the one against x86.git

it will use fix e820 for gart.


YH


[PATCH] x86-64: fix e820 for GART or disable the GART early


For K8 system: 4G RAM with memory hole remapping enabled, or more than 4G RAM

installed.


when try to use kexec second kernel, and the first doesn't include gart_shutdown.

the second kernel could have different aper position than the first kernel. and second

kernel could use that hole as RAM that is still used by GART set by the first kernel.

esp. when try to kexec 2.6.24 with sparse mem enable from previous kernel (from RHEL 5

or SLES 10). the new kernel will use aper by GART (set by first kernel) for vmemmap.

and after new kernel setting one new GART. the position will be real RAM. the _mapcount

set is lost.


Bad page state in process 'swapper'

page:ffffe2000e600020 flags:0x0000000000000000 mapping:0000000000000000 mapcount:1 count:0

Trying to fix it up, but a reboot is needed

Backtrace:

Pid: 0, comm: swapper Not tainted 2.6.24-rc7-smp-gcdf71a10-dirty #13


Call Trace:

 [<ffffffff8026401f>] bad_page+0x63/0x8d

 [<ffffffff80264169>] __free_pages_ok+0x7c/0x2a5

 [<ffffffff80ba75d1>] free_all_bootmem_core+0xd0/0x198

 [<ffffffff80ba3a42>] numa_free_all_bootmem+0x3b/0x76

 [<ffffffff80ba3461>] mem_init+0x3b/0x152

 [<ffffffff80b959d3>] start_kernel+0x236/0x2c2

 [<ffffffff80b9511a>] _sinittext+0x11a/0x121


and

 [ffffe2000e600000-ffffe2000e7fffff] PMD ->ffff81001c200000 on node 0

phys addr is : 0x1c200000


RHEL 5.1 kernel -53 said:

PCI-DMA: aperture base @ 1c000000 size 65536 KB


new kernel said:

Mapping aperture over 65536 KB of RAM @ 3c000000


So could try to disable that GART if possible.


add e820 modification for gart inconsistent setting.


gart_fix_e820=off could be used to disable e820 fix.


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


Index: linux-2.6/arch/x86/kernel/aperture_64.c

===================================================================

--- linux-2.6.orig/arch/x86...
[ message continues ]
[view original message]
From: Ingo Molnar <mingo@...>
To: Yinghai Lu <Yinghai.Lu@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86-64: disable the GART early v2
Date: Monday, January 14, 2008 - 4:35 am

thanks, applied.


Sidenote - there were a few checkpatch failures:


 ERROR: use tabs not spaces

 #121: FILE: arch/x86/kernel/aperture_64.c:233:

 +        return 0;$


 ERROR: use tabs not spaces

 #173: FILE: arch/x86/kernel/aperture_64.c:285:

 +                fix = 1;$


i fixed them up. (please run "scripts/checkpatch.pl your.patch" in the

future.)


	Ingo

--
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Ingo Molnar <mingo@...>, Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Thomas Gleixner <tglx@...>
Cc: Christoph Lameter <clameter@...>, <ebiederm@...>, LKML <linux-kernel@...>
Subject: [PATCH] x86_64: checking aperture report for node instead of cpu
Date: Monday, January 14, 2008 - 9:39 pm

this one is against x86.git


[PATCH] x86_64: checking aperture report for node instead of cpu


currently when gart iommu is enabled by BIOS or previous we got


"

Checking aperture...

CPU 0: aperture @4000000 size 64MB

CPU 1: aperture @4000000 size 64MB

"

we should use use Node instead.


we will get

"

Checking aperture...

Node 0: aperture @4000000 size 64MB

Node 1: aperture @4000000 size 64MB

"


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


Index: linux-2.6/arch/x86/kernel/aperture_64.c

===================================================================

--- linux-2.6.orig/arch/x86/kernel/aperture_64.c

+++ linux-2.6/arch/x86/kernel/aperture_64.c

@@ -312,6 +312,7 @@ void __init gart_iommu_hole_init(void)

 	u32 aper_size, aper_alloc = 0, aper_order = 0, last_aper_order = 0;

 	u64 aper_base, last_aper_base = 0;

 	int fix, num, valid_agp = 0;

+	int node;


 	if (gart_iommu_aperture_disabled || !fix_aperture ||

 	    !early_pci_allowed())

@@ -320,6 +321,7 @@ void __init gart_iommu_hole_init(void)

 	printk(KERN_INFO  "Checking aperture...\n");


 	fix = 0;

+	node = 0;

 	for (num = 24; num < 32; num++) {

 		if (!early_is_k8_nb(read_pci_config(0, num, 3, 0x00)))

 			continue;

@@ -332,8 +334,8 @@ void __init gart_iommu_hole_init(void)

 		aper_base = read_pci_config(0, num, 3, 0x94) & 0x7fff;

 		aper_base <<= 25;


-		printk(KERN_INFO "CPU %d: aperture @ %Lx size %u MB\n",

-				num-24, aper_base, aper_size>>20);

+		printk(KERN_INFO "Node %d: aperture @ %Lx size %u MB\n",

+				node++, aper_base, aper_size >> 20);


 		if (!aperture_valid(aper_base, aper_size)) {

 			fix = 1;

--
[view original message]
From: Ingo Molnar <mingo@...>
To: Yinghai Lu <Yinghai.Lu@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Thomas Gleixner <tglx@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86_64: checking aperture report for node instead of cpu
Date: Tuesday, January 15, 2008 - 9:22 am

please dont put side-effects into printks. (put the node++ iteration out

into a separate line) Your patch looks fine otherwise.


	Ingo

--
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Ingo Molnar <mingo@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Thomas Gleixner <tglx@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, LKML <linux-kernel@...>
Subject: [PATCH] x86_64: checking aperture report for node instead
Date: Tuesday, January 15, 2008 - 2:31 pm

[PATCH] x86_64: checking aperture report for node instead of cpu v2


currently when gart iommu is enabled by BIOS or previous we got


"

Checking aperture...

CPU 0: aperture @4000000 size 64MB

CPU 1: aperture @4000000 size 64MB

"

we should use use Node instead.


we will get

"

Checking aperture...

Node 0: aperture @4000000 size 64MB

Node 1: aperture @4000000 size 64MB

"


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


Index: linux-2.6/arch/x86/kernel/aperture_64.c

===================================================================

--- linux-2.6.orig/arch/x86/kernel/aperture_64.c

+++ linux-2.6/arch/x86/kernel/aperture_64.c

@@ -312,6 +312,7 @@ void __init gart_iommu_hole_init(void)

 	u32 aper_size, aper_alloc = 0, aper_order = 0, last_aper_order = 0;

 	u64 aper_base, last_aper_base = 0;

 	int fix, num, valid_agp = 0;

+	int node;


 	if (gart_iommu_aperture_disabled || !fix_aperture ||

 	    !early_pci_allowed())

@@ -320,6 +321,7 @@ void __init gart_iommu_hole_init(void)

 	printk(KERN_INFO  "Checking aperture...\n");


 	fix = 0;

+	node = 0;

 	for (num = 24; num < 32; num++) {

 		if (!early_is_k8_nb(read_pci_config(0, num, 3, 0x00)))

 			continue;

@@ -332,8 +334,9 @@ void __init gart_iommu_hole_init(void)

 		aper_base = read_pci_config(0, num, 3, 0x94) & 0x7fff;

 		aper_base <<= 25;


-		printk(KERN_INFO "CPU %d: aperture @ %Lx size %u MB\n",

-				num-24, aper_base, aper_size>>20);

+		printk(KERN_INFO "Node %d: aperture @ %Lx size %u MB\n",

+				node, aper_base, aper_size >> 20);

+		node++;


 		if (!aperture_valid(aper_base, aper_size)) {

 			fix = 1;

--
[view original message]
From: Ingo Molnar <mingo@...>
To: Yinghai Lu <Yinghai.Lu@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Thomas Gleixner <tglx@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86_64: checking aperture report for node instead of?cpu v2
Date: Friday, January 18, 2008 - 9:02 am

thanks, applied.


	Ingo

--
[view original message]
From: Yinghai Lu <Yinghai.Lu@...>
To: Ingo Molnar <mingo@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>, LKML <linux-kernel@...>
Subject: [PATCH] x86-64: fix e820 for GART or disable the GART early
Date: Friday, January 11, 2008 - 5:38 pm

[PATCH] x86-64: fix e820 for GART or disable the GART early


For K8 system: 4G RAM with memory hole remapping enabled, or more than 4G RAM

installed.


when try to use kexec second kernel, and the first doesn't include gart_shutdown.

the second kernel could have different aper position than the first kernel. and second

kernel could use that hole as RAM that is still used by GART set by the first kernel.

esp. when try to kexec 2.6.24 with sparse mem enable from previous kernel (from RHEL 5

or SLES 10). the new kernel will use aper by GART (set by first kernel) for vmemmap.

and after new kernel setting one new GART. the position will be real RAM. the _mapcount

set is lost.


Bad page state in process 'swapper'

page:ffffe2000e600020 flags:0x0000000000000000 mapping:0000000000000000 mapcount:1 count:0

Trying to fix it up, but a reboot is needed

Backtrace:

Pid: 0, comm: swapper Not tainted 2.6.24-rc7-smp-gcdf71a10-dirty #13


Call Trace:

 [<ffffffff8026401f>] bad_page+0x63/0x8d

 [<ffffffff80264169>] __free_pages_ok+0x7c/0x2a5

 [<ffffffff80ba75d1>] free_all_bootmem_core+0xd0/0x198

 [<ffffffff80ba3a42>] numa_free_all_bootmem+0x3b/0x76

 [<ffffffff80ba3461>] mem_init+0x3b/0x152

 [<ffffffff80b959d3>] start_kernel+0x236/0x2c2

 [<ffffffff80b9511a>] _sinittext+0x11a/0x121


and

 [ffffe2000e600000-ffffe2000e7fffff] PMD ->ffff81001c200000 on node 0

phys addr is : 0x1c200000


RHEL 5.1 kernel -53 said:

PCI-DMA: aperture base @ 1c000000 size 65536 KB


new kernel said:

Mapping aperture over 65536 KB of RAM @ 3c000000


So could try to disable that GART if possible.


add e820 modification for gart inconsistent setting.


gart_fix_e820=off could be used to disable e820 fix.


Signed-off-by: Yinghai Lu <yinghai.lu@sun.com>


 Documentation/kernel-parameters.txt |    3 +

 arch/x86/kernel/aperture_64.c       |   86 ++++++++++++++++++++++++++++++++++++

 arch/x86/kernel/e820_64.c           |   12 +++++

 arch/x86/kernel/setup_64.c          |    ...
[ message continues ]
[view original message]
From: Yinghai Lu <yhlu.kernel@...>
To: Ingo Molnar <mingo@...>
Cc: Andi Kleen <ak@...>, Andrew Morton <akpm@...>, Christoph Lameter <clameter@...>, <ebiederm@...>, Thomas Gleixner <tglx@...>, LKML <linux-kernel@...>
Subject: Re: [PATCH] x86-64: disable the GART early v2
Date: Friday, January 11, 2008 - 4:34 am

good, i will update the patch to check that hole with e820 map.


that also make code short so don't need to check if agp is there or not.


YH

--
Previous thread: Re: Fwd: Fwd: laptop / computer hardlocks during execution of 32bit applications(binaries) on 64bit system (Gentoo) by Ed Tomlinson on Thursday, January 10, 2008 - 10:22 pm. (2 messages)

Next thread: Splice(): exports for module programmers by Chris Smowton on Thursday, January 10, 2008 - 10:43 pm. (2 messages)