Home » Mailing list archives » linux-kernel » 2008 » October » 2

Re: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions

Previous thread: Tux3 Report: Extolling the Extensive Virtues of Extents by Daniel Phillips on Wednesday, October 1, 2008 - 4:35 am. (1 message)

Next thread: [PATCH] thermal: Make it simpler to use the thermal layer inside the kernel by Matthew Garrett on Wednesday, October 1, 2008 - 5:04 am. (9 messages)
[view original message]
From: Dean Nelson
Subject: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions
Date: Wednesday, October 1, 2008 - 4:44 am

Provide a means for UV interrupt MMRs to be setup with the message to be sent
when an MSI is raised.

Signed-off-by: Dean Nelson <dcn@sgi.com>

---

This functionality is needed by drivers/misc/sgi-xp. And a patch will be
submitted shortly.

 arch/x86/kernel/Makefile    |    2 
 arch/x86/kernel/io_apic.c   |   95 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/uv_irq.c    |   50 +++++++++++++++++++++
 include/asm-x86/uv/uv_irq.h |   34 ++++++++++++++
 kernel/irq/chip.c           |    1 
 5 files changed, 181 insertions(+), 1 deletion(-)

Index: linux/arch/x86/kernel/io_apic.c
===================================================================
--- linux.orig/arch/x86/kernel/io_apic.c	2008-09-30 09:07:42.000000000 -0500
+++ linux/arch/x86/kernel/io_apic.c	2008-09-30 12:54:12.000000000 -0500
@@ -58,6 +58,8 @@
 #include <asm/setup.h>
 #include <asm/irq_remapping.h>
 #include <asm/hpet.h>
+#include <asm/uv/uv_hub.h>
+#include <asm/uv/uv_irq.h>
 
 #include <mach_ipi.h>
 #include <mach_apic.h>
@@ -3694,6 +3696,99 @@ int arch_setup_ht_irq(unsigned int irq, 
 }
 #endif /* CONFIG_HT_IRQ */
 
+#ifdef CONFIG_X86_64
+static void noop(unsigned int irq)
+{
+}
+
+static unsigned int noop_ret(unsigned int irq)
+{
+	return 0;
+}
+
+static void ack_apic(unsigned int irq)
+{
+	ack_APIC_irq();
+}
+
+static struct irq_chip uv_irq_chip = {
+	.name		= "UV_MSI",
+	.startup	= noop_ret,
+	.shutdown	= noop,
+	.enable		= noop,
+	.disable	= noop,
+	.ack		= noop,
+	.mask		= noop,
+	.unmask		= noop,
+	.eoi		= ack_apic,
+	.end		= noop,
+};
+
+/*
+ * Re-target the irq to the specified CPU and enable the specified MMR located
+ * on the specified blade to allow the sending of MSIs to the specified CPU.
+ */
+int arch_enable_uv_irq(char *irq_name, unsigned int irq, int cpu, int mmr_blade,
+		       unsigned long mmr_offset)
+{
+	const cpumask_t *eligible_cpu = get_cpu_mask(cpu);
+	struct irq_cfg *cfg;
+	int mmr_pnode;
+	unsigned long mmr_value;
+	struct ...
[ message continues ]
[view original message]
From: Ingo Molnar
Subject: Re: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions
Date: Thursday, October 2, 2008 - 1:42 am

unrelated change that belongs into a separate patch.

patch looks quite clean otherwise.

	Ingo
--


[ message continues ]
[view original message]
From: Dean Nelson
Subject: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions v.2
Date: Thursday, October 2, 2008 - 4:45 am

Provide a means for UV interrupt MMRs to be setup with the message to be sent
when an MSI is raised.

Signed-off-by: Dean Nelson <dcn@sgi.com>

---


Good question. Sometimes the obvious gets overlooked in the midst of

Agreed. The change is needed by drivers/misc/sgi-xp once it has been
modified to call uv_setup_irq() and uv_teardown_irq(). I'll move this
change into that patch (or should it be in a patch all by itself?).

The patch to drivers/misc/sgi-xp to call the functions introduced by
this patch, will be submitted shortly.

Thanks,
Dean

 arch/x86/kernel/Makefile    |    2 -
 arch/x86/kernel/io_apic.c   |   68 +++++++++++++++++++++++++++++++++++
 arch/x86/kernel/uv_irq.c    |   77 ++++++++++++++++++++++++++++++++++++++++
 include/asm-x86/uv/uv_irq.h |   36 ++++++++++++++++++
 4 files changed, 182 insertions(+), 1 deletion(-)

Index: linux/arch/x86/kernel/io_apic.c
===================================================================
--- linux.orig/arch/x86/kernel/io_apic.c	2008-10-02 06:12:54.000000000 -0500
+++ linux/arch/x86/kernel/io_apic.c	2008-10-02 06:15:48.000000000 -0500
@@ -58,6 +58,8 @@
 #include <asm/setup.h>
 #include <asm/irq_remapping.h>
 #include <asm/hpet.h>
+#include <asm/uv/uv_hub.h>
+#include <asm/uv/uv_irq.h>
 
 #include <mach_ipi.h>
 #include <mach_apic.h>
@@ -3694,6 +3696,72 @@ int arch_setup_ht_irq(unsigned int irq, 
 }
 #endif /* CONFIG_HT_IRQ */
 
+#ifdef CONFIG_X86_64
+/*
+ * Re-target the irq to the specified CPU and enable the specified MMR located
+ * on the specified blade to allow the sending of MSIs to the specified CPU.
+ */
+int arch_enable_uv_irq(char *irq_name, unsigned int irq, int cpu, int mmr_blade,
+		       unsigned long mmr_offset)
+{
+	const cpumask_t *eligible_cpu = get_cpu_mask(cpu);
+	struct irq_cfg *cfg;
+	int mmr_pnode;
+	unsigned long mmr_value;
+	struct uv_IO_APIC_route_entry *entry;
+	unsigned long flags;
+	int err;
+
+	err = assign_irq_vector(irq, *eligible_cpu);
+	if (err != 0)
+		return ...
[ message continues ]
[view original message]
From: Yinghai Lu
Subject: Re: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions
Date: Thursday, October 2, 2008 - 9:19 am

anything to do with MSI?

YH
--


[ message continues ]
[view original message]
From: H. Peter Anvin
Subject: Re: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions
Date: Thursday, October 2, 2008 - 9:44 am

Not really.  It probably should be called "UV-CORE" or something like that.

	-hpa
--


[ message continues ]
[view original message]
From: Dean Nelson
Subject: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions v.3
Date: Thursday, October 2, 2008 - 10:18 am

Provide a means for UV interrupt MMRs to be setup with the message to be sent
when an MSI is raised.

Signed-off-by: Dean Nelson <dcn@sgi.com>

---


Good question. Sometimes the obvious gets overlooked in the midst of

Agreed. The change is needed by drivers/misc/sgi-xp once it has been
modified to call uv_setup_irq() and uv_teardown_irq(). I'll move this
change into that patch (or should it be in a patch all by itself?).


Agreed and changed.

The patch to drivers/misc/sgi-xp to call the functions introduced by
this patch, will be submitted shortly.

Thanks,
Dean

 arch/x86/kernel/Makefile    |    2 -
 arch/x86/kernel/io_apic.c   |   68 +++++++++++++++++++++++++++++++++++
 arch/x86/kernel/uv_irq.c    |   77 ++++++++++++++++++++++++++++++++++++++++
 include/asm-x86/uv/uv_irq.h |   36 ++++++++++++++++++
 4 files changed, 182 insertions(+), 1 deletion(-)

Index: linux/arch/x86/kernel/io_apic.c
===================================================================
--- linux.orig/arch/x86/kernel/io_apic.c	2008-10-02 06:12:54.000000000 -0500
+++ linux/arch/x86/kernel/io_apic.c	2008-10-02 06:15:48.000000000 -0500
@@ -58,6 +58,8 @@
 #include <asm/setup.h>
 #include <asm/irq_remapping.h>
 #include <asm/hpet.h>
+#include <asm/uv/uv_hub.h>
+#include <asm/uv/uv_irq.h>
 
 #include <mach_ipi.h>
 #include <mach_apic.h>
@@ -3694,6 +3696,72 @@ int arch_setup_ht_irq(unsigned int irq, 
 }
 #endif /* CONFIG_HT_IRQ */
 
+#ifdef CONFIG_X86_64
+/*
+ * Re-target the irq to the specified CPU and enable the specified MMR located
+ * on the specified blade to allow the sending of MSIs to the specified CPU.
+ */
+int arch_enable_uv_irq(char *irq_name, unsigned int irq, int cpu, int mmr_blade,
+		       unsigned long mmr_offset)
+{
+	const cpumask_t *eligible_cpu = get_cpu_mask(cpu);
+	struct irq_cfg *cfg;
+	int mmr_pnode;
+	unsigned long mmr_value;
+	struct uv_IO_APIC_route_entry *entry;
+	unsigned long flags;
+	int err;
+
+	err = assign_irq_vector(irq, *eligible_cpu);
+	if ...
[ message continues ]
[view original message]
From: Ingo Molnar
Subject: Re: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions v.3
Date: Friday, October 3, 2008 - 1:59 am

v3 looks good - applied to tip/x86/uv, thanks Dean!


changed that to a _GPL export. This is rather lowlevel functionality.

	Ingo
--


[ message continues ]
[view original message]
From: Ingo Molnar
Subject: x86, UV: add uv_setup_irq() and uv_teardown_irq() functions, v3, fix
Date: Friday, October 3, 2008 - 2:40 am

-tip testing found a build error - fixlet below.

	Ingo

------------>
From 1b2127d363a68a388e7c57610bb5ceeb743c89d2 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@elte.hu>
Date: Fri, 3 Oct 2008 11:38:37 +0200
Subject: [PATCH] x86, UV: add uv_setup_irq() and uv_teardown_irq() functions, v3, fix

fix:

 arch/x86/kernel/uv_irq.c: In function 'uv_ack_apic':
 arch/x86/kernel/uv_irq.c:26: error: implicit declaration of function 'ack_APIC_irq'

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/uv_irq.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/uv_irq.c b/arch/x86/kernel/uv_irq.c
index 6bd26c9..aeef529 100644
--- a/arch/x86/kernel/uv_irq.c
+++ b/arch/x86/kernel/uv_irq.c
@@ -10,6 +10,8 @@
 
 #include <linux/module.h>
 #include <linux/irq.h>
+
+#include <asm/apic.h>
 #include <asm/uv/uv_irq.h>
 
 static void uv_noop(unsigned int irq)
--


[ message continues ]
[view original message]
From: Dean Nelson
Subject: sgi-gru drivers need of up to 8192 irqs
Date: Wednesday, November 12, 2008 - 11:31 am

Hi Ingo,

You may remember a patch that I submitted not so long ago that added
uv_setup_irq() and uv_teardown_irq() functions for UV on x86.

It gave us (SGI) a way to 'allocate' an irq/vector pair and write an MMR
with a CPU's APCID and the vector number. (Two functions were added to
/arch/x86/kernel/io_apic.c to accomplish this -- arch_enable_uv_irq() and
arch_disable_uv_irq().)

This solution was what was suggested by the community after much discussion.

I'm now trying to modify the GRU driver to call uv_setup_irq(). One of the
issues that is arising is that we need to be able to allocate lots of irqs.
For the GRU alone we need up to 32 irqs per blade (based on the number of CPUs)
and there can be up to 256 blades per SSI. (A total of 8192 irqs per SSI.)

Currently I'm finding that NR_IRQS=4352. (It is constrained to the lesser of
NR_CPUS and MAX_IO_APICS.) But the system really runs off of nr_irqs which
gets set down by probe_nr_irqs() to 96 in one configuration I've been running.
This function bases its answer on io_apic_get_redir_entries().

For the GRU there isn't an io_apic component, its needs are based on NR_CPUS.
So there isn't something readily available to cause probe_nr_irqs() to
generate a larger number.

Anyway, I'm looking for any suggestions as to what we should do. (Which may
include abandoning the using of irqs for the GRU driver and just go with two
hardwired system vectors?

Also, what has happened to the sparse irq patch? It doesn't seem to exist
anywhere. Has it been abandoned? I thought it was the means by which we would
be able to have huge numbers of irqs.

Thanks,
Dean

--


[ message continues ]
[view original message]
From: Ingo Molnar
Subject: Re: sgi-gru drivers need of up to 8192 irqs
Date: Wednesday, November 12, 2008 - 11:46 am

(Cc:-ed Yinghai and Mike Travis.)


it's being worked on, with the first half of the changes upstream 
already, and the latest version of the second half of sparseirq (patch 
v11) was posted by Yinghai to lkml today, under this thread:

Subject: [PATCH] sparse_irq aka dyn_irq v11

	Ingo
--


[ message continues ]
Previous thread: Tux3 Report: Extolling the Extensive Virtues of Extents by Daniel Phillips on Wednesday, October 1, 2008 - 4:35 am. (1 message)

Next thread: [PATCH] thermal: Make it simpler to use the thermal layer inside the kernel by Matthew Garrett on Wednesday, October 1, 2008 - 5:04 am. (9 messages)