Re: [PATCH v3 0/4] Introduce hardware spinlock framework

Previous thread: [PATCH 0/5] KVM&genirq: Enable adaptive IRQ sharing for passed-through devices by Jan Kiszka on Friday, December 3, 2010 - 4:37 pm. (16 messages)

Next thread: [PATCH RFCv3 0/2] CARMA Board Support by Ira W. Snyder on Friday, December 3, 2010 - 4:57 pm. (3 messages)
From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 4:50 pm

OMAP4 introduces a Hardware Spinlock device, which provides hardware
assistance for synchronization and mutual exclusion between heterogeneous
processors and those not operating under a single, shared operating system
(e.g. OMAP4 has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP).

The intention of this hardware device is to allow remote processors,
that have no alternative mechanism to accomplish synchronization and mutual
exclusion operations, to share resources (such as memory and/or any other
hardware resource).

This patchset adds hwspinlock framework that makes it possible
for drivers to use those hwspinlock devices and stay platform-independent.

Currently there are two users for this hwspinlock interface:

1. Inter-processor communications: on OMAP4, cpu-intensive multimedia
tasks are offloaded by the host to the remote M3 and/or C64x+ slave
processors.

To achieve fast message-based communications, a minimal kernel support
is needed to deliver messages arriving from a remote processor to the
appropriate user process.

This communication is based on a simple data structure that is shared between
the remote processors, and access to it is synchronized using the hwspinlock
module (remote processor directly places new messages in this shared data
structure).

2. On some OMAP4 boards, the I2C bus is shared between the A9 and the M3,
and the hwspinlock is used to synchronize access to it.

While (2) can get away with an omap-specific hwspinlock implementation,
(1) is by no means omap-specific, and a common hwspinlock interface is
needed to keep it generic, in hopes that it will be useful for other
platforms as well.

Changes v2->v3:
- Remove the timeout-less _lock API variant (Tony)
- s/state->io_base/io_base/ (Ionut)
- Remove the "generic" wording (David)
- s/hwspinlock_/hwspin_lock_/ (Mugdha)
- Use MAX_SCHEDULE_TIMEOUT to indicate no timeout (Mugdha)
- Be more verbose on egregious API misuse (Olof)
- locking API misuse is BUG_ON material (Russell)

  Note: ...
From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 4:50 pm

Add a platform-independent hwspinlock framework.

Hardware spinlock devices are needed, e.g., in order to access data
that is shared between remote processors, that otherwise have no
alternative mechanism to accomplish synchronization and mutual exclusion
operations.

Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Hari Kanigeri <h-kanigeri2@ti.com>
Cc: Benoit Cousson <b-cousson@ti.com>
Cc: Kevin Hilman <khilman@deeprootsystems.com>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Tony Lindgren <tony@atomide.com>
---
 Documentation/hwspinlock.txt         |  299 ++++++++++++++++++
 drivers/Kconfig                      |    2 +
 drivers/Makefile                     |    1 +
 drivers/hwspinlock/Kconfig           |   13 +
 drivers/hwspinlock/Makefile          |    5 +
 drivers/hwspinlock/hwspinlock.h      |   61 ++++
 drivers/hwspinlock/hwspinlock_core.c |  557 ++++++++++++++++++++++++++++++++++
 include/linux/hwspinlock.h           |  298 ++++++++++++++++++
 8 files changed, 1236 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/hwspinlock.txt
 create mode 100644 drivers/hwspinlock/Kconfig
 create mode 100644 drivers/hwspinlock/Makefile
 create mode 100644 drivers/hwspinlock/hwspinlock.h
 create mode 100644 drivers/hwspinlock/hwspinlock_core.c
 create mode 100644 include/linux/hwspinlock.h

diff --git a/Documentation/hwspinlock.txt b/Documentation/hwspinlock.txt
new file mode 100644
index 0000000..65324ce
--- /dev/null
+++ b/Documentation/hwspinlock.txt
@@ -0,0 +1,299 @@
+Hardware Spinlock Framework
+
+1. Introduction
+
+Hardware spinlock modules provide hardware assistance for synchronization
+and mutual exclusion between heterogeneous processors and those not operating
+under a single, shared operating system.
+
+For example, OMAP4 has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP,
+each of which is running a different Operating System (the master, A9,
+is usually running Linux and the slave processors, the M3 and the DSP,
+are running some ...
From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 4:50 pm

From: Simon Que <sque@ti.com>

Add hwspinlock support for the OMAP4 Hardware Spinlock device.

The Hardware Spinlock device on OMAP4 provides hardware assistance
for synchronization between the multiple processors in the system
(dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP).

[ohad@wizery.com: adapt to hwspinlock framework, tidy up]
Signed-off-by: Simon Que <sque@ti.com>
Signed-off-by: Hari Kanigeri <h-kanigeri2@ti.com>
Signed-off-by: Krishnamoorthy, Balaji T <balajitk@ti.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Benoit Cousson <b-cousson@ti.com>
Cc: Kevin Hilman <khilman@deeprootsystems.com>
Cc: Grant Likely <grant.likely@secretlab.ca>
Cc: Tony Lindgren <tony@atomide.com>
---
 drivers/hwspinlock/Kconfig           |    9 ++
 drivers/hwspinlock/Makefile          |    1 +
 drivers/hwspinlock/omap_hwspinlock.c |  231 ++++++++++++++++++++++++++++++++++
 3 files changed, 241 insertions(+), 0 deletions(-)
 create mode 100644 drivers/hwspinlock/omap_hwspinlock.c

diff --git a/drivers/hwspinlock/Kconfig b/drivers/hwspinlock/Kconfig
index 9dd8db4..eb4af28 100644
--- a/drivers/hwspinlock/Kconfig
+++ b/drivers/hwspinlock/Kconfig
@@ -11,3 +11,12 @@ config HWSPINLOCK
 	  coprocessors).
 
 	  If unsure, say N.
+
+config HWSPINLOCK_OMAP
+	tristate "OMAP Hardware Spinlock device"
+	depends on HWSPINLOCK && ARCH_OMAP4
+	help
+	  Say y here to support the OMAP Hardware Spinlock device (firstly
+	  introduced in OMAP4).
+
+	  If unsure, say N.
diff --git a/drivers/hwspinlock/Makefile b/drivers/hwspinlock/Makefile
index b9d2b9f..5729a3f 100644
--- a/drivers/hwspinlock/Makefile
+++ b/drivers/hwspinlock/Makefile
@@ -3,3 +3,4 @@
 #
 
 obj-$(CONFIG_HWSPINLOCK)		+= hwspinlock_core.o
+obj-$(CONFIG_HWSPINLOCK_OMAP)		+= omap_hwspinlock.o
diff --git a/drivers/hwspinlock/omap_hwspinlock.c b/drivers/hwspinlock/omap_hwspinlock.c
new file mode 100644
index 0000000..b5867e1
--- /dev/null
+++ b/drivers/hwspinlock/omap_hwspinlock.c
@@ -0,0 +1,231 @@
+/*
+ * OMAP hardware ...
From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 4:50 pm

From: Benoit Cousson <b-cousson@ti.com>

Add hwspinlock hwmod data for OMAP4 chip

Signed-off-by: Cousson, Benoit <b-cousson@ti.com>
Signed-off-by: Hari Kanigeri <h-kanigeri2@ti.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Paul Walmsley <paul@pwsan.com>
---
 arch/arm/mach-omap2/omap_hwmod_44xx_data.c |   64 ++++++++++++++++++++++++++++
 1 files changed, 64 insertions(+), 0 deletions(-)

diff --git a/arch/arm/mach-omap2/omap_hwmod_44xx_data.c b/arch/arm/mach-omap2/omap_hwmod_44xx_data.c
index d258936..e577d54 100644
--- a/arch/arm/mach-omap2/omap_hwmod_44xx_data.c
+++ b/arch/arm/mach-omap2/omap_hwmod_44xx_data.c
@@ -1376,6 +1376,67 @@ static struct omap_hwmod omap44xx_gpio6_hwmod = {
 	.slaves_cnt	= ARRAY_SIZE(omap44xx_gpio6_slaves),
 	.omap_chip	= OMAP_CHIP_INIT(CHIP_IS_OMAP4430),
 };
+
+/*
+ * 'spinlock' class
+ * spinlock provides hardware assistance for synchronizing the processes
+ * running on multiple processors
+ */
+
+static struct omap_hwmod_class_sysconfig omap44xx_spinlock_sysc = {
+	.rev_offs	= 0x0000,
+	.sysc_offs	= 0x0010,
+	.syss_offs	= 0x0014,
+	.sysc_flags	= (SYSC_HAS_AUTOIDLE | SYSC_HAS_CLOCKACTIVITY |
+			   SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE |
+			   SYSC_HAS_SOFTRESET | SYSS_HAS_RESET_STATUS),
+	.idlemodes	= (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART),
+	.sysc_fields	= &omap_hwmod_sysc_type1,
+};
+
+static struct omap_hwmod_class omap44xx_spinlock_hwmod_class = {
+	.name = "spinlock",
+	.sysc = &omap44xx_spinlock_sysc,
+};
+
+/* spinlock */
+static struct omap_hwmod omap44xx_spinlock_hwmod;
+static struct omap_hwmod_addr_space omap44xx_spinlock_addrs[] = {
+	{
+		.pa_start	= 0x4a0f6000,
+		.pa_end		= 0x4a0f6fff,
+		.flags		= ADDR_TYPE_RT
+	},
+};
+
+/* l4_cfg -> spinlock */
+static struct omap_hwmod_ocp_if omap44xx_l4_cfg__spinlock = {
+	.master		= &omap44xx_l4_cfg_hwmod,
+	.slave		= &omap44xx_spinlock_hwmod,
+	.clk		= "l4_div_ck",
+	.addr		= omap44xx_spinlock_addrs,
+	.addr_cnt	= ...
From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 4:50 pm

From: Simon Que <sque@ti.com>

Build and register an hwspinlock platform device.

Although only OMAP4 supports the hardware spinlock module (for now),
it is still safe to run this initcall on all omaps, because hwmod lookup
will simply fail on hwspinlock-less platforms.

Signed-off-by: Simon Que <sque@ti.com>
Signed-off-by: Hari Kanigeri <h-kanigeri2@ti.com>
Signed-off-by: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Benoit Cousson <b-cousson@ti.com>
Cc: Kevin Hilman <khilman@deeprootsystems.com>
Cc: Paul Walmsley <paul@pwsan.com>
---
 arch/arm/mach-omap2/Makefile     |    1 +
 arch/arm/mach-omap2/hwspinlock.c |   63 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 0 deletions(-)
 create mode 100644 arch/arm/mach-omap2/hwspinlock.c

diff --git a/arch/arm/mach-omap2/Makefile b/arch/arm/mach-omap2/Makefile
index fbc8739..0ea2df3 100644
--- a/arch/arm/mach-omap2/Makefile
+++ b/arch/arm/mach-omap2/Makefile
@@ -197,3 +197,4 @@ obj-y					+= $(smc91x-m) $(smc91x-y)
 
 smsc911x-$(CONFIG_SMSC911X)		:= gpmc-smsc911x.o
 obj-y					+= $(smsc911x-m) $(smsc911x-y)
+obj-$(CONFIG_ARCH_OMAP4)		+= hwspinlock.o
diff --git a/arch/arm/mach-omap2/hwspinlock.c b/arch/arm/mach-omap2/hwspinlock.c
new file mode 100644
index 0000000..06d4a80
--- /dev/null
+++ b/arch/arm/mach-omap2/hwspinlock.c
@@ -0,0 +1,63 @@
+/*
+ * OMAP hardware spinlock device initialization
+ *
+ * Copyright (C) 2010 Texas Instruments Incorporated - http://www.ti.com
+ *
+ * Contact: Simon Que <sque@ti.com>
+ *          Hari Kanigeri <h-kanigeri2@ti.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ ...
From: David Daney
Date: Friday, December 3, 2010 - 5:29 pm

Does anything other than OMAP4 have one of these things?  And are there 
any devices that are commonly encountered across more than one platform 
that might have these hardware spinlocks, thus making it advantageous to 
have a common framework?

If not why a general purpose framework for a very chip specific feature?

There are chips with hardware atomic memory, but the only time it makes 
sense to use it is in conjunction with specialize on-chip devices.  So 
the implementation details are kept in the drivers rather than creating 
a general purpose hwatomic (with its hwatomic_add() et al.) type.

--

From: Ohad Ben-Cohen
Date: Friday, December 3, 2010 - 6:28 pm

I'm not aware of any, but David Brownell had some ideas about it in
our previous v2 discussion (see
http://www.mail-archive.com/linux-omap@vger.kernel.org/msg39413.html).

Btw, you might want to read throughout that entire discussion with

Such devices are just multiple processors that have no alternative
mechanism to accomplish synchronization.

OMAP4 has a few non-coherent heterogeneous processors that do not
support fancy synchronization primitives, so it needs this hwspinlock
peripheral.

Otherwise, I don't know how common that is (/might become), and I'd
hate speculating, but I suspect that OMAP is not going to be the only
platform with multiple coprocessors, that have no other means of
achieving synchronization, and with which inter-processor

We started out with an omap-specific driver (see first iteration at
http://lwn.net/Articles/410375/), but were asked to make it a
platform-agnostic framework, in order to keep the IPC drivers that
will use it generic (and assuming that more users will show up for
such framework).

To me it sounds reasonable, but again, I prefer not to speculate how
many users will show up for this, if any.


This case is a bit different IMHO: some of the potential users of
hwspinlocks are just generic IPC drivers that are by no means
platform-specific. It's not a special on-chip device: it's just one
processor, trying to achieve mutual exclusion with another processor,
before manipulating some shared data structure.
--

From: Ohad Ben-Cohen
Date: Tuesday, December 14, 2010 - 7:31 am

Hi Greg, Tony,



Can you please have a look and say if this looks OK ?

If it does, where would you prefer this to go through ?

Thanks,
Ohad.
--

From: Greg KH
Date: Tuesday, December 14, 2010 - 10:06 am

Look at what, I don't see a patch here.

I've seen a lot of discussion about this, are all of the review comments
now addressed?  Like the most important one, why is this generic code if
it's only for one specific platform?

thanks,

greg k-h
--

From: Ohad Ben-Cohen
Date: Tuesday, December 14, 2010 - 11:40 am

Here's the complete patchset:

http://www.mail-archive.com/linux-omap@vger.kernel.org/msg39833.html


Yes, all comments were addressed in this v3 iteration, and this thread

We started out with an omap-specific driver, but Tony preferred that we
make this a platform-agnostic framework, in order to keep the IPC drivers that
will use it generic, and assuming that more users will show up for
such framework.

To me it sounds reasonable, but both ways (framework / omap-specific
driver) will work for us just fine, and I can switch back to a misc
driver if this is preferred.

The complete discussion of v1 is at:
http://comments.gmane.org/gmane.linux.kernel/1049802

We also discussed this at v2 of the patches with David, see the
complete discussion at:
http://comments.gmane.org/gmane.linux.kernel/1067016

Thanks,
--

From: Ohad Ben-Cohen
Date: Thursday, December 16, 2010 - 1:42 pm

I was just told about additional users for this (thanks Mugdha):

1) There are several platforms (such as omap3530 and omapl1xx) that
have no such hardware support, but would still need to use the same
IPC drivers (to communicate with their DSP).

The only way to achieve mutual exclusion on those platforms is by
using a shared-memory synchronization algorithm called Peterson's
Algorithm [1]. We would still need the same hwspinlock framework for
that - the busy looping, the timeout, the various locking schemes, the
resource allocation - are all still valid. The only difference is the
actual lock implementation.

2) The C6474, which is a multi-core DSP device [2], have Linux running
on one of its cores, and would be using the same IPC drivers, too.
C6474 has hardware support for synchronization, which would also
benefit from such hwspinlock module (btw the C6474 has a richer
hardware module that would need more than the hwspinlock framework
offer today - it also supports queuing, owner semantics and interrupt
notification to let a processor know when it acquires a lock, so it
wouldn't have to spin..)

Thanks,
Ohad.

[1] http://en.wikipedia.org/wiki/Peterson's_algorithm
--

From: Greg KH
Date: Thursday, December 16, 2010 - 2:08 pm

That's because we are all busy with other things :(

Sorry, I really don't have the time at the moment to review this code,
nor does it seem to affect areas that I maintain, so I don't understand
what you are looking for from me.

confused,

greg k-h
--

From: Ohad Ben-Cohen
Date: Thursday, December 16, 2010 - 2:34 pm

Oh sure, my intention was only positive (to demonstrate that there are
no outstanding comments), sorry if it sounded differently than

Sorry for the noise - I contacted you because you maintain the driver
core, in the hope you could suggest where this can go through.

Tony, Andrew, can you please have a look ?

Any comment or suggestion is appreciated.

Thanks,
Ohad.
--

From: Tony Lindgren
Date: Friday, December 17, 2010 - 5:53 pm

Looks sane to me from omap point of view and it seems the locks
are now all timeout based:

Acked-by: Tony Lindgren <tony@atomide.com>

Regards,

Tony
--

From: Ohad Ben-Cohen
Date: Tuesday, January 4, 2011 - 5:23 am

Hi Andrew,


Can you please have a look at this patch set (see link no. [1] below) ?

Short summary:

This hwspinlock framework adds support for hardware-based locking
devices, needed to accomplish synchronization and mutual exclusion
operations between remote processors that have no alternative
mechanism to do so.

This patch set adds a framework and an OMAP implementation. It had
circulated several times in the relevant mailing lists, and all
comments were addressed. The third version of this patch set [1] was
submitted about a month ago and so far there is no outstanding
comment.

Users for this framework that we are aware of:

1. OMAP4 and Davinci Netra (DM8168): both of these SoC have the same
hwspinlock module and will share the same host implementation.

2. Other platforms (such as omap3530 and omapl1xx) that have no such
hardware support, but would still need to achieve synchronization (to
communicate with their DSP).

The only way to achieve mutual exclusion on those platforms is by
using a shared-memory synchronization algorithm called Peterson's
Algorithm [2].

We would still need the same hwspinlock framework for that - the busy
looping, the timeout, the various locking schemes, the lock resource
allocation - are all still valid. The only difference would be the
actual lock implementation, therefore we will add another host
implementation
for these platforms.

3. The C6474, a multi-core DSP device [3], have Linux running on one
of its cores, and hardware support for synchronization (btw the C6474
has a richer hardware module that would need more than the hwspinlock
framework offer today - it also supports queuing, owner semantics and
interrupt notification to let a processor know when it acquires a
lock, so it wouldn't have to spin..).  Disclaimer: it will probably
take some time until c6x support is merged upstream, but this is
something that is being actively worked on [4].


Any comment or suggestion is appreciated.

Thanks,
Ohad.

[1] ...
From: Andrew Morton
Date: Tuesday, January 4, 2011 - 1:19 pm

On Tue, 4 Jan 2011 14:23:20 +0200

I looked - it looks reasonable.  This is exactly the wrong time to be
looking at large new patchsets - please refresh, retest and resend
after -rc1 is released?

--

From: Ohad Ben-Cohen
Date: Tuesday, January 4, 2011 - 1:42 pm

On Tue, Jan 4, 2011 at 10:19 PM, Andrew Morton

Sure, thanks !
--

Previous thread: [PATCH 0/5] KVM&genirq: Enable adaptive IRQ sharing for passed-through devices by Jan Kiszka on Friday, December 3, 2010 - 4:37 pm. (16 messages)

Next thread: [PATCH RFCv3 0/2] CARMA Board Support by Ira W. Snyder on Friday, December 3, 2010 - 4:57 pm. (3 messages)