Re: [PATCH] input: mt: introduce MT event slots

Previous thread: Re: + x86-fix-handling-of-the-reservetop-boot-option.patch added to -mm tree by Yinghai on Wednesday, April 7, 2010 - 4:52 pm. (14 messages)

Next thread: [PATCH] stmmac: Fix incorrect resource calculation by Axel Lin on Wednesday, April 7, 2010 - 6:28 pm. (2 messages)
From: Henrik Rydberg
Date: Wednesday, April 7, 2010 - 5:13 pm

With the rapidly increasing number of intelligent multi-contact and
multi-user devices, the need to send digested, filtered information
from a set of different sources within the same device is imminent.
This patch adds the concept of slots to the MT protocol. The slots
enumerate a set of identified sources, such that all MT events
can be passed independently and selectively per identified source.

The protocol works like this: Instead of sending a SYN_MT_REPORT
event immediately after the contact data, one sends a SYN_MT_SLOT
event immediately before the contact data. The input core will only
emit events for the slots corresponding to the contacts that have
changed. It is assumed that the same slot is used for the duration
of an initiated contact.

Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
---
 drivers/input/input.c |   62 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/input.h |   26 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 0 deletions(-)

diff --git a/drivers/input/input.c b/drivers/input/input.c
index afd4e2b..217e976 100644
--- a/drivers/input/input.c
+++ b/drivers/input/input.c
@@ -181,6 +181,22 @@ static void input_stop_autorepeat(struct input_dev *dev)
 #define INPUT_PASS_TO_DEVICE	2
 #define INPUT_PASS_TO_ALL	(INPUT_PASS_TO_HANDLERS | INPUT_PASS_TO_DEVICE)
 
+static void input_handle_mt_event(struct input_dev *dev,
+				  unsigned int code, int value)
+{
+	int oldval = dev->mt[dev->slot].abs[code];
+	value = input_defuzz_abs_event(value, oldval, dev->absfuzz[code]);
+	if (value == oldval)
+		return;
+	dev->mt[dev->slot].abs[code] = value;
+	dev->sync = 0;
+	if (dev->slot != dev->last_slot) {
+		dev->last_slot = dev->slot;
+		input_pass_event(dev, EV_SYN, SYN_MT_SLOT, dev->slot);
+	}
+	input_pass_event(dev, EV_ABS, code, value);
+}
+
 static void input_handle_event(struct input_dev *dev,
 			       unsigned int type, unsigned int code, int value)
 {
@@ -204,6 +220,10 @@ static void input_handle_event(struct ...
From: Michael Poole
Date: Wednesday, April 7, 2010 - 8:07 pm

How would the slot number for a contact be chosen?  If the kernel makes
that assignment, what should a "slot" correspond to from a computer
user's perspective?  "Set[s] of identified sources" is a little vague:
Does it mean contacts from one hand, contacts in one displayed window
(assuming the touch surface is a screen), or something else?  (I assume
it would not duplicate the blob or tracking IDs already defined for MT
events.)  It seems like those would be important aspects of the protocol
to document in Documentation/input/multi-touch-protocol.txt --
otherwise, driver implementers or application developers might get it
wrong.

Michael Poole
--

From: Henrik Rydberg
Date: Thursday, April 8, 2010 - 2:43 am

Michael Poole wrote:

The device driver determines how to use the slots. The driver calls
input_mt_slot(dev, slot), sends the data for the slot, picks another slot, and

The slot is only used for data communication. Think of the slot as a combined,
unique identifier. For example, imagine a device driver dealing with contacts
labeled with both a USER_ID and a TRACKING_ID. The driver assigns every active
(USER_ID, TRACKING_ID) contact to a specific slot, and uses it to communicate
all changes to that contact. When the contact is destroyed (for instance by
sending a zero ABS_MT_PRESSURE on that slot), the slot is free to be used for

Certainly.

Cheers,
Henrik

--

From: Michael Poole
Date: Thursday, April 8, 2010 - 4:40 am

For hardware with touch tracking support, what does a slot ID provide
for user-space that the tracking ID doesn't?  (TRACKING_ID is already
supposed to be unique for the life of the touch.)  For hardware without
touch tracking, is the driver expected to implement the Euclidean
bipartite matching in order to assign touch reports to slots?

(Pardon me if I'm being dense -- I'm more familiar with the kernel
driver side, not the X server's implementation or dispatch of multitouch
events.)

Michael Poole
--

From: Henrik Rydberg
Date: Thursday, April 8, 2010 - 6:01 am

Michael Poole wrote:

The purpose of the slot is twofold. Firstly, it reintroduces the ability for the
kernel to filter ABS_MT events, which reduces the number of events emitted from
the input core by a large factor. Secondly, it allows the driver to send partial
information without breaking the protocol.

The current MT protocol is designed for anonymous contacts, which dictates that
all data for all fingers has to be sent between every synchronization point.
Although this is fine for a handful of fingers, it does not play well with a
larger scenario. The slot concept allows for a minimum of information to be
emitted from the input core, without breaking compatibility with the current MT
protocol. If a single attribute of a single finger of a single user is changed,
the event sequence will simply be:

SYN_MT_SLOT <slot-in-use>
ABS_MT_ATTRIBUTE <some-value>

If the contact gets destroyed and replaced by another one, there is not even a
need to send that information explicitly, but this sequence would suffice:

SYN_MT_SLOT <slot-in-use>
ABS_MT_USER_ID <replacing-user>
ABS_MT_TRACKING_ID <replacing-tracking-id>
ABS_MT_ATTRIBUTE1 <some-value>
ABS_MT_ATTRIBUTE2 <some-value>
...


One might argue that a similar sequence could be implemented within the existing
 MT protocol. However, it would at least require a different semantic usage of
the SYN_MT_REPORT event, without the ability to detect what semantics to use.
Besides, a larger amount of events would be needed than with the proposed slot

This is a possible scenario, yes. The driver would of course use a common kernel
library for the matching task. Whether or not this will happen depends somewhat

Thank you for your questions, the discussion is very much appreciated.

Cheers,
Henrik

--

From: Hennerich, Michael
Date: Friday, April 16, 2010 - 12:40 am

Sorry, I must have missed something -
What is ABS_MT_USER_ID?

Regards,
Michael

Analog Devices GmbH      Wilhelm-Wagenfeld-Str. 6      80807 Muenchen
Sitz der Gesellschaft Muenchen, Registergericht Muenchen HRB 4036 Geschaeftsfuehrer Thomas Wessel, William A. Martin, Margaret Seif


--

From: Henrik Rydberg
Date: Friday, April 16, 2010 - 7:29 am

Hennerich, Michael wrote:

Oh, it just served as an example of possible future extensions, it is not part
of the current MT protocol. Sorry about the confusion.

Cheers,
Henrik

--

From: Rafi Rubin
Date: Thursday, April 8, 2010 - 12:15 pm

Is there any particular downside to defaulting to implicit slot ids?

For drivers/hardware that don't handle tracking, SYN_MT_REPORT could 
just result in dev->slot++ and a SYN_REPORT resets dev->slot to 0;

For tracking hardware do you envision waiting for TRACKING_ID before 
selecting a slot?  If so then either with explicit or implicit slot ids, 
we would need to cache event until the tracking id is read, and either 
use SYN_MT_SLOT or some other mechanism to denote known slot id.


I like the idea and am just wondering if we can simplify the burden for 
 > Cheers,
 > Henrik

Please clarify which slots are emitted to userspace.  At some point you 
mentioned that if any are changed all will be emitted, is that still 
your intent?  Have you reconsidered using an explicit event to signify 
the end of a contact/slot?

Rafi
--

From: Henrik Rydberg
Date: Thursday, April 8, 2010 - 1:25 pm

Yes. The device driver should not have to update every slot between

Drivers that do not handle tracking should not use the slots at all. The slot
concept requires that whatever gets communicated over it is identifiable, or
else it would not be possible to send changes. Drivers without tracking

The slot protocol propagates changes, which implies that the full state of each
initiated contact has to reside in the receiving end. Upon receiving an MT
event, one simply updates the appropriate attribute of the active slot. Done.
The TRACKING_ID is also an attribute of the slot, and gets updated the same way.
 Keeping track of fingers should thus be trivial. All in all, the SYN_MT_SLOT
method is probably more intuitive to use in the consumer end than the more

There will be more updates of the documentation following this, of course.

Picture an array of slots in the driver and an array of slots in the receiver.
For each slot, the driver keeps everything it can report, and the receiver keeps
everything if can use. The receiver may also keep an array of active tracking
ids. Each MT event updates an attribute of a slot in the receiver. Once the
synchronization event arrives, the receiver just reads off whatever the state
is. Every slot with a valid tracking id (nonzero pressure or nonzero touch_major
or the like) is an existing contact. Compare to the array of active tracking

No. The current MT protocol, using the SYN_MT_REPORT event, is well-defined in
that respect, will always send all data for all fingers, and requires no further
changes. For the slot extension, which uses SYN_MT_SLOT instead of
SYN_MT_REPORT, the issue with contact destruction arises in the drivers that
currently only report ABS_MT_POSITION_X and ABS_MT_POSITION_Y. Adding either
ABS_MT_PRESSURE or ABS_MT_TOUCH_MAJOR to those drivers, as intended in the MT
protocol, resolves the problem. The explicit contact destruction then comes
naturally as a zero touch event.

Cheers,
Henrik
--

From: Rafi Rubin
Date: Thursday, April 8, 2010 - 2:23 pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


That's unfortunate.

I think tracking upsets are generally quite rare (at least for the n-trig
hardware), and we would see most of the benefit of jitter and bandwidth
reduction even if we use contact ordering for slots.  Tracking upsets would
still flow downstream, a large state change should cause the slot to emit the
new position.

I was also hoping the slotting mechanism might be a good place to inject generic
tracking support later.

Rafi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAku+SWcACgkQwuRiAT9o60894wCg1lQIzcFgmUNqUpiKJSDigxNE
QVcAn3YylXnlNaieGTJyQ2UblpqR5X7q
=EokA
-----END PGP SIGNATURE-----
--

From: Henrik Rydberg
Date: Thursday, April 8, 2010 - 3:12 pm

Rafi Rubin wrote:

But it is! It was not my intention to discourage the slot protocol for a driver
that *wants* to track contacts, only the ones that do not. As you already
guessed, there is a natural migration path towards using the slot extension and
kernel-provided software finger tracking.

Cheers,
Henrik

--

From: Dmitry Torokhov
Date: Thursday, April 15, 2010 - 4:25 pm

Hi Henrik,


I see the reason for doing this however I would like to hold off
applying it till we get a couple of users and prove that the scheme


It would be nice to use abs[ABS_MT_MAX - ABS_MT_TOUCH_MAJOR] to save
some memory.

-- 
Dmitry
--

From: Henrik Rydberg
Date: Friday, April 16, 2010 - 8:09 am

Spot on! I was choosing between different strategies here: from a) mapping just
the ABS_MT values to keep memory down (but introduce yet another mapping); to b)
opening up the protocol for all kinds of event types (after all, it is only
backwards compatibility that prevents the slots from being used also for, say,
button events). It seemed best to keep the possibility for new event types to be
used with slots, thus using a straight mapping to the ABS events and ignoring
the extra memory used. However the word "new" is not enforced with this patch,
which your change remedies in the most satisfactory way. Nice.

I will wait for some more implementation test results (I know of one currently
in progress), then I will be back with an updated patch and accompanying
documentation.

Cheers,
Henrik
--

From: Henrik Rydberg
Date: Tuesday, May 4, 2010 - 12:57 am

Is it possible this was meant to read "ABS_MAX - ABS_MT_TOUCH_MAJOR"? It would
explain my confusion.

Henrik

--

From: Dmitry Torokhov
Date: Tuesday, May 4, 2010 - 10:29 am

By ABS_MT_MAX I meant highest MT event currently defined. No need to go all
the way to ABS_MAX if there are no MT events defined there.

-- 
Dmitry
--

From: Henrik Rydberg
Date: Tuesday, May 4, 2010 - 2:41 pm

I see, thanks.

Henrik
--

Previous thread: Re: + x86-fix-handling-of-the-reservetop-boot-option.patch added to -mm tree by Yinghai on Wednesday, April 7, 2010 - 4:52 pm. (14 messages)

Next thread: [PATCH] stmmac: Fix incorrect resource calculation by Axel Lin on Wednesday, April 7, 2010 - 6:28 pm. (2 messages)