Gestures for Touchpad / Touchscreen targets

[NarcoticV]

Discussion started in: http://forums.rockbox.org/index.php/topic,43431.0.html

Gestures such as flicking through songs and lists, etc, would be a nice addition to the existing control scheme for touchscreens and touchpads.

A few ways to fit this into Rockbox were stated in the topic above. In this topic we can brainstorm about ways to do it, what to use it for, etc.


 - Would there have to be a setting for it?
                 - Yes, because people will strongly disagree about (some) gestures
                 - No, because having customizable controls is a lot of hassle and there should be a "best" way
                 - etc

 - How to combine gesture code with the existing button system?
                 - Have the gesture support at a low, target-specific level, and convert gestures to button presses
                 - Implement a whole new control system parallel to the existing button system, starting from scratch
                 - etc

 - How to give plug-ins access to gesture support and also directly to touchpad input data?

 - Ideas for gestures that would be cool to have?
                 - flicking through items
                 - a virtual scrollwheel a la old IPods
                 - zooming with multiple fingers for devices that support it
                 - an unlock sequence like smartphones have
                 - etc

[metaphys]

regarding how many target are on the line with touchpad I think that the fuze+ is pretty much alone for the moment:
There are other HAVE_TOUCHPAD targets but when ones have a closer look at them it turns out that those are not actual touchpad in a sense of a plain square sensitive board. Some have only a virtual scroll bar, some other are a collection of several touch button and so on.

regarding a way to implement cleanly a general way to add gesture for all touchpad target, It would start in my opinion by having a mandatory function in all touchpad driver providing at least absolute coordinate of a touch, the number of them.

regarding the simulator, when I started with idea of implementing gestures, I figured out that it will be almost mandatory for testing purpose. One idea would be to be able to use the device interface in usb mode to control the simulator. Pamaury had some thought about it but he doesn't had time and it's should be a lot of work.

I don't have a lot of time either but I would be happy to help from time to time if someone starts giving some time to this

[NarcoticV]

@metaphys: I agree about the touchpad driver providing generic touchpad data, which every touchpad configuration should be able to comply to. So coordinates, touchpad size, number of fingers, so plugins and gesture code can at least poll this data.

Tinkering with the code, I am also starting to think that mapping gestures to buttons (aka a flick gesture would return BUTTON_UP to button_read_device() ) seems to be a really bad idea. In combination with button modifiers such as BUTTON_REPEAT and BUTTON_REL it quickly becomes a puzzle to "emulate" a normal button press to fool the button system.

I think there should be a gesture system parallel to the button system. They would come together only in two places: in the low-level, target-specific code that processes touchpad data, and in the keymap file. Gestures would be reported parallel to buttons, the common code would know that they are gestures and not buttons, and they would be directly mapped to actions in the keymap file.

Maybe the gesture system (say gesture.c) should be closer to the touchpad driver than the button system: after all, a button press can be regarded as a gesture in its own right, and the gesture code should be able to differentiate between button presses (taps) and other gestures. If a tap is recognized, the coordinates of the tap can get passed to button_<target>.c .

[Lorenzo92]

I also like the idea about gestures very much! Nowadays there are more and more touchscreens and Rockbox must be in the future too
(personally I don't like touch-based players, but I have one and I'm willing to improve rockbox support for it!)
Unfortunately I don't have much time yet to spend on this, but I want to help as much as possible....

[monoid]

Maybe the gesture system (say gesture.c) should be closer to the touchpad driver than the button system: after all, a button press can be regarded as a gesture in its own right, and the gesture code should be able to differentiate between button presses (taps) and other gestures. If a tap is recognized, the coordinates of the tap can get passed to button_<target>.c .

I do not know Rockbox code and structure, but this sounds correct. The gesture layer IMO should be closer to the physical layer and touch key layer above gesture layer. The gesture layer would "eat" all gesture related touch events and pass on only the rest of events to touch key layer.

This way gesture layer could be seamlessly integrated into Rockbox structure and switched on/off. And no code change would be needed in touch key layer.

The only problém, I see, might be, that some key events might be delayed (if gesture layer "waits" whether consequtive event comes or not...). One would have to try it in real life.

[NarcoticV]

The gesture layer would "eat" all gesture related touch events and pass on only the rest of events to touch key layer.

This way gesture layer could be seamlessly integrated into Rockbox structure and switched on/off. And no code change would be needed in touch key layer.

The only problém, I see, might be, that some key events might be delayed (if gesture layer "waits" whether consequtive event comes or not...). One would have to try it in real life.

I agree. There will be some challenges with this: in the current implementation, the button code gets notified immediately when the finger comes down somewhere, and can do BUTTON_REPEAT and BUTTON_REL modifiers accordingly. This is a form of gesture detection on its own. To keep it consistent that touchpad gestures are detected by the gesture code and not the button code, that existing functionality should not be used for touchpad buttons: a "tap" gesture with coordinates from the gesture code should be directly translated into an action, skipping over the button code.
Downside is that there is some re-invention of wheels: the gesture code would detect button long/double presses and taps, bahaviour that is also already in the button code. I still think this is the best way though.

To see it a bit more clearly, I used mediocre powerpoint skills to create some proposals for the new file structure. See the included PDF.

I would go for 2 or 3. I prefer 3 actually, because the only difference between a touchpad and a touchscreen is that the touchscreen is mapped to the screen resolution (and needs calibration), and the touchpad isn't and doesn't. That, in my opinion, is too small a difference to justify an entire separate API with many very similar functions.

Implementing this can be quite easily split up into multiple patches, starting with restructuring the APIs.

What do you guys think?

--edit: there is a small mistake in the last slide (proposal 3): it says it's almost the same as proposal 1. I meant proposal 2. --

[metaphys]

I'll also got for 2 or 3. I'm not sure about sparing file for case that seems to be the same but could bring slight differences later and need a rewrite....