Finger and hand detection for multi-touch interfaces based on maximally stable extremal regions

Detection of multiple hands and fingers. Note that the system is robust against false positives from contact with the palm of the hand.

Abstract

We propose a new approach for touch detection on optical multi-touch devices that exploits the fact that the camera images reveal not only the actual touch points, but also objects above the screen such as the hand or arm of a user. Our touch processing relies on the Maximally Stable Extremal Regions algorithm for finding the users' fingertips in the camera image. The hierarchical structure of the generated extremal regions serves as a starting point for agglomerative clustering of the fingertips into hands. Furthermore, we suggest a heuristic supporting the identification of individual fingers as well as the distinction between left hands and right hands if all five fingers of a hand are in contact with the touch surface.

Our evaluation confirmed that the system is robust against detection errors resulting from non-uniform illumination and reliably assigns touch points to individual hands based on the implicitly tracked context information. The efficient multi-threaded implementation handles two-handed input from multiple users in real-time.

Publication

Philipp Ewerling, Alexander Kulik, and Bernd Froehlich
Finger and hand detection for multi-touch interfaces based on maximally stable extremal regions. 
In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12). pp. 173-182,  November 2012.
DOI=10.1145/2396636.2396663
[preprint]