Abstract Designing of touchless user interface is gaining popularity in various contexts. Users can interact with electronic devices using such interfaces even when their hands are dirty or non-conductive. Also, users with partial physical disability can interact with electronic devices with the help of touchless interfaces. In this paper, we propose a Leap Motion controller-based methodology to facilitate rendering of 2D and 3D shapes on display devices. The proposed method tracks finger movements while users perform natural gestures within the field of view of the motion sensor. Then, trajectories are analyzed to extract extended Npen++ features in 3D. These features capture finger movements during the gestures and they are fed to unidirectional left-to-right Hidden Markov Model (HMM) for training. A one-to-one mapping between gestures and shapes, is proposed. Finally, the shapes corresponding to these gestures are rendered over the display using a typical MuPad supported interface. We have created a dataset of 5400 samples recorded by 10 volunteers. Our dataset contains 18 geometric and 18 non-geometric shapes such as “circle”, “rectangle”, “flower”, “cone”, “sphere”, etc. The proposed method has achieved 92.87 accuracy using a 5-fold cross validation scheme. Experiments reveal that the extended 3D features perform better than the existing 3D features when applied for shape representation and classification. The method can be used for developing diverse HCI applications suitable for smart display devices.
@cite_0: The complexity of hand function is such that most existing upper limb rehabilitation robotic devices use only simplified hand interfaces. This is in contrast to the importance of the hand in regaining function after neurological injury. Computer vision technology has been used to identify hand posture in the field of Human Computer Interaction, but this approach has not been translated to the rehabilitation context. We describe a computer vision-based classifier that can be used to discriminate rehabilitation-relevant hand postures, and could be integrated into a virtual reality-based upper limb rehabilitation system. The proposed system was tested on a set of video recordings from able-bodied individuals performing cylindrical grasps, lateral key grips, and tip-to-tip pinches. The overall classification success rate was 91.2 , and was above 98 for 6 out of the 10 subjects.
@cite_14: An exercise and therapy hand device consists of a glove for receiving a hand. The glove includes a palm portion, one or more finger portions for receiving a finger, a back portion, and a compressible substance that is coupled to the palm portion. Also included in the glove are one or more elastic members that span from the compressible substance to respective ones of the one or more finger portions. The compressible substance exercises the muscles of the hand that control the closing of the hand, while the one or more elastic members exercise the muscles of the hand that control the opening of the hand.
@cite_33: This paper introduces a human computer interaction system by using hand gesture recognition to apply into game-based rehabilitation application. The basic concept of this project is to assist patient to carry out rehabilitation with hand gesture recognition, meanwhile therapist is able to follow recovery progress of patient through results. In this manner, a rehabilitation system created aimed to increase efficacy in patient treatment with interactive game. The system is applied in real time application to contribute as a function of a virtual mouse. Approach proposed including hand region segmentation, hand palm partition and finger counting. Computer mouse is configured to move according to the location of hand palm region, and perform function of ordinary mouse according to number of fingers detected. In the analysis, experiments is carried out to show how this system collaborate with the simple rehabilitation game in computer. Experiment evaluation as shown has successfully demonstrate the effectiveness and robustness in performing the mouse function and feasible to be implemented into rehabilitation games.
Though, normal camera-based systems are popular in gesture recognition, however, they have certain disadvantages as compared to IR camera-based or sensor-based systems. For example, vision-based gesture recognition system proposed by @cite_0 suffers from segmentation error. The method proposed by @cite_33 assumes a simple background to avoid segmentation error, which is not realistic. On the other hand, sensor-based systems are more accurate since the signals acquired by IMU sensor are less affected by variations in illumination or segmentation error @cite_14 .