Abstract Designing of touchless user interface is gaining popularity in various contexts. Users can interact with electronic devices using such interfaces even when their hands are dirty or non-conductive. Also, users with partial physical disability can interact with electronic devices with the help of touchless interfaces. In this paper, we propose a Leap Motion controller-based methodology to facilitate rendering of 2D and 3D shapes on display devices. The proposed method tracks finger movements while users perform natural gestures within the field of view of the motion sensor. Then, trajectories are analyzed to extract extended Npen++ features in 3D. These features capture finger movements during the gestures and they are fed to unidirectional left-to-right Hidden Markov Model (HMM) for training. A one-to-one mapping between gestures and shapes, is proposed. Finally, the shapes corresponding to these gestures are rendered over the display using a typical MuPad supported interface. We have created a dataset of 5400 samples recorded by 10 volunteers. Our dataset contains 18 geometric and 18 non-geometric shapes such as “circle”, “rectangle”, “flower”, “cone”, “sphere”, etc. The proposed method has achieved 92.87 accuracy using a 5-fold cross validation scheme. Experiments reveal that the extended 3D features perform better than the existing 3D features when applied for shape representation and classification. The method can be used for developing diverse HCI applications suitable for smart display devices.
@cite_22: Rehabilitation for chronic pain follows a multidisciplinary approach, which despite the effort, often lacks the long term success and patients often fail to translate the skills learned in therapy to every day life. Serious games are hypothesized to support patients to self manage their complaints and keep training their physical functions by themselves, especially, when the game is controlled by the patient's own body performance. In this paper we present the implementation of a system providing multimodal input, including our own full body motion capture system, a low cost motion capture system (Microsoft Kinect) and biosignal acquisition devices to a game engine. In addition, a workflow has been established, that enables the use of the acquired multimodal data for serious games in a medical environment. Finally, a serious game has been implemented, targeting rehabilitation of patients with chronic pain of the lower back and neck. The focus of this work is on the multimodal input and how it is used in a game to support rehabilitation of chronic pain patients. A brief comparison of a marker-based full body MoCap system and Microsoft's Kinect is included. Preliminary results of tests currently underway are provided.
@cite_7: This paper presents a development program of a new rehabilitation workstation for hand dysfunction. The rehabilitation workstation has functions as follows: hand dysfunction diagnosis, assessment of disability, hand rehabilitation by physical training. With the assistance of data gloves as well as hand strength measuring training devices designed by the present authors, this workstation measures the hand joints movement accurately, and assesses and trains muscle strength. The system assists physiatrists to perform clinical activities and conduct scientific researches such as disability evaluation, treatment decision, recovery evaluation, clinical information analysis and data filing. As a result, an integral system can be established for rehabilitation of hand dysfunction. This system makes the simultaneous treatments available to patients by simply increasing the number of hand-training devices; the efficiency of rehabilitation is improved consequently.
@cite_3: Physical rehabilitation supported by the computer-assisted-interface is gaining popularity among health-care fraternity. In this paper, we have proposed a computer-vision-assisted contactless methodology to facilitate palm and finger rehabilitation. Leap motion controller has been interfaced with a computing device to record parameters describing 3-D movements of the palm of a user undergoing rehabilitation. We have proposed an interface using Unity3D development platform. Our interface is capable of analyzing intermediate steps of rehabilitation without the help of an expert, and it can provide online feedback to the user. Isolated gestures are classified using linear discriminant analysis (DA) and support vector machines (SVM). Finally, a set of discrete hidden Markov models (HMM) have been used to classify gesture sequence performed during rehabilitation. Experimental validation using a large number of samples collected from healthy volunteers reveals that DA and SVM perform similarly while applied on isolated gesture recognition. We have compared the results of HMM-based sequence classification with CRF-based techniques. Our results confirm that both HMM and CRF perform quite similarly when tested on gesture sequences. The proposed system can be used for home-based palm or finger rehabilitation in the absence of experts.
@cite_6: In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller's hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our free-hand Fruit Ninja's score is a good indicator of the patient's hand function and therefore will be informative if used in their rehabilitation.
@cite_47: 
Despite good accuracy of the sensor-based systems, they have certain drawbacks; (i) Contact-based systems are a burden to the users because often they feel uncomfortable with such artificial attachments @cite_7 (ii) Some of the existing systems require external power through battery @cite_22 . Therefore, contact-less vision-guided systems are preferred for such applications. Researchers have shown that Leap Motion device can be successfully used for palm rehabilitation @cite_3 , upper limb rehabilitation @cite_47 , stroke rehabilitation @cite_6 etc.