Wisture: RNN-based Learning of Wireless Signals for Gesture Recognition in Unmodified Smartphones
Mohamed Abudulaziz Ali Haseeb Ramviyas Parasuraman
cs.HC cs.LG cs.NI












Wisture: RNN-based Learning of Wireless Signals for Gesture Recognition in Unmodified Smartphones

    Mohamed Abudulaziz Ali Haseeb, Ramviyas Parasuraman
         M. Abdulaziz is affiliated with the Robotics, Perception and Learning (RPL) Lab, KTH Royal Institute of Technology, Stockholm 100 44, and with Watty AB, Sweden. 
         
R. Parasuraman is with Purdue University, West Lafayette, 47906, USA.
         
email: moaah@kth.se; ramviyas@purdue.edu
    
=====================================================================================================================================================================================================================================================================================================================================================================







This paper introduces Wisture, a new online machine learning solution for recognizing touch-less dynamic hand gestures on a smartphone. Wisture relies on the standard Wi-Fi Received Signal Strength (RSS) using a Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN),
thresholding filters and traffic induction.
Unlike other Wi-Fi based gesture recognition methods, the proposed method does not require a modification of the smartphone hardware or the operating system, and performs the gesture recognition without interfering with the normal operation of other smartphone applications. 

We discuss the characteristics of Wisture, and conduct extensive experiments to compare its performance against  state-of-the-art machine learning solutions in terms of both accuracy and time efficiency. The experiments include a set of different scenarios in terms of both spatial setup and traffic between the smartphone and Wi-Fi access points (AP). The results show that Wisture achieves an online recognition accuracy of up to 94% (average 78%) in detecting and classifying three hand gestures.




Radio Signal, Gesture recognition, Wi-Fi, Smartphones, LSTM RNN, Traffic Induction.








§ INTRODUCTION



smartphones have become the first high performance computing and sensing devices that are carried by a majority of the population in many countries.  Today, smartphones are responsible for the majority of internet search traffic <cit.> as well as online media consumption <cit.>. Consumer reports reveal that users spend more time on smartphones than they do on desktop computers <cit.>.  However, their inherent limitations in terms of physical size, screen size, computing capacity and battery power, calls for interaction modalities beyond the smartphone touch screens.
Therefore, researchers and engineers are continually looking for novel methods to enrich and simplify how humans interact with their smartphones, such as natural language processing <cit.>, and touch-less gestures <cit.>.



Modern gesture recognition methods mainly use two sensing modalities: intertial measurements <cit.> and images from the camera <cit.>.
The advantages of these modalities include availability and accuracy. However,
intertia-based systems require the smartphone to be held by the user, and image based techniques
 suffer from drawbacks such as limited sensing range, sensitivity to lighting conditions, and more importantly high-power consumption.

Therefore, in recent years,
Radio Frequency signals and Received Signal Strength (RSS) have been exploited  for sensing human activities and gestures <cit.>, mainly due to their advantages in terms of low power consumption, and being able to handle  non line-of-sight conditions. For instance, Google's project Soli <cit.> aims at creating  a rich gesture recognition interface using a specialized radar device that can be embedded into wearables, phones, computers, and Internet of Things (IoT) devices. 

The ubiquitous nature of Wi-Fi technology makes it attractive  for gesture recognition in smartphones. Thus, a number of novel methods have been proposed for using Wi-Fi RSS (e.g. <cit.>). However, they require either special hardware or special software modifications such as root privileges, blocked data traffic to other applications, etc. to operate.








In this paper, we propose a novel machine learning method based on LSTM RNN for recognizing gestures in unmodified smartphones, based on artificially induced data traffic between the smartphone and the Wi-Fi Access Point (AP). The main contributions of this paper are:


  * We demonstrate that the Wi-Fi RSS can be used to recognize hand gestures near smartphones (see Fig. <ref>) using a fusion of machine learning techniques and custom signal processing algorithms.

  * We conduct several experiments under various conditions to validate the gesture classification performance of the proposed method, and compare it against state-of-the-art machine learning methods.

  * We release the experiment dataset (an extensive collection of labeled Wi-Fi RSS measurements corresponding to multiple hand gestures made near a smartphone under different spatial and data traffic scenarios.) to enable comparison with future approaches[The dataset is available at <https://goo.gl/2AQKdT>, and will be published to the public CRAWDAD repository (<http://crawdad.org>).]. 


  * We openly share the source codes of part of the Wisture[Accessible at <https://github.com/mohaseeb/wisture>. We also release the Android apps "Wisture" and "Winiff", available in Google Play Store.], specifically to record high frequency Wi-Fi RSS measurements using the induced traffic approach.






§ RELATED WORKS


Radio signals, notably Wi-Fi signals, have been recently exploited for sensing and recognizing human activities <cit.>. For instance, customized hardware-based active sensing solution is introduced in <cit.> using transmit and receive arrays/antennas along with Fourier/Doppler analysis of the RSS data. They achieved a recognition accuracy of 94% to classify 9 whole-body gestures in a home environment. Similar custom hardware-based solutions (e.g. <cit.>) are limited to the application environments and the resources. Nevertheless, antenna-array based methods find sophisticated novel solutions like seeing through walls using radio signals <cit.>.


In <cit.>, the authors used both the Wi-Fi RSS and the Channel State Information (CSI) to recognize hand gestures using a signal conditioning and thresholding based gesture recognition algorithm, and achieved a classification accuracy of 91% on a laptop. 
Note, the CSI provides detailed channel features including the sub-carrier level phase information, but is supported only by a very limited set of Wi-Fi devices[Currently, the CSI data is available only from Intel's Wi-Fi Link 5300 drivers, and not available on smartphones Wi-Fi devices/drivers.]. Several other works such as <cit.> also proposed CSI-based solutions for gesture or activity recognition, however they are subjected to the same limitations.


K-Nearest Neighbors (K-NN) based classifiers have been widely used to recognize gestures from the RSS data. In <cit.> and <cit.>, the authors used window-based statistical features (e.g. mean, variance, maximum, number of peaks, etc.) applied to a K-NN classifier for recognizing hand gestures on a smartphone. They achieved accuracies of 50% (with K=20, 11 hand gestures) and 90% (with K=5, four hand gestures) respectively. 
However, their solutions require modified device firmware, root access to the OS, and dedicated applications that limits smartphone's Wi-Fi traffic.

A Discrete Wavelet Transform (DWT) based approach is presented in <cit.> to transform the RSS data intro three primitive signals: rising edges, falling edges and pauses. It achieved high accuracy (90%) by using a classifier that compares the series of primitive signals to a set of pre-defined rules. However, such a solution requires high-frequency sampling of RSS and extensive computation abilities.

We depart from the literature works in four different ways: (1) we do not make any modifications to the existing hardware or software applications of the phone; (2) we introduce a new traffic induction approach to enable high-frequency RSS measurements; (3) we use custom but simple signal processing techniques and an efficient LSTM-RNN machine learning method to classify over-the-air hand gestures; (4) we share the experiment datasets and the partial source codes of our solution for the community to build this research further.



To the best of our knowledge, very few works uses deep learning or neural network based methods to radio signal based activity/gesture classification. For example, the authors in <cit.> used a Convolutional Neural Network (CNN) for classifying user driving behaviors based on narrow-band radio signals and achieved 88% accuracy. On a higher level, the CNN is ideal for image processing due to its nature of recognizing patterns within the data (across space), whereas the RNN is ideal for speech/time-series processing due to its between-data (across sequence) recognition capabilities. Thus we choose RNN as the core machine learning algorithm in our Wisture solution.




§ BACKGROUND



 §.§ Radio signal propagation

An RF signal propagating through a medium is subject to several environmental factors that impact its characteristics. In the absence of nearby obstacles, the signal strength will be reduced by the free-space path loss (FSPL) caused by the spreading out of the signal energy in space. 
The RSS is usually modeled using a log-distance path loss model (an extension of Friss transmission equation) <cit.>:

    P_r(dBm) = L_0 - 10n log_10(R)_Path Loss + χ_σ_Shadowing + ε_multipath,

where L_0 is the reference RSS at 1 meter from the transmitter (which depends on transmitter and receiver antenna properties as well as the signal frequency), n is the decay exponent depending on the environment, R is the distance between the transmitter and receiver, χ_sigma = 𝒩(0,σ) is a zero-mean Gaussian variable with variance σ^2 used to represent the shadow fading caused by surrounding objects due to reflection, absorption, and diffractions <cit.>. ε is a random variable following the Nakagami distribution  used to represent multi-path fading caused by the fact that the signals take multiple paths to the destination, and to represent interferences and spatio-temporal noise <cit.>. 
The combination of χ_σ and ε models the different ways by which human body presence and movements impacts the strength of the radio signal received by a nearby wireless device, such as a smartphone.

Friis transmission equation (Eq. <ref>) describes that the received power is inversely proportional to the square of the distance between the transmitter and the receiver <cit.>. 


    P_r(W) = P_tG_rG_t(λ/4π R)^2

In the above equation, P_t and G_t are the transmitter output power and its antenna gain, P_r and G_r are the receiver input power and its antenna gain, R is the distance between the antennas and λ is the signal wavelength.






 §.§ Wi-Fi RSS measurements

Wi-Fi technology is based on the IEEE 802.11 standards and typically use 2.4 or 5 GHz ISM frequency spectrum. The data is communicated as frames. For every received frame, the device measures the RSS and reports the Received Signal Strength Indicator (RSSI), usually in dBm. 
Figure <ref> shows an illustration of a sequence of RSSI measurements generated by a Wi-Fi receiver.
The RSSI together with additional measurements (e.g Link Quality, Noise level, etc.) at the interface provide information on the transmission channel. 
The frequency by which the RSSI measurements are taken is not deterministic, but depends on the traffic sent to the Wi-Fi receiver. This burstiness nature of the RSSI measurements makes it hard to use for gesture detection. 

In an idle Wi-Fi network, the majority of the traffic is formed of so-called beacon frames. Beacon frames are sent periodically by the AP to signal the presence of the Wi-Fi network. The time between two beacon frame transmissions is configurable, but typically set to 102 milliseconds <cit.>. This means that a smartphone connected to a Wi-Fi network, and not actively receiving data, will have around nine RSSI measurements every second corresponding to the beacon frames received during that second.

Note the RSSI measurements made on a laptop or a computer with commercial Wi-Fi adapters, can provide much stronger and high-frequency data compared to measurements on a smartphone. This is likely due to the antenna properties, device drivers, and limited resources on a smartphone, which alludes to the difficulty of using Wi-Fi RSSI for recognizing gestures in smartphones compared to computers.
Figure <ref> shows how the RSSI stream recorded in a computer exhibits more and stronger RSSI variations compared to that recorded in a smartphone. These differences are likely attributed to the limited smartphone resources, which alludes to the difficulty of using Wi-Fi RSSI for recognizing gestures in smartphones compared to computers.





 §.§ Hand gestures

We first show in Fig. <ref> the RSSI data recorded in a smartphone while a person is performing an activity such as walking past the phone, typing on a keyboard, and swiping his/her hand over the phone. Note the unique pattern in the RSSI stream created by the hand gesture.

In this paper, we consider three hand gestures, see Fig. <ref>: swipe, push, and pull. 


  * Swipe gesture:  moving the hand about five centimeters above the smartphone from one side to the other and back to the starting point.

  * Push gesture: moving the hand downward towards the smartphone and holding it steadily about five centimeters above it for around two seconds.

  * Pull gesture: placing the hand about five centimeters above the smartphone, holding it  there for about two seconds before moving it upward.

If more gestures are needed one could either introduce new gestures, or make combinations of these three primitive gestures as in <cit.>.
The question of which option is best is however beyond the scope of this paper, as we focus on the best machine learning solution.



§ PROBLEM FORMULATION

The problem of recognizing hand gestures from RSS values can be viewed as a classification problem where the objective is to learn a mapping from the RSS values to the probability distribution over the possible hand gestures P(y|x).

    x → P(y|x;θ)

Where x ∈ℝ^τ is a sequence of input RSS values with length τ, y∈{0,1,…,K} is the list of gestures, θ is a parameterization of the mapping.

Given a dataset of m sample gestures that is formed from the inputs X=[x_1 x_2 … x_m] and their corresponding outputs Y=[y_1 y_2 … y_m]. The maximum likelihood (ML) method can then be used to find a good estimate of θ as below:

    θ_ML=*arg max_θ P(Y|X;θ)


Assuming the dataset samples are independent and identically distributed (i.i.d.). Equation <ref> can be rewritten as follows:

    θ_ML=*arg max_θ∏_i=1^m P(y_i|x_i;θ)

The above probability product can become very small and hence render the problem computationally unstable. This can be solved by taking the logarithm of the likelihood, which transforms the product of probabilities into a sum.


    θ_ML=*arg max_θ∑_i=1^m log P(y_i|x_i;θ)

The estimate of θ can be expressed as minimizing a loss function L (also referred to as cost) defined as below 

    L = ∑_i=1^m -log P(y_i|x_i;θ)

This loss is known as negative log-likelihood (NLL).

As will be described below, an RNN is used to model the conditional probability P(y|x;θ). An ML estimate of θ is found  using the Stochastic Gradient Descent (SGD) algorithm, to minimize the NLL in a collected hand gesture dataset.




§ PROPOSED SOLUTION: WISTURE

We first describe the Wisture solution in general and then present how to train the system with the training data.


 §.§ Signal processing and machine learning method

Figure <ref> shows an overview of the proposed gesture recognition solution. The different submodules are described below.


  §.§.§ Traffic induction

As discussed earlier, the smartphone Wi-Fi interface makes new RSSI measurement only when a new Wi-Fi frame is received. To guarantee that the wireless device makes enough updated RSSI measurements, we induce artificial traffic between the AP and the smartphone by sending a continuous stream of Internet Control Message Protocol (ICMP) echo requests to the AP. For every ICMP echo request, the AP will send an ICMP echo reply back to the smartphone which will make an updated RSSI measurement. 
This enables us to have enough RSSI measurements while avoiding the need for a custom firmware or putting the Wi-Fi interface in monitor mode as in <cit.>.



  §.§.§ RSSI collection

This module extracts a stream of up to ∼200 RSSI values/second from the Wi-Fi interface. In our Android implementation, the RSSI measurements are collected using the wireless extension for Linux user interface <cit.>, which is exposed as a pseudo file named /proc/net/wireless.



  §.§.§ Windowing

The incoming RSSI stream is split into overlapping windows of T seconds length, and d seconds gaps between window starts. In all the experiments the gap d was set to 1 second. Different values of T are investigated and reported in the experiments section. Since the incoming RSSI stream rate is varying around 200 values per second, the output windows will have a variable length.



  §.§.§ Noise detection

We note that only windows with high activity, identified by a theshold on the window variance, are likely to be caused by hand gestures. Thus only the windows that pass this criteria will be forwarded to the subsequent steps. All windows that have a variance less than the threshold will be predicted as no gesture or Noise. The process of estimating the variance threshold is described in Sec. <ref>.



  §.§.§ Preprocessing

This submodule transforms the incoming windows into windows of equal number of feature values (τ). Each incoming window is processed as below:


  * Mean subtraction: The window values are centered around zero, by subtracting the window mean from all RSSI values in the window. This increases the system robustness against changes in the RSSI values due to, for example, RSSI increases or decreases when the smartphone is moved close or away from the AP respectively.

  * Sampling: This steps samples τ feature values with a time difference between consecutive samples equal to T/τ on average. Different values of τ have been investigated and reported in the experiments section.

  * Standardizing: Each one of the τ feature values is reduced by the training data mean of that feature value. 

  * Normalizing: Each one of the τ feature values (standardized in the previous step) is divided by the training data standard deviation of that feature value.

A detailed description of the standardizing and normalizing steps is provided in Sec. <ref>.



  §.§.§ Inference

The LSTM RNN model takes an input of τ feature values, and outputs three values proportional to the conditional probability of each possible gesture on the inputs. These values are referred to as logits, because they are the inputs of the softmax layer used in calculating the model loss during training. The Softmax function is a generalization of the logistic function, and its inputs are referred to as logits.



  §.§.§ Logits thresholding

This step discards LSTM RNN model predictions that are below a specific threshold and predicts Noise for those inputs.



  §.§.§ Prediction decision rules

This submodule keeps a short history of the previous predictions, and applies a set of rules to accept or reject the current prediction made by the preceding steps. These rules are:


  * Allow Pull gestures only after Push gestures. The Pull gesture RSSI signature appears as a pause followed by an increase. This signature is similar to those caused by some background activities, e.g. when an AP increases its output signal power. This rule reduces the number of false positive Pull predictions caused by such interfering background activities. 

Note: However, the solution is prone to confusing decreases in RSSI values caused by interfering background activities (e.g. the AP reducing its output signal power) as Push gesture, and no solution was proposed in this work to harden the system against such interference.

  * A prediction that is different from its immediate predecessor is ignored (and Noise is predicted instead). Exempted from this rule are: 


  * Swipe or Push following a Noise prediction.

  * Pull prediction that follows a Push.

  * Noise predictions.

The rationale for this rule is that each prediction window overlaps with the previous window (three seconds overlap in most experiments). In many cases, if the preceding window contained a gesture, the succeeding window RSSI stream signature might look similar to another gesture than the performed one. For example, the end of Swipe gestures look similar to Pull gestures (see Fig. <ref>).




 §.§ System training

Training the LSTM RNN model and selecting the different hyperparameters and thresholds are performed in an offline setting. The training procedure is detailed below.



  §.§.§ Data preprocessing




The online machine learning solution preprocesses the incoming RSSI windows in the way described below.


  * The RSSI values are read from the collected data files, and then split into D windows (corresponding to the gestures), each being T seconds long.

  * For each window, the mean is calculated and then subtracted from the individual window values.

  * τ values that are equally spaced in time are then sampled from each window. The result is a dataset of shape D windows each having τ features.

  * The dataset is then randomly split into training (D_train=0.75D) and testing (D_test=0.25D) sets. Furthermore, when a model hyper parameter selection is done, 0.8D_train of the training set is used to train the model, and the remaining D_val=0.20D_train is used to select the hyper parameters (validation set).

  * Using the training set (D_train×τ), the mean and standard deviation of each one of the τ features is calculated as below.

    x^(i)_train_mean=1/M∑_j=1^M x^(i)_train, j


    x^(i)_train_std=√(1/M∑_j=1^M (x^(i)_train, j - x^(i)_train_mean)^2)

Where M is the training set D_train size and x^(i)_train, j is feature i value of sample j from the training set.

  * All training and testing set windows were standardized and normalized using the training mean and standard deviation. Let x=[x^(1) x^(2)… x^(i)… x^(τ)] be some input window (from training or testing), the output of the standardization and normalization steps x_o=[x^(1)_o x^(2)_o … x^(i)_o … x^(τ)_o] can be described as below:

    x^(i)_o= x^(i)- x^(i)_train_mean/x^(i)_train_std





  §.§.§ LSTM RNN model training

Figure <ref> shows an illustration of the LSTM RNN model used in this work (inspired by <cit.>). The model was trained to minimize the NLL loss, using a variant of SGD known as Adaptive Moment Estimation, or shortly ADAM.

Most of the model hyper parameters (τ, N, number of layers and others) were selected by performing a grid search in the parameters space. Each parameter setting is evaluated using a four folds cross validation.



  §.§.§ Thresholds selection

The variance threshold used by the Noise detection step, is initially estimated as the training data minimum  window variance. This value is then manually optimized to maximize the online prediction accuracy. The same approach is used to select the logits thresholds.




§ EXPERIMENTS AND RESULTS



 §.§ Experiment Dataset

This section contains the details of the dataset used to train and evaluate the recognition system.



  §.§.§ Spacial setup

Figure <ref> presents the experimental setup. The dataset was collected under two different spatial configurations of the Wi-Fi AP and the smartphone:


  * (Room A): The AP and the smartphone were placed two meters apart in room A with a line of sight (LoS) between them. The AP was placed on a table slightly lower than the table where the smartphone was placed.

  * (Room A & B): The AP was placed in room B and the smartphone in room A, see Fig. <ref>, and the distance between them was ∼4.5 meters. The two rooms were separated by a wall made mainly of wood and gypsum, thus there is no line of sight (nLoS). Both the AP and the smartphone were placed on tables of similar height. 






  §.§.§ Traffic scenarios

Three different WiFi traffic scenarios were considered when collecting the data:


  * (Internet access + traffic induction): in this scenario the AP was connected to the Internet and  the smartphone was connected to the AP. The smartphone was continuously sending ICMP requests to the AP (pinging) at a rate of ∼700 times/second.

  * (No Internet access + traffic induction): neither the AP nor the smartphone had Internet access, but the smartphone was continuously pinging the AP at a rate of ∼700 times/second.

  * (No Internet access + no traffic induction): neither the AP nor the smartphone had Internet access, and there was no traffic induction. 




  §.§.§ Data collection procedure

An Android mobile application was developed specifically for recording the Wi-Fi RSSI data using the induced traffic approach (called "Winiff"). It records the RSSI measurements made by the smartphone at a frequency of ∼200 samples/second.
A typical collection session is described below:


  * The AP and the smartphone are placed as per one of the spatial setups described earlier.

  * The subject performing the experiment sits in a chair facing the smartphone.

  * The smartphone is connected to the AP.

  * The RSSI collection application is started.

  * At a specific point in time (start time), the subject starts performing the gestures, leaving a gap of ten seconds between consecutive gestures (gap time). Both the start and gap times are noted down and used later to extract the gesture windows.

  * The collected RSSI stream is stored in a text file.




The collected dataset details are summarized in Table <ref>.





  §.§.§ Evaluation metric

The system was evaluated using the accuracy measure, defined as the percentage of correctly predicted gestures. For a set of test gestures X=[x_1 x_2… x_i … x_m] with corresponding true labels Y=[y_1 y_2 … y_i … y_m], the accuracy is defined as:

    accuracy = 100 ×1/m∑_i=1^m I_y_i(ŷ_̂î)    %
 
Where ŷ_̂î is the system prediction for input gesture x_i, and I_y_i(ŷ_̂î) is 1 if ŷ_̂î = y_i and 0 otherwise.



 §.§ Results and Discussions

Here we list the conducted experiments and the obtained results. The RNN model was trained as described in Sec. <ref>. The model parameters used in the experiments are shown in Table <ref>.
The reported mean accuracies in  experiments are calculated by evaluating the RNN model ten times on the specific configurations being tested, each using a different random split of the data into training and testing sets.




  §.§.§ RNN model accuracy on different datasets

Table <ref> lists the recognition model accuracy when evaluated on the different datasets. 
Due to the increased data size of the collective dataset (Dataset1 + Dataset2 + Dataset4), the number of training iterations for this configuration is 1000 instead of 600.

If we exclude the Swipe gesture in Dataset3, the prediction accuracy jumps to 97% (±1.5).
The gesture recognition accuracy is poor for Dataset3, which can be attributed to fact that only a small number of (low-frequency) RSSI measurements were available in the dataset because both the internet and the traffic induction were disabled. The significant increase in accuracy from 78% to 91% when the induction is enabled demonstrates the high dependency of the accuracy on the amount of data traffic at the Wi-Fi interface.

Figure <ref> compares the RSSI values recorded for the Swipe and Push gestures performed while the induction is enabled and disabled. 

It is clear that with no traffic induction, the RSSI values corresponding to the Swipe gesture are not distinguishable from those of the Push gesture, and consequently the classification ability is severely impacted.




  §.§.§ Comparison with the state-of-the-art methods

A set of time series classification algorithms, including state-of-the-art ones, were evaluated on Dataset1, to compare them to the RNN model (Table <ref>). The Collective of Transformation Ensembles (COTE), Elastic Ensemble (EE), Shapelet Transform ensemble (STE) <cit.>, which are ensemble methods that employs multiple classifiers underneath, performed better than or equal to the RNN model. The Learning Time-series Shapelets (LTS)[We implemented a python version of the LTS method, available at <https://github.com/mohaseeb/shaplets-python>.] <cit.> performed better in terms of accuracy but poorer in terms of training/prediction time.. The K-Nearest Neighbor Dynamic Time Warp (K-NN DTW) algorithm and the Fast Shapelets (FS) <cit.> performed worse, with the k-NN DTW having a very slow prediction time of almost one second.




  §.§.§ Impact of dataset size on the accuracy

Figure <ref> shows the model accuracies when trained with different dataset sizes. It shows that the model accuracy increases and becomes more stable (reduced variance) as it is trained with more data.




  §.§.§ Impact of model parameters on the accuracy



  
Model complexity
As can be seen in Fig. <ref>, the experiments show that increasing the number of layers initially increases the model accuracy, but then decreases it, most likely due to overfitting.




  
Samples per window
The results in Fig. <ref> shows that the model accuracy increases as the number of samples per window increases, but after a specific point the accuracy decreases. This decrease in accuracy can be explained by the increase in model complexity as the number of input samples increases (and hence increases the RNN time steps) which causes the model to overfit the training data.




  
Window size
In Fig. <ref>, the impact of the prediction window length on the RNN model accuracy with the Dataset1 is shown. The model achieves the highest accuracy for the two seconds window length. However, online experiments showed that a two seconds window length provides a short context that results in a higher rate of confusion between gestures, see Fig. <ref>, and false positive predictions. Thus, a four seconds window is used in the online experiments.



  §.§.§ Impact of spacial setup on the accuracy

Table <ref> and Fig. <ref> summarize the various online experiments performed[Experiments made using the Wisture Android App. The RNN model had the same parameters as in Table <ref>, except no dropout was used due to a limitation in the Tensorflow Android library.] and the obtained results[A video demonstration of sample results is available at <https://www.youtube.com/watch?v=rMv_bKkDtbU>.]. Accordingly it can be concluded that the solution: (1) generalizes to AP-smartphone spatial configurations that are different from the training ones, and (2) performs better in settings where there is a line-of-sight (LoS) between the smatphone and the AP (81% accuracy), than where there is no LoS (74% accuracy). The exception to this is the no-LoS online experiment number three, where an accuracy of 93% was achieved, which is higher than all recorded LoS experiments accuracies. This might be due to the peculiarity of the Wi-Fi signal path between the smartphone and the AP in these cases. Refer to Sec. <ref> for more information on RF signal propagation.

Figure <ref> shows that the system has approximately equal performance in recognizing the different gestures. Although Push has the highest average accuracy, the standard deviation (std) of the individual gestures show that, Push and Pull accuracies have a wider variation across the different test scenarios (0.29 std for Push and 0.3 std for Pull) compared to the Swipe gesture (0.15 std). 

We believe the low accuracy achieved in experiment six is because of the fact that the AP and the smartphone were far apart (9 m), and the line-of-sight was blocked by a wall, a dishwasher and a fridge.





  §.§.§ False positive predictions

To estimate the robustness of the system against RSSI changes caused by interfering background activities, the recognition application was left running for a period of thirty minutes on a table inside a room, while a person was in the same room typing on a computer (placed on the same table as the smartphone), and occasionally moving in the room. Table <ref> summarizes the results, which show an average false positive rate of 8%.




  §.§.§ Resource consumption

The  Wisture app was found to utilize around ∼13% of the total CPU time (9% system time + 4% user time) when traffic induction is disabled. The utilization increased to ∼25% (13% system time + 12% user time) when induction is enabled. Note, the screen usage of the app are also counted into the resources (the screen was always on display to observe the experiment results). No CPU or power usage figures were reported by previous works.



 §.§ Summary

Using the proposed recognition solution, the experiments showed that it is possible to detect and classify three different hand gestures with an accuracy of 78%, across a variety of spatial and traffic scenarios without modifying the smartphone hardware, operating system or firmware (on the contrary to the works in <cit.>[In <cit.>, the wireless interface was in the "Monitor" mode, and hence captured all Wi-Fi traffic exchanged by all smartphones and APs in the smartphone vicinity. As a result, the available RSSI measurements increased and the solution achieved a high accuracy ( 90%). However, in addition to the need of a custom firmware, the "Monitor" mode comes at the cost of prohibiting all smartphone applications from sending or receiving traffic over Wi-Fi, and a likely increase in battery power consumption.], <cit.>,<cit.>, and <cit.>). To the best of our knowledge, the work presented here is the first to demonstrate this. Note, although the works in <cit.> demonstrated gesture recognition on unmodified devices, the solutions are tailored to a PC based implementation with higher resources than a smartphone[In <cit.>, CSI data (currently available on Intel's Wi-Fi Link 5300 device driver) is used to identify hand gestures. In <cit.>, the RSS collection procedure is not described in detail.].



§ LIMITATIONS AND FUTURE WORK



 §.§ Limitations



  §.§.§ Background activities

The solution was sensitive to interference to background activities, but we did not analyze such impacts. They may affect the prediction performance in general, and false positives specifically. 





  §.§.§ New gestures

To support new gestures, a new dataset for the new gesture is needed. The developed recognition solution is trained with a dataset of 1000+ sample gestures (300+ samples per gesture). The requirement of huge number of training samples can be seen as a limitation. However, we can see in Fig. <ref> that the training data size can be reduced  to a tenth of the original set (100+ samples), with a modest decrease in accuracy to 83% from 94%. 




  §.§.§ Calibration to different Smartphones or devices

Since the solution is based on training data from one particular smartphone, using the trained RNN on a different smartphone may result in a lower accuracy. To address this issue, further work is necessary to calibrate the training data to new smartphones or other devices (laptop, tablet, or a wearable device).



 §.§ Lessons learned


Below we list some of the design and implementation choices that were discarded since they had negative or no impact on the model accuracy.



  
Wi-Fi Link Quality Indicator (LQI)
The initial exploration of the collected RSSI and LQI data showed a high correlation between the two, and neither provided additional information that is not present in the other. Consequently, we did not use the LQI data as an extra input signal.



  
Smoothing the RSS data
Exponential and moving window based RSSI smoothing techniques affected the LSTM-RNN classification accuracy negatively. We believe that the smoothing process removed a few important characteristics of the RSS signal variations, however further investigation is needed to analyze the impact.



  
Smartphone Wi-Fi interfaces
Our experiments showed that the model ability to recognize more gestures can be improved by measuring the Wi-Fi RSS at higher frequencies. Designing smartphone Wi-Fi interfaces that support improved radio signals sensing has a great potential for smartphone vendors and users. We believe the smartphone device manufacturers provide such capabilities in the future.





 §.§ Future work


Leaving the recognition application running for a long time while no gesture is performed, increases the probability of false positive predictions and consumes the smartphone battery. This can be addressed by introducing a new mode of operation in which the recognition application waits for a special preamble gesture that has two characteristics: (1) easy to separate from background noise; (2) requires small resource utilization to be detected. After a preamble detection mode, the application could detect further complex gestures. 

In our further works, we plan to use the preamble gesture concept, for instance with a sequence of Push and Pull gesture as a preamble. 
The results in Table <ref> show that Push and Pull gestures can be recognized with an accuracy of 97% even when no traffic is induced between the smartphone and the AP. Thus, during the preamble detection operation mode, traffic induction can be disabled, significantly reducing the power consumption and resource utilization of the application. Further, we will also investigate the influence of the distance and spatial characteristics between the smartphone and the hand making the gestures.



 §.§ Improved sensing of Wi-Fi RSS

A high resolution RSSI stream will contain features that increase the models ability to detect and classify an increased number of gestures with high accuracy, see Sec. <ref> and Fig. <ref>.

Designing the smartphone Wi-Fi system with the goal of improving the radio signal sensing by: 1) allowing RSSI measurements at high frequency, 2) native support for traffic induction and 3) providing simplified access to the RSSI values, has several benefits. It will enable  descent gesture recognition capabilities that are superior to what is demonstrated in this work, at a lower cost and with a shorter time-to-market compared to other approaches, like designing specialized sensing hardware for detecting gestures, such as the Google Soli project <cit.>.




§ CONCLUSION
 
In this work, we have demonstrated  that it is possible to recognize and classify contact-less moving hand gestures near smartphones without modification to the smartphone hardware or existing software. The proposed solution used a custom signal processing techniques, an artificial traffic induction approach, and a LSTM RNN based machine learning model to detect and classify the performed hand gesture from the smartphone Wi-Fi RSSI measurements. The solution achieved an average on-phone recognition accuracy of 78% on average, and up to 94% on specific datasets, when tested under several configurations (scenarios) including the ones that were different from the training data scenarios. We believe this accuracy qualifies the solution for non mission-critical applications.



In our future works, we aim to reduce the limitations of the solution such as vulnerability to interfering background activities (for instance by introducing a preamble detection mode), and calibrating the training data to new devices/gestures.



§ ACKNOWLEDGMENT

We would like to thank Petter Ögren for his invaluable input and advice on this work.

IEEEtran
	

