©2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers
or lists, or reuse of any copyrighted component of this work in other works.
To appear in the Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN 2017)

arXiv:1705.02643v1 [cs.LG] 7 May 2017

DropIn: Making Reservoir Computing Neural
Networks Robust to Missing Inputs by Dropout
Davide Bacciu and Francesco Crecchi

Davide Morelli

Dipartimento di Informatica
Università di Pisa
Largo B. Pontecorvo 3
Pisa, Italy
Email: bacciu@di.unipi.it

Biobeats Ltd
London, UK
Email: davide@biobeats.com

Abstract—The paper presents a novel, principled approach to
train recurrent neural networks from the Reservoir Computing
family that are robust to missing part of the input features
at prediction time. By building on the ensembling properties
of Dropout regularization, we propose a methodology, named
DropIn, which efficiently trains a neural model as a committee
machine of subnetworks, each capable of predicting with a
subset of the original input features. We discuss the application of the DropIn methodology in the context of Reservoir
Computing models and targeting applications characterized by
input sources that are unreliable or prone to be disconnected,
such as in pervasive wireless sensor networks and ambient
intelligence. We provide an experimental assessment using realworld data from such application domains, showing how the
Dropin methodology allows to maintain predictive performances
comparable to those of a model without missing features, even
when 20%-50% of the inputs are not available.

I. I NTRODUCTION
The increasing diffusion of networks of pervasively distributed environmental and personal sensor devices requires
computational models capable of dealing with continuous
streams of sensor data under the form of time series of
measurements. Machine learning models, in this context,
serve to make sense of such multivariate sequences of heterogeneous sensor information by providing predictions supporting context-awareness and ambient intelligence functions.
Numerous applications have been developed by modeling
them as regression and classification tasks on sensor streams,
including event recognition, fault and anomaly detection, human activity recognition and, in general, supporting robotic
and Intenet-of-Things (IoT) applications by providing adaptivity and context awareness mechanisms.
Recurrent Neural Networks (RNN) are a popular and
effective means to deal with sequential information thanks
to their ability in encoding the history of the past inputs
within the network state. However, these networks, as well
as their feedforward counterpart, require a set of fixed input
features to be available both at training as well as at a
test time. Their predictive performance tends to abruptly
decline when one or more of the input features on which
it has been trained is missing when querying the model for
predictions [1]. Many practical ubiquitous computing and

IoT scenarios comprise information generated by networks of
loosely connected devices, which are often battery-operated
and communicating through wireless channels with little
quality of service guarantees. In such scenarios, it is not
unlikely to have to deal with missing information, which
might result in a RNN performing recognition or prediction
tasks missing a part of its inputs.
The goal of this work is to propose, and to experimentally
assess, a principled approach to make neural networks robust
to such missing inputs, at query time (i.e. at prediction),
focusing in particular on efficient RNN models from the
Reservoir Computing (RC) [2] paradigm. The large majority
of the works in literature addresses the problem of missing
inputs solely with respect to information that is missing at
training time. In this context, dealing with missing information amounts to finding the best strategies to impute values
of certain features which are missing in some of the training
samples: [3] provides a recent survey on this problem. In
this work, instead, we consider a scenario in which we
are able to train a RNN using complete data (i.e. with no
missing inputs) but where part of the inputs may become
unavailable, even for a long time, during system operation
(e.g. due to a sensor failure). One of the typical approaches to
deal with this problem exploits imputation, i.e. substituting
the missing feature by inferring it from other information.
In [4], for instance, a spatial-temporal replacement scheme
is proposed for a fuzzy Adaptive Resonance Theory (ART)
neural network used for anomaly detection over a Wireless
Sensor Network (WSN). Here missing input information
related to a WSN node is imputed based on a majority voting
between the readings of the devices that are spatially closer
to the failing one. Similarly, [5] uses a k-Nearest Neighbor
(kNN) to replace missing sensor values with spatially and
temporally correlated sensor values. This approach has a
major drawback in the fact that it assumes that sensor devices
are of homogenous type and that it is possible to find
a spatially/temporally related sensor, of the same type as
the failing one, to replace its measurements. An alternative
approach to the problem is based on fitting a probability
distribution of the missing features given the observable
inputs, e.g. by using a Parzen window estimator [6], and to

use it to sample replacement measurements for the missing
information [7], [8]. In [9], it is proposed a discriminative
solution that uses a RNN to model static (i.e. non sequential)
problems while recurrent layers are used to estimate the
values of the missing features. A rather different approach to
information imputation uses committee machines or model
ensembles [1]: these models accommodate data with missing
features by training an ensemble of predictors with random
subsets of the total number of available features.
None of the approaches described above attempt to make
the original model robust to missing information. They either
impute the missing information or enrich the original model
by training multiple instances of it with different input
configurations, with considerable impact on the training time
as the number of input features increases. We propose a novel
solution to the missing input problem that exploits dropout
[10], a regularization technique that has become widely
popular in the context of deep learning in the latter years.
Dropout prevents overfitting by randomly selecting, during
training, a subset of neurons or synapses which are dropped
(i.e. their activation is zeroed) for the current training sample.
The key intuition guiding this work is that one of the effects
of Dropout is to train the full neural model as if it were an
ensemble of thinned models obtained by the random dropout
disconnections. As such, we believe that it can be exploited
to build a model robust to missing inputs as an ensemble
of thinned models obtained by applying dropout to the input
neurons and connections alone.
In the following, we discuss an application of such a
technique to Reservoir Computing (RC) [2], a family of
efficient RNN models which is quite popular in applications
dealing with sensor streams due to its robustness to noise
and computational efficiency, e.g. see [11] for an embedded
implementation on low-power WSN devices. RC networks
are based on the separation between a recurrent dynamical
layer, i.e. the reservoir, and a non-recurrent output layer,
i.e. the readout. In particular, we will focus on the Echo
State Network (ESN) model [12] where only the output
layer connections are trained while the input and recurrent
weights are left randomly initialized. Note that this is the
first application of the dropout techniques to RC models
and it is also the first attempt to make these networks
robust to missing inputs. The choice of ESN as reference
model is motivated, on the one hand, by its relevance and
popularity in ambient intelligence, pervasive computing and
IoT applications. On the other hand, we would like to assess
the effect of our dropout approach, referred to as DropIn
from here onwards, on the most challenging conditions, i.e.
where only the output connections are adaptive to see if the
beneficial effect of the input dropout propagates to them.
Nevertheless, the proposed DropIn technique can be applied
to any recurrent or feedforward neural network, as in the
standard dropout technique.
The remainder of the paper is organized as follows:
Section II provides a brief background on dropout regularization and on the reservoir computing paradigm; Section

III describes DropIn application to ESN, while Section IV
provides an experimental assessment of the proposed model
on two real-world datasets from ambient intelligence and
pervasive computing applications. Section V concludes the
paper.
II. BACKGROUND
In this section, we provide a brief overview of the Dropout
approach, in Section II-A, followed by an introduction to RC
and ESN models in Section II-B.
A. Dropout Regularization
Dropout is a regularization technique for fully connected
neural network layers with adaptive synaptic weights [10].
The key intuition of Dropout is to prevent units from coadapting by randomly disconnecting a subset of them from
the network during training. Dropout is also though to act
as a model combination, or ensembling technique such that
it can efficiently combine an exponential number of neural
network architectures corresponding to the thinned networks
obtained by randomly dropping the neuron subsets.
Dropout training works by randomly determining which
units are kept at a given training instant; each unit is retained
with a fixed probability p, independently from other units, or
is dropped with probability 1 − p. The retention probability
p is typically determined by validation. A dropped out unit
is temporarily removed from the network, along with all its
incoming and outgoing connections: in practice, this amounts
to zeroing the dropped unit output. In this respect, a neural
network with N units is an ensemble of 2N thinned networks
whose total number of parameters is still bound by O(N 2 )
since they all share the weights. At test time, it is not feasible
to reproduce the network thinning process to compute the
ensemble prediction from all the 2N models. However, [10]
shows how this can be efficiently approximated by using
the original un-thinned network with weights scaled by the
retention probability p.
The Dropout approach has been later generalized by the
DropConnect model [13]. Instead of dropping out single
units, DropConnect sets a random subsets of the network
weights to zero, such that each unit receives input from
a portion of neurons in the previous layer. To do so, it
exploits, again, a retention probability p which is independently applied to the single elements of the neural network
connectivity matrix. This is shown to enhance the ensembling
effect with respect to Dropout, as the number of thinned
network in DropConnect is O(2|M | ), where M is the matrix
of randomly selected zero and ones used for weight masking
[13].
B. Reservoir Computing
Reservoir Computing [2] is a RNN paradigm based on
the separation between the recurrent part of the network, the
reservoir, from the feedforward neurons, including the output
layer referred to as readout. The reservoir encodes the history
of the input signals. The activations of its neurons (i.e. the
network state) are combined by the readout layer to compute

the network predictions. One of the key aspect of RC is its
training efficiency as the readout layer is the only trained
part of the network, whereas the input and reservoir connections are randomly initialized, under conditions ensuring
contractivity [14], and then left untrained.
Among the different RC approaches, here we focus on the
popular ESN model [12] due to its suitability to the ambient
intelligence applications considered in this work. An ESN
comprises an input layer with NU units, a reservoir layer
with NR units and a readout with NY units. The reservoir
is a large, sparsely-connected layer of recurrent non-linear
units (typically tanh) which is used to perform a contractive
encoding [14] of the history of driving input signals into
a state space. The readout comprises feedforward linear
neurons computing ESN predictions as a weighted linear
combination of the reservoir activations. In this work, we
focus on the leaky integrator ESN (LI-ESN), a variant of the
standard ESN model which applies an exponential moving
average to the reservoir state space values. This allows a
better handling of input sequences that change slowly with
respect to sampling frequency [2] and it has been shown
to work best, in practice, when dealing with sensor data
streams [11]. At each time step t, the activation the reservoir
activation of a LI-ESN is computed by
x(t) = (1 − a)x(t − 1) + af (Win u(t) + Wh x(t − 1)) (1)
where u(t) is the vector of NU inputs at time t, Win is
the NR × NU input-to-reservoir weight matrix, Wh is the
NR × NR recurrent reservoir weight matrix and f is the
component-wise reservoir activation function. The term a ∈
[0, 1] is a leaking rate which controls the speed of LI-ESN
state dynamics, with larger values denoting faster dynamics.
Reservoir parameters are left untrained after a random
initialization subject to the constraints given by the so called
Echo State Property (ESP) [14], requiring that network state
asymptotically depends on the driving input signal and any
dependency on initial conditions is progressively lost. In [14],
it is provided a necessary and a sufficient condition for the
ESP. The sufficient condition states that the largest singular
value of the reservoir weight matrix must be less than 1.
For a LI-ESN model, such condition applies to the leaky
integrated matrix
W̃ = (1 − a)I + aWh ,
such that
σ(W̃) < 1

(2)

where σ(W̃) is the largest singular value of W̃. The necessary condition [14], on the other hand, says that if the
spectral radius ρ(W̃), i.e. the largest absolute eigenvalue
of the matrix W̃, is larger than 1, the network as an
asymptotically unstable null states and hence lacks the ESP.
The sufficient condition in (2) is considered by to be too
restrictive [14] for practical purposes. Instead, the W̃ matrix
is often initialized to satisfy the necessary condition, i.e.
ρ(W̃) < 1,

(3)

with values of the spectral radius that are, typically, close to
the stability threshold 1. Input weights are randomly chosen
from a uniform distribution over [−sin , sin ] (where sin is
an input scaling parameter), while Wh is typically from a
uniform distribution in [−1, 1] and then scaled so that Eq. 3
holds.
The LI-ESN output is computed by the readout through
the linear combination
y(t) = Wout x(t)

(4)

where Wout is the NY × NR reservoir-to-readout weight
matrix. The readout output can be computed for each time
step t or only for a subset of them depending on the
application (e.g. for sequence classification only the output
corresponding to the last sequence item is computed, typically). Training of an ESN model amounts to learning the
values of the Wout matrix which implies the solution of a
linear least squares minimization problem: this is typically
achieved by efficient batch linear methods such as MoorePenrose pseudo-inversion and ridge regression [2].
III. D ROP I N R ESERVOIR C OMPUTING
We describe a novel use of the Dropout technique as
a principled approach to make neural networks robust to
missing inputs at prediction time. We name this approach
DropIn as it is based on the use of unit-wise Dropout [10] at
the level of the input neurons of the network. The rationale
inspiring our work is the observation that the application of
Dropout to NU input units is essentially training a committee
machine of 2NU thinned networks with shared weights. In
this sense, our approach recalls the committee machine by
[1]. Differently from this work, our method does not require
to define a specific committee machine architecture nor it
induces any increase in the parametrization of the original
model. The choice of using unit-wise Dropout [10] in place
of the connection-wise DropConnect [13] is motivated by
the fact that the latter approach would not make the neural
network robust to losing one or more of its inputs; rather,
it would make the network somewhat less focused on the
exploitation of a specific input which, in principle, can be
very discriminant for the task. Some preliminary experiments
performed with the DropConnect approach (not reported in
this paper for brevity) confirm our intuition that it performs
significantly worse than Dropout on our tasks.
The DropIn approach is general, as it can be applied
to any neural network model for which Dropout applies.
Here, we focus on the use of DropIn in the context of RC
and, in particular, with the ESN model. The motivation for
such a choice is twofold. First, RC models are a popular
approach in ambient intelligence, pervasive computing and
IoT applications [11], [15], [16], that are areas where one
has to deal with input information collected by low-fidelity
devices (e.g. battery operated) and transmitted over loose
communication channels, thus involving an high likelihood
of faults and missing data [5]. The second motivation is
more related to the exploration of the effect of DropIn in a

Fig. 1. ESN model with DropIn application: the input masking vector M is
generated each time a training sequence is fed to the network. The decision
to retain an input unit (i.e. value 1 in vector M) is taken independently for
each feature by random sampling from a uniform probability with retention
probability p.

learning paradigm which is, perhaps, the least straightforward
in terms of Dropout application. The characteristics of RC,
in fact, makes it benefit poorly from Dropout in terms of
regularization, as only a minor part of the network weights
are trained. In fact, to the extent of our knowledge, this is
the first work in which Dropout techniques are being used in
RC. Here, we seek to determine if the application of Dropout
to inputs with untrained weights has an effect propagating
throughout the untrained reservoir all the way to the readout,
which is the only trained part of the network and hence the
only part where the effect of DropIn can be recorded.
The application of DropIn to an ESN requires it to be
trained through an iterative learning algorithm. In fact, in order to make the network robust to missing inputs, we need the
ESN to process the single training sequences multiple times,
each time with a fixed probability 1 − p of independently
missing the single inputs, where p is a retention probability
as in Dropout. As discusses in Section II-B, training an ESN
entails solving a least squares minimization problem to find
the readout weights Wout . The simplest approach would
tackle this problem as a gradient descent in the direction
of minimizing the instantaneous squared error (at time t)
ky∗ (t)−y(t)k2 , where y ∗ (t) is the desired prediction at time
t. Alternatively, the Recursive Least Squares (RLS) algorithm
has been proposed in [12] as a fast online learning approach.
At each time step, RLS minimizes the discounted error
E(y, y∗ ) =

T
1 X T −t ∗
λ
ky (t) − y(t)k2
NY t=1

(5)

where 0 ≤ λ ≤ 1 is an error forgetting factor.
Our DropIn-ESN model embeds input units dropout within
the steps of incremental RLS learning. A schematic view of
the procedure is provided by the pseudo-code in Algorithm 1
for the ESN architecture depicted in Fig. 1. Assuming we are
provided with a properly initialized ESN (see Section II-B),
the procedure runs for several epochs through the dataset
(each time reshuffling the order of sequences) until an error
stability criterion is met (e.g. early stopping on validation
error) or the maximum number of epochs is reached. The
DropIn mask M in Fig. 1 is computed before processing
a whole sequence, hence the mask is the same for all the
elements of the sequence (see the first operation in the outer

Algorithm 1 DropIn Training of an ESN by RLS
Require: A dataset of L pairs of input-output sequences
∗
(u1 , y1∗ ), . . . , (uL , yL
); a properly initialized ESN model;
a forgetting factor λ; a regularization term δ and an input
unit retention probability p.
Init S−1 (0) = δ −1 · I
n=0
while learning not converged do
Shuffle sequence order
for i = 1 to L do
Compute masked inputs vector M using p
for t = 1 to Tj do
n=n+1
Mask current inputs ui (t) using M
Compute reservoir activation using (1)
Compute readout activation using (4)
Compute error e(n) = (yi∗ (t) − yi (t))
Φ(n) = S −1 (n − 1) · xi (t)
K(n) = Φ(n) · (λ + Φ(n) · xi (t))−1
S−1 (n) = λ−1 (S−1 (n − 1) − K(n) · Φ(n))
Wout (n) = Wout (n − 1) + (K(n) · e(n))T
end for
end for
end while
return Wout

for loop in Algorithm 1). This ensures that the readout
weights are modified to take into account reservoir activations corresponding to certain inputs being missing. Clearly,
the same training sequence can be expected to be processed
again in the next epochs, but this time with different input
features missing. As in standard Dropout, the decision to
drop an input unit is taken independently for each input
feature with probability (1 − p). Input masking can then be
easily obtained by zeroing the elements of current input ui (t)
using the masking vector M. The equations in the innermost
loop of Algorithm 1 implement the standard RLS algorithm
for ESN, whose details can be found in [12]. The DropInESN model in Algorithm 1 has been implemented in Matlab
[17]: the reference source code used for the experimental
assessment in Section IV can be found here1 .
Once an ESN model has been trained using DropIn, it can
be used for prediction without any weight re-scaling. Note
that this is different from how standard Dropout works. In a
DropIn-ESN, in fact, masking of the input units does not bear
any effect on the input-to-reservoir weights, whose values are
frozen to their random initialization. Readout weights are
the only ones that are affected by DropIn but the effect on
them is indirect and mediated by the reservoir activations.
Preliminary experiments on the effect of weight re-scaling
(not reported here) show that it negatively affects predictive
performance. The reasons underlying such effect deserve
further studies: possibly, the fact that the DropIn networks
do not require re-scaling might be associated with the fact
1 https://github.com/FrancescoCrecchi/DropIn-ESN

that at testing time we are simulating missing inputs, hence
allowing the network to work under conditions similar to
those of the dropout training phase.

label is provided only for the last element of each training
sequence and it thus predicted only for the last time step of
each test sequence.

IV. E XPERIMENTAL R ESULTS

The second dataset, referred to as Kitchen Cleaning [18],
concerns an Ambient Assisted Living scenario where a
cleaning robot operates in a home environment located in
the Ängen senior residence facilities in Örebro Universitet.
In this task, the learning model is required to learn to predict
a user preference that was not modelled in the robotic planner
domain knowledge, that is the fact of not having the robot
cleaning the kitchen when the user is in. This task is part of a
larger effort to assess the self-adaptation abilities of a robotic
ecology planner developed as part of the RUBICON project,
including also automated feature selection mechanisms on
time series [19]. The experimental scenario consists of a
real-world flat sensorized by an RFID floor, a mobile robot
with range-finder localization and a WSN with six mote-class
devices. Each device is equipped with light, temperature,
humidity and passive infrared (PIR) presence sensors. For
our Kitchen Cleaning experiment, we consider only those
inputs which have been deemed relevant by the feature
selection analysis in [19], that are the PIR readings of the
first five motes plus the information on the x position of
the robot in the environment as measured by the rangefinder localization system. The dataset contains information
on the robot initiating 104 Kitchen Cleaning tasks: these
consist in the robot moving from its base station in the
living room to the kitchen, where a user might be present
or not or might enter the kitchen during robot navigation.
Information is recorded (sampling at 2Hz) from the WSN
and the robot localization system across the whole robot
navigation towards the kitchen, thus collecting 104 time
series of 6 input features. One half of them is associated to a
cleaning task that is correctly completed, while the remainder
are associated to task failures due to the presence of the user
in the kitchen. The target is to learn the Kitchen Cleaning
task preference, that is a target output equal to 0 when the
user enters the kitchen at any point of the cleaning task, while
this is set to 1 when the kitchen is free. Differently from the
previous dataset, target values are associated to each element
of the input sequence and a prediction is thus performed at
each time step t.

We provide an experimental assessment of the effect of
DropIn in two real-world ambient intelligence applications
collected within the scope of the European project RUBICON [15]. Both applications involve a WSN, for which we
simulate the effect of sensor devices faults at prediction
time. For the sake of this work, we assume instead that
a sufficient amount of training data without missing information is available, as tackling with missing information at
training is outside of the scope of this work. Section IV-A
briefly describes the two case studies and summarizes the
experimental setup, while Section IV-B analyzes the results
of the experiments.
A. Data and Experimental Setup
The first dataset is a human movement prediction benchmark based on WSN information which has been originally
presented in [11] and that is available for download on the
UCI repository2 . This task deals with the prediction of room
changes in an office environment, based on the Received
Signal Strength (RSS) of packets exchanged between a WSN
mote worn by the user and four anchor devices deployed
at fixed positions in the corners of the rooms. The experimental data in [11] consists of different setups: in this
work, we refer to the homogenous setup described in the
original paper. Such data is composed of 210 RSS sequences
concerning trajectories of different users moving between 2
room couples separated by an hallway. The trajectories start
at different corners of the rooms and are such that some of
them lead to the user changing room while others end up
with the user remaining in the same room. The task requires
the learning model to predict whether the user is going to
change room or to stay in the current one based on RSS
information recorded from the starting point of the trajectory.
The prediction concerning room change has to be taken at a
marker point P, located at 0.6m from the door, which is the
same for all the movements; therefore, different paths cannot
be distinguished based only on the RSS values collected at
M. This benchmark is formalized as a binary classification
task on the RSS time series. The collected measurements
denote RSS samples (integer values ranging from 0 to 100)
gathered by sending a beacon packet from the anchors to
the mobile at regular intervals, 8 times per second, using
the full transmission power of an IRIS-type mote. The RSS
values from the four anchors are the input features for the
model and are organized into sequences of varying length
corresponding to trajectory measurements from the starting
point until marker P. A target classification label is associated
to each input sequence to indicate whether the user is about to
change room (label +1) or not (label −1). This classification
2 https://archive.ics.uci.edu/ml/datasets/Indoor+User+Movement+
Prediction+from+RSS+data

The experimental assessment is intended to confront the
performance of a standard LI-ESN, i.e. trained without
dropout, with respect to a DropIN LI-ESN model, using
different input unit retention probabilities p ∈ {0.8, 0.5, 0.3}.
The latter p value, in particular, has been added to assess
the performance break-point of introducing input dropout. A
model selection scheme has been set-up to assess learning
performance. First, for each dataset, a hold-out set of 20% of
the total sequences has been extracted to create an external
test set. A k-fold cross-validation approach is applied to the
remaining 80% of the data for model selection purposes,
with k = 5 and k = 3 for the first and second dataset,
respectively. Both standard LI-ESN and DropIn LI-ESN have
been trained by RLS with hyperparameters varying as fol-

lows: number of reservoir neurons in [50, 100, 300, 500] with
10% connectivity, leaky parameter α ∈ [0.1, 0.2, 0.3, 0.5, 1]
and RLS term δ ∈ [0.001, 0.01, 0.1, 1, 10, 100, 1000]. The
λ value for RLS has been fixed to 0.9999995 as discussed
in [12]. For each configuration of the hyperparameters,
we have generated three random reservoir topologies and
random weight initializations using a uniform distribution
in [−0.4, 0.4] for both input and reservoir weights, along the
lines of [11]. The reservoir weights have been re-scaled to
satisfy the necessary ESP condition, such that the spectral
radius ρ(W̃) = 0.99. A validation error is computed for
each hyperparameter configuration (by averaging over the kfolds and over the three network topologies) to select the
best model: this is then trained on the full training data and
tested on the hold-out information.
B. Results and Analysis
The aim of this experimental assessment is to assess the
effect of missing inputs on the predictive performance of
the LI-ESN trained with and without DropIn and using
different unit retention probabilities. To this end, we first
provide baseline results for the models without missing
inputs. Then, we show the effect of an increasing number of
missing input features, by performing test predictions with
input features removed. For instance, the performance with a
single missing input is obtained by removing a single input
feature for each test sequence at a time, then the network
prediction is computed and the process is iterated for all
the input features and an average prediction performance is
computed. Similarly, when testing multiple missing inputs,
we perform predictions using all possible combinations of
missing features and we then provide performance statistics
averaged on all combinations.
Table I shows the baseline results (no missing inputs) and
the model selected configuration for the first dataset. Here
model performance is assessed by classification accuracy.
These baseline results show that the LI-ESN model trained
without DropIn achieves higher validation and test accuracies, while decreasing levels of input retention probabilities
yield to decreasing predictive accuracies when all inputs are
present. This is not surprising as, for instance, a retention
probability p = 0.3 on this task means, essentially, that the
model is being trained to perform predictions using only
information from a single input randomly selected each time
from the available four. When using p = 0.8, the DropIn
ESN performance is not far from that of the standard LIESN: the performance difference between the two can be
explained in the terms of the excessive regularization the
DropIn-ESN is subject to, which yields to an underfitting.
This can be clearly seen by confronting the training accuracy
of the models, where standard LI-ESN achieves an 100%
classification accuracy, while the DropIn-ESN (with p = 0.8)
stops at 91.46%.
Figure 2, on the other hand, shows clearly the advantage
of DropIn training when dealing with missing inputs. The
predictive performance of a standard LI-ESN drops considerably (to about 60%) already when a single input is missing.

Fig. 2. Test accuracy of the LI-ESN vs DropInESN as a function of the
number of missing inputs for the movement prediction task.

On the contrary, the performance of the DropIn network with
p = 0.8 increases with respect to the baseline reaching 90%
and remaining well above the standard LI-ESN model even
when more inputs are lost. The reason for the performance
increase with respect to baseline has to be sought in the fact
that by dropping a single input at test time we are basically
allowing the network with p = 0.8 to work under the same
training conditions, i.e. missing on average one input each
time. The performance of DropIn when using p = 0.5 is also
better than LI-ESN, whereas p = 0.3 is clearly the breakpoint
for the amount of inputs that can be dropped in training for
this task.
Table II shows the baseline results for the Kitchen Cleaning data: here performance is assessed using Mean Absolute
Error (MAE) as targets and outputs are available for each
element of the sequence. On this dataset, the beneficial
effect of the DropIn training is even more evident. Its strong
regularization effect, in this case, ensures that all DropIn
networks yield to better test set performances than standard
LI-ESN even when using all the features, despite the fact that
the latest model achieves the lowest validation error (also the
lowest training error, not shown here). The DropIn network
with p = 0.3 is the one that fits less the training data and,
yet, it generalizes better to the test set. This might be due to
the specifics of the task which, in particular, is characterized
by the presence of 5 PIR inputs which have an high degree
of redundancy with each other, hence favouring the more
constrained model.
The advantage of the DropIn approach is evident especially when considering the effect of missing inputs, as
shown by the plot in Fig. 3. Again, the standard LI-ESN
model does not cope well already with a single missing
input feature, with a MAE jumping to 0.35 and raising up
to about 0.45 when 4 inputs are missing. On the other hand,
the DropIn model with p = 0.3 has the best MAE for most
of the missing inputs configurations: in particular, the loss
of a single input does not shift significantly its error with
respect to the baseline in Table II.
Summarizing, the results of this preliminary analysis sug-

TABLE I
M ODEL SELECTION AND TESTING RESULTS FOR THE MOVEMENT PREDICTION TASK . T HE TABLE REPORTS INFORMATION ON THE SELECTED
CONFIGURATION ( NUMBER OF RESERVOIR NEURONS NR , LEAKY PARAMETER α AND RLS TERM δ) AS WELL AS THE AVERAGE CLASSIFICATION
ACCURACY IN TRAINING AND VALIDATION OVER THE MULTIPLE RANDOM INITIALIZATIONS ( STANDARD DEVIATION IS IN BRACKETS ). T EST
ACCURACY IS REPORTED FOR THE BEST CONFIGURATION SELECTED IN VALIDATION .
Model

NR

α

δ

Training

Valid

Test

LI-ESN
DropInESN (0.8)
DropInESN (0.5)
DropInESN (0.3)

500
500
50
500

0.1
0.2
0.1
0.3

0.1
0.001
0.1
0.1

1.000 (0)
0.9494 (0.027)
0.7386 (0.032)
0.6052 (0.029)

0.9326 (0.025)
0.867 (0.055)
0.6786 (0.068)
0.5734 (0.056)

0.9206
0.8333
0.7778
0.5714

TABLE II
M ODEL SELECTION AND TESTING RESULTS FOR THE K ITCHEN C LEANING TASK . T HE TABLE REPORTS INFORMATION ON THE SELECTED
CONFIGURATION ( NUMBER OF RESERVOIR NEURONS NR , LEAKY PARAMETER α AND RLS TERM δ) AS WELL AS THE AVERAGE MAE IN TRAINING
AND VALIDATION OVER THE MULTIPLE RANDOM INITIALIZATIONS ( STANDARD DEVIATION IS IN BRACKETS ). T EST ERROR IS REPORTED FOR THE
BEST CONFIGURATION SELECTED IN VALIDATION .
Model

NR

α

δ

LI-ESN
DropInESN (0.8)
DropInESN (0.5)
DropInESN (0.3)

500
500
100
500

0.3
0.5
0.5
0.5

10
10
0.1
100

Training
0.0655
0.0904
0.1829
0.1834

(0.033)
(0.029)
(0.015)
(0.037)

Valid
0.1831
0.2056
0.2182
0.2378

Test

(0.058)
(0.092)
(0.055)
(0.063)

0.2136
0.1764
0.1810
0.1471

for some specific tasks.
V. C ONCLUSIONS

Fig. 3. Test MAE of the LI-ESN vs DropInESN as a function of the number
of missing inputs for the Kitchen Cleaning task: a maximum of 4 PIR inputs
is assumed to be missing altogether.

gest that the DropIn approach can indeed be effective in
dealing with missing inputs at prediction time, although this
might come at the cost of a slightly reduced performance
when all inputs are available, depending on the characteristics
of the task. In terms of lesson learned it seems reasonable,
when deploying a predictor which can be subject to missing
inputs, to train two models: the first is a standard predictor
trained without DropIn to work at maximum predictive
performance when all inputs are available. The second is
a DropIn trained model to be queried when one or more
of the inputs is missing to maintain the highest achievable
predictive performances. This seems a feasible and practical
approach, given the low computational impact of the DropIn
procedure. Nevertheless, as suggested by the performance
on the second dataset, a DropIn trained model can also be a
good candidate for being the sole predictor being deployed

We have proposed a novel approach to deal with the problem of missing input information at prediction time in neural
networks, which are related to the well known Dropout
regularization. Our approach, named DropIn, applies Dropout
at the level of input neurons. We exploit the ensembling effect
produced by Dropout to train a neural network that can be
interpreted as a committee of (potentially exponential) subnetworks, each capable of making predictions using only a
subset of the available inputs as determined by the neuron
retention probability. The proposed approach is principled
and general, as it can be seamlessly applied to any artificial
neural network model for which Dropout applies and it
does not require to define ad-hoc committee architectures
or to define data imputation algorithms and models. Further,
DropIn is simple to embed in the training phase of the neural
model and has a minor computational impact.
We have assessed DropIn performance in conjunction with
Reservoir Computing neural models. These are the least
straightforward in terms of dropout application due to the
untrained nature of their input and recurrent connection
weights, which makes recording of the DropIn effect in the
synaptic connections quite challenging. At the same time, to
the extent of our knowledge, this is the first time in which
Dropout techniques are applied to RC models as well as it is
the first work specifically addressing the problem of missing
input at prediction time in RNNs.
The experimental assessment provides a snapshot of the
potential of the approach when applied to real-world predictive tasks comprising input information collected by networks of distributed, fragile and loosely coupled sensor devices. In particular, we have shown how ESN models trained

with DropIn can maintain a good predictive performance
even when a pair of input features is missing, e.g. due
to a sensor fault. Conversely, an ESN model trained with
the standard procedure has a neat predictive performance
degradation already when missing a single input. Such experimental outcome suggests that DropIn can become a useful
methodology when developing ambient intelligence, IoT and
pervasive computing applications that need to continuously
stream their predictions despite some of the inputs being
missing due to device or communication faults.
Future developments of this work will consider extending
the application and assessment of the DropIn methodology
to other neural models, both recurrent and static, also in
conjunction with different case studies from those considered
in this paper. On a longer term, instead, we are interested in
studying if the same technique can be exploited to efficiently
train recursive neural network models that can deal with
structured data where different samples are characterized by
different connectivity and topology [20].
ACKNOWLEDGMENT
This work has been partially supported by the industrial
research project funded by Biobeats Ltd a the Dipartimento
di Informatica, Università di Pisa. D. Bacciu would like to
acknowledge support from the Italian Ministry of Education,
University, and Research (MIUR) under project SIR 2014
LIST-IT (grant n. RBSI14STDE).
R EFERENCES
[1] S. Krause and R. Polikar, “An ensemble of classifiers approach for
the missing feature problem,” in Proc. of the International Joint
Conference on Neural Networks (IJCNN) 2003, vol. 1. IEEE, 2003,
pp. 553–558.
[2] M. Lukosevicius and H. Jaeger, “Reservoir computing approaches to
recurrent neural network training,” Computer Science Review, vol. 3,
no. 3, pp. 127 – 149, 2009.
[3] P. J. Garcı́a-Laencina, J.-L. Sancho-Gómez, and A. R. Figueiras-Vidal,
“Pattern classification with missing data: a review,” Neural Computing
and Applications, vol. 19, no. 2, pp. 263–282, 2010.
[4] Y. Li and L. E. Parker, “Classification with missing data in a wireless
sensor network,” in IEEE SoutheastCon 2008, April 2008, pp. 533–
538.
[5] ——, “Nearest neighbor imputation using spatialtemporal correlations
in wireless sensor networks,” Information Fusion, vol. 15, pp. 64 –
79, 2014.
[6] V. Tresp, R. Neuneier, and S. Ahmad, “Efficient methods for dealing
with missing data in supervised learning,” in Advances in neural
information processing systems. Morgan Kaufmann, 1995, pp. 689–
696.
[7] V. Tresp, S. Ahmad, and R. Neuneier, “Training neural networks with
deficient data,” in Advances in neural information processing systems.
Morgan Kaufmann, 1994, pp. 128–128.
[8] V. Tresp and R. Hofmann, “Missing and noisy data in nonlinear timeseries prediction,” in Neural Networks for Signal Processing [1995]
V. Proceedings of the 1995 IEEE Workshop. IEEE, 1995, pp. 1–10.
[9] Y. Bengio and F. Gingras, “Recurrent neural networks for missing
or asynchronous data,” Advances in neural information processing
systems, pp. 395–401, 1996.
[10] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neural networks
from overfitting.” Journal of Machine Learning Research, vol. 15,
no. 1, pp. 1929–1958, 2014.
[11] D. Bacciu, P. Barsocchi, S. Chessa, C. Gallicchio, and A. Micheli,
“An experimental characterization of reservoir computing in ambient
assisted living applications,” Neural Computing and Applications, vol.
24 (6), pp. 1451–1464, 2014.

[12] H. Jaeger and H. Haas, “Harnessing nonlinearity: Predicting chaotic
systems and saving energy in wireless communication,” Science, vol.
304, no. 5667, pp. 78–80, 2004.
[13] L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization
of neural networks using DropConnect,” in Proceedings of the 30th
International Conference on Machine Learning (ICML-13), 2013, pp.
1058–1066.
[14] H. Jaeger, “The ”echo state” approach to analysing and training recurrent neural networks,” GMD - German National Research Institute
for Computer Science, Tech. Rep., 2001.
[15] M. Dragone, G. Amato, D. Bacciu, S. Chessa, S. Coleman, M. D.
Rocco, C. Gallicchio, C. Gennaro, H. Lozano, L. Maguire, M. McGinnity, A. Micheli, G. M. OHare, A. Renteria, A. Saffiotti, C. Vairo, and
P. Vance, “A cognitive robotic ecology approach to self-configuring
and evolving AAL systems,” Engineering Applications of Artificial
Intelligence, vol. 45, pp. 269 – 280, 2015.
[16] O. Obst, “Distributed fault detection in sensor networks using a
recurrent neural network,” Neural Processing Letters, vol. 40, no. 3,
pp. 261–273, 2014.
[17] MATLAB, version 9.1 (R2016b). Natick, Massachusetts: The MathWorks Inc., 2016.
[18] D. Bacciu, C. Gallicchio, A. Micheli, M. Di Rocco, and A. Saffiotti,
“Learning context-aware mobile robot navigation in home environments,” in IISA 2014, The 5th International Conference on Information, Intelligence, Systems and Applications, July 2014, pp. 57–62.
[19] D. Bacciu, “Unsupervised feature selection for sensor time-series in
pervasive computing applications,” Neural Computing and Applications, vol. 27, no. 5, pp. 1077–1091, 2016.
[20] D. Bacciu, A. Micheli, and A. Sperduti, “Compositional generative
mapping for tree-structured data - part I: Bottom-up probabilistic
modeling of trees,” IEEE Trans. Neural Netw. Learning Syst., vol. 23,
no. 12, pp. 1987–2002, 2012.

