Non-linear motor control by local learning in spiking neural networks

Aditya Gilra 1 Wulfram Gerstner 1

arXiv:1712.10158v1 [q-bio.NC] 29 Dec 2017

Abstract
Learning weights in a spiking neural network
with hidden neurons, using local, stable and online rules, to control non-linear body dynamics is
an open problem. Here, we employ a supervised
scheme, Feedback-based Online Local Learning
Of Weights (FOLLOW), to train a network of
heterogeneous spiking neurons with hidden layers, to control a two-link arm so as to reproduce a
desired state trajectory. The network first learns
an inverse model of the non-linear dynamics, i.e.
from state trajectory as input to the network, it
learns to infer the continuous-time command that
produced the trajectory. Connection weights are
adjusted via a local plasticity rule that involves
pre-synaptic firing and post-synaptic feedback of
the error in the inferred command. We choose a
network architecture, termed differential feedforward, that gives the lowest test error from different feedforward and recurrent architectures. The
learned inverse model is then used to generate a
continuous-time motor command to control the
arm, given a desired trajectory.

1. Introduction
Motor control requires building internal models of the
muscles-body system (Conant & Ashby, 1970; Pouget &
Snyder, 2000; Wolpert & Ghahramani, 2000; Lalazar &
Vaadia, 2008). The brain possibly uses random movements
during pre-natal (Khazipov et al., 2004) and post-natal development (Petersson et al., 2003) termed motor babbling
(Meltzoff & Moore, 1997; Petersson et al., 2003) to learn
internal models that associate body movements to neural
motor commands (Lalazar & Vaadia, 2008; Wong et al.,
2012; Sarlegna & Sainburg, 2009; Dadarlat et al., 2015).
For motor control, given a desired state trajectory for the
dynamical system formed by the muscles and body, net1
School of Computer and Communication Sciences, and
Brain-Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne, 1015 Lausanne EPFL, Switzerland.
Correspondence to: Aditya Gilra <aditya.gilra@epfl.ch>.

Copyright 2017 by the author(s).

works of spiking neurons in the brain must learn to produce the time-dependent neural control input that activates
the muscles to produce the desired movement. Abstracting
the muscles-body dynamics as
d~x/dt = f~(~x, ~u),

(1)

we require the network to generate control input ~u(t), given
desired state trajectory ~xD (t), that makes the state evolve
as ~x(t) ≈ ~xD (t). We train a network of spiking neurons with hidden layers, to learn the inverse model of the
muscle-body dynamics, namely to infer control input ~u(t)
given state trajectory ~x(t). The inverse model is then used
to control the body, in closed loop mode.
In networks of continous-valued or spiking neurons,
training the weights of hidden neurons with a local learning rule is considered difficult due to the credit assignment problem (Bengio et al., 1994; Hochreiter et al.,
2001; Abbott et al., 2016). Supervised learning in networks has typically been accomplished by backpropagation which is non-local (Rumelhart et al., 1986; Williams
& Zipser, 1989), reservoir computing which trains only
output weights (Jaeger, 2001; Maass et al., 2002; Legenstein et al., 2003; Maass & Markram, 2004; Jaeger & Haas,
2004; Joshi & Maass, 2005; Legenstein & Maass, 2007),
FORCE learning which requires weight changes faster than
the requisite dynamics (Sussillo & Abbott, 2009; 2012; DePasquale et al., 2016; Thalmeier et al., 2016; Nicola &
Clopath, 2016), and other schemes for example (Sanner
& Slotine, 1992; DeWolf et al., 2016; MacNeil & Eliasmith, 2011; Bourdoukan & Denève, 2015). Here, we employ a recent learning scheme called Feedback-based Online Local Learning Of Weights (FOLLOW) (Gilra & Gerstner, 2017), which draws upon function and dynamics approximation theory (Funahashi, 1989; Hornik et al., 1989;
Girosi & Poggio, 1990; Sanner & Slotine, 1992; Funahashi
& Nakamura, 1993; Pouget & Sejnowski, 1997; Chow &
Li, 2000; Seung et al., 2000; Eliasmith & Anderson, 2004;
Eliasmith, 2005) and adaptive control theory (Morse, 1980;
Narendra et al., 1980; Slotine & Coetsee, 1986; Slotine &
Weiping Li, 1987; Narendra & Annaswamy, 1989; Sastry
& Bodson, 1989; Ioannou & Sun, 2012). FOLLOW is a
synaptically local and provably stable and convergent alternative for training the feedforward and recurrent weights
of hidden spiking neurons in a network, that was employed

to predict non-linear dynamics (Gilra & Gerstner, 2017).
While the authors of the FOLLOW scheme (Gilra & Gerstner, 2017) learned a forward model of arm dynamics, i.e.
predicting arm state given neural control input, we train
the network to perform full motor control, first learning the
inverse model to infer control input given current arm trajectory, and then employing it in closed loop to make the
arm replicate a desired trajectory.

A

2. Network architecture and FOLLOW
scheme to learn inverse model
We use a network of heterogeneous leaky-integrate-andfire (LIF) neurons, with different biases to learn the inverse model of an arm, as depicted in Figure 1. The arm,
adapted from (Li, 2006), is modelled as a two-link pendulum moving in a vertical plane under gravity, with friction
at the joints, 0◦ as the equilibrium downwards position, and
soft bounds on motion beyond ±90◦ . As shown in Figure
1A, random 2-dimensional torque input uγ (t), γ = 1, 2 is
provided to the arm to generate random state trajectories
~x(t) analogous to motor babbling. The 4-dimensional state
of the arm (2 joint angles and 2 joint velocities), denoted
xβ , β = 1, .., 4, is fed as input to the network via fixed,
random weights. The network must learn to infer ~u(t) that
generated the arm state trajectory ~x(t).

B

Figure 1. Network configuration for learning the inverse
model. A. During learning, random motor commands cause arbitrary movements of the arm (motor babbling). Given the resulting
state of the arm, the network must infer the continuous-time motor commands that caused it. The observed 4-dimensional state
variables xβ of the arm (which can be obtained from visual and
proprioceptive feedback) are provided as input to the network via
fixed random weights e(ff)
iβ . The 2-dimensional motor command
ûγ is linearly decoded from the filtered output spike trains of
the network via fixed weights dγi . A copy of the random motor command (input to the arm) is used, after a delay of ∆u ,
to compute the error in the inferred motor command, i.e. error
γ (t) = uγ (t−∆u )− ûγ (t). This deviation of the predicted command from the reference command, is fed back into the network
with fixed random encoding weights keiγ . This error signal is
used to update the network weights so as to infer the motor commands causing the observed state. Twin lines in the connection
arrows denote multi-dimensional signals, but their number is not
representative of the dimensionality. B. Learning inverse model
with a differential feedforward network. The learning paradigm
and configuration are as in panel A. The input is sent undelayed
to a layer of neurons ff1 via encoding weights e(ff1)
kβ and sent delayed by ∆ to another layer of neurons ff2 via encoding weights
e(ff2)
lβ . These two layers feed into the next ‘computation’ layer with

We chose the network in Figure 1B, termed the differential feedforward network, from a variety of network architectures (see next section), to learn the inverse model. We
have K = L = 3000 neurons each in the two feedforward
hidden layers indexed by k = 1 . . . K, and l = 1 . . . L,
followed by N = 5000 neurons in the ‘computation’ hidden layer indexed by i = 1 . . . N . The state vector ~x is
fed to the two differential feedforward layers, undelayed to
one and delayed to the other by interval ∆, via fixed ran(ff1)
(ff2)
dom weights ekβ and elβ respectively. The current into
a neuron k in the undelayed hidden layer is given by
X
Jk =
eff1
(2)
kβ xβ (t) + bk ,
β

where bk is a fixed, random neuron-specific bias. Similarly,
the current into a neuron l in the delayed hidden layer is
given by
X
Jl =
eff2
(3)
lβ xβ (t − ∆) + bl ,
β

(1)

where bl is also a fixed, random neuron-specific bias.
The output ~û(t) of the network is a linearly weighted
sum of filtered spike trains Si (t) of the computation layer
neurons:
Z t
X
ûγ (t) =
dγi
Si (s)κ(t − s)ds,
(4)
i

(2)

weights wik and wil that are plastic (in red). The error is fedback to the ‘computation’ layer with weights keiγ . This error projected into each neuron along with the filtered pre-synaptic spike
train is used to update the feedforward weights in red.

−∞

2

with the filtering kernel κ(t) a decaying exponential with
time constant 20 ms, and the readout weights dγi . This
readout ~û(t) of the network is compared to a time-delayed
version of the command input, with delay ∆u ms, to
causally infer the past command. Borrowing from adaptive
control theory (Narendra & Annaswamy, 1989; Ioannou &
Sun, 2012), the error γ ≡ uγ (t−∆u )−ûγ (t) is fed back as
neural currents to the computation layer neurons with fixed
random feedback weights eiγ multiplied by a gain k = 10.
The total current into neuron i in the ‘computation’ layer,
having fixed random bias bi , is a sum of the feedforward1,
feedforward2 and error currents:
Ji =

X
k

(1)

wik (Skff1 ∗ κ)(t) +
X
γ

X
l

After training the network on inferring the command input given the state trajectory during motor babbling, we
expect that when we remove the error feedback loop, the
network will still be able to infer the command input given
the current trajectory. We show the various stages of learning and testing in Figure 2, for the differential-feedforward
network of Figure 1B.

3. Learning inverse model: inferring control
command given arm trajectory
At the start of learning, all trainable weights were initialized to zero. The state trajectory was fed as input to
the network, while the network had to learn to produce the
command (delayed by ∆u ) that caused the trajectory.

(2)

wil (Slff2 ∗ κ)(t)+

keiγ (γ ∗ κ)(t) + bi . (5)

In Figure 2, we show the different stages in FOLLOW
learning for the differential feedforward network (Fig. 1B).
Before learning, without feedback, the network output remained zero. With feedback on, even before the weights
have been learned, the network output followed the desired
output i.e. the time-delayed reference command. With
feedback on, the network was trained with the FOLLOW
learning rule. At the end of 10,000 s of FOLLOW learning
on the feedforward weights, at learning rate 2e-3, we froze
the weights and removed the feedback, in order to test if the
network had learned the inverse model. We also tested the
network on a more structured task trajectory, in addition to
random motor babbling. The network output inferred the
command given the state trajectory, even without feedback
indicating that the network had learned the inverse model.

The trick in the FOLLOW scheme is to pre-learn the
readout weights dγi to be an auto-encoder with respect to
error feedback weights eiγ . Learning the auto-encoder can
be accomplished by existing schemes (Voegtlin, 2006; Burbank, 2015; Urbanczik & Senn, 2014), and so the autoencoder was pre-learned algorithmically here. Due to this
auto-encoder, the error feedback with high gain k acts as a
negative feedback that serves to make the network output
~û(t) follow the true command torque ~u(t − ∆), even without the hidden weights being learned. The system dynamics and command torque are required to vary slower than
the synaptic timescale. As explained in (Gilra & Gerstner,
2017), even the hidden neurons are entrained to fire as they
ideally should, and this enables the feedforward (or recurrent) hidden weights to be learned, using only synaptically
available quantities, as:

We used a causal delay ∆u in supplying the target commands, since the command in the past that caused the current state input needs to be inferred. The differential delay
∆ between the undelayed and delayed feedforward layers
possibly controls the accuracy of the temporal derivative.
We swept the delays ∆u and ∆ over typical network time
scales, shown in Figure 3, and found that ∆u ≈ 50 ms and
∆ ≈ 50 ms gave the lowest test error. This ∆u is consistent
with the delay due to two synaptic filtering time constants
of 20 ms each from the input state to the inferred command.
For a command delayed by ∆u = 50 ms to be inferred,
a ∆ = 50 ms differential delay allows the most accurate
computation of the derivative, perhaps because it is closest
to the time of the reference command, than a ∆ = 10 or
20 ms delay, as seen in Figure 3. Thus, for all remaining
simulations including Figure 2, we used ∆ = 50 ms and
∆u = 50 ms.

dwij
= η (Ii ∗ κ )(Sj ∗ κ)(t),
(6)
dt
P
where Ii ≡ k α eiα α is the error current injected into

each neuron, κ is a decaying exponential filter of time constant 200 ms and Sj is the pre-synaptic spike train.
This FOLLOW learning rule is applied on the plastic
weights (in red in Fig. 1B), while the current state of the
arm is fed to the network, with the network output following the time-delayed command input torque due to the error
feedback loop. FOLLOW learning has been shown, under
reasonable approximations, to be uniformly stable using a
Lyapunov approach, with the squared error in the learned
output tending asymptotically to zero (Gilra & Gerstner,
2017). Details and parameters for generating the random
input, the arm dynamics and the fixed random network parameters are as in (Gilra & Gerstner, 2017).

We asked whether the seemingly low-pass filtering of the
inferred torque during testing in Figure 2B, was because
the command varied faster than what the network could approximate. We used low-pass filtered commands for both
3

A

|

xβ

1.5

| noise

start Learning end

Testing

structured

0.0
1.5
feedback off | feedback on
0.6

û 2 , u2

B

feedback on |feedback off

feedback off

0.0
0.6

0

time (s)

2

9995

9997

time (s)

0

time (s)

2

Figure 2. Stages during FOLLOW learning of inverse model using the differential feedforward network (Fig. 1B). A,B. The
vertical red lines divide the figure into three time stages: before learning without feedback on the left, during learning with feedback in
the middle, and during testing without feedback on the right. A. The input to the network ~
x(t) namely the 4-dimensional state trajectories
are shown in 4 different colours. B. A component of the reference torque u2 (t − ∆) to the arm (in blue) is shown along with the network
readout uˆ2 (t) (in red). Before learning without feedback (on the left), the network output (red) doesn’t infer the command torque (blue).
During learning with feedback (middle), the network output follows the command torque due to the negative feedback (Fig. 1), while the
error, which decreases over time, is used to train the hidden weights. After learning (right), i.e. freezing weights and removing feedback,
the network infers the command torque even without feedback, indicating that it has learned the inverse model.

0.025

­

err 2

­

®

err 2

®

Nd , t

Nd , t

0.025

0.000
20

∆ u (ms

)

100 100

∆

)
(ms

0

0.000

Figure 3. Mean squared test error versus ∆ and ∆u . For different values of differential delay ∆ and causal torque delay ∆u ,
we learned the inverse model using the differential feedforward
network, having 200 neurons in each of the undelayed and delayed layers and 500 neurons in the computation layer. We plot
the mean squared test error without feedback, after approximately
10,000s of learning, averaged over 4 s and per state dimension.
The star marks the mean squared test error for ∆ = 50 ms and
∆u = 50 ms.

0

torque filtering τ (ms)

100

Figure 4. Mean squared test error versus torque filtering τ .
For increasing values of torque filtering time constant τ , we
learned the inverse model using the same differential feedforward
network as in Figure 3, with ∆ = 50 ms and ∆u = 50 ms. We
plot the mean squared test error without feedback, as in Figure 3,
with the same range for comparison. The star for τ = 0 marks the
mean squared error for the same simulation as the star in Figure
3, while the diamond below it marks the mean squared error for
the simulation with larger number of neurons in Figure 2.

4

A

the arm and the network, with increasing filtering time constants, using a decaying exponential kernel, but the test error remained almost the same, as shown in Figure 4. With a
larger number of neurons, the test error came down without
any filtering on the torque (Fig. 4), so the approximation error due to finite number of neurons seemed larger than that
due to any fast timescales in the torque. Thus, in all further
simulations, we did not use any filtering on the torque.
The network approximates the inverse dynamics using
the tuning curves of the heterogeneous neurons as an overcomplete set of basis functions (Funahashi, 1989; Hornik
et al., 1989; Girosi & Poggio, 1990; Sanner & Slotine,
1992; Funahashi & Nakamura, 1993; Pouget & Sejnowski,
1997; Chow & Li, 2000; Seung et al., 2000; Eliasmith &
Anderson, 2004; Eliasmith, 2005). As studied in adaptive control theory (Ioannou & Sun, 2012; Narendra & Annaswamy, 1989; Ioannou & Fidan, 2006), the error due to
approximation causes a drift in weights, which can lead
to error increasing after some training time. The same
literature also suggests ameliorative techniques which include a weight leakage term switched on slowly, whenever
a weight crosses a set value (Ioannou & Tsakalis, 1986;
Narendra & Annaswamy, 1989), or a dead zone policy of
not updating weights if the error falls below a threshold
(Slotine & Coetsee, 1986; Ioannou & Sun, 2012). However, we did not need to implement these techniques for
learning the inverse model with our networks, suggesting
that the approximations with our finite number of neurons
were reasonable.

B

Figure 5. Recurrent network cannot learn the inverse model
via FOLLOW learning. A. The learning configuration is same
as Figure 1A, except that the spiking network is recurrently connected. The feedforward weights effiβ and the recurrent weights
wij are plastic under the FOLLOW rule. B. We trained the recurrent network on the inverse model as in A, with 900 neurons in
the recurrent layer and ∆u = 50 ms, at increasing values of the
learning rate (x-axis, log scale) on the recurrent weights, while
the feedforward weights were learned at a constant rate. We plot
the mean squared test error (y-axis, log scale) without feedback,
after approximately 500s of learning, averaged over 4 s and per
state dimension.

In learningRthe forward model, the command ~u must be
integrated as f (~x, ~u)dt to obtain the state ~x, see equation
(1). However, for the inverse model, the state ~x has to be
differentiated, and the function f inverted, to infer the command ~u. In principle, we could use a recurrent network, as
it is Turing complete (Siegelmann & Sontag, 1995), to approximate the inverse model, provided all the input, recurrent and output weights are wired correctly. However, in
the FOLLOW learning scheme (Gilra & Gerstner, 2017),
the output weights are fixed to match the error encoding
weights (auto-encoder), to enable local learning. Indeed,
we tried to learn the inverse model with a recurrent network as in Figure 5A. The total current into neuron i in the
recurrent layer, having fixed random bias bi , was a sum of
the feedforward, recurrent and error currents:
Ji =

X
β

effiβ xβ (t) +

X
j

from effectively no learning to a reasonable learning rate
2e-3 that had worked very well for learning the forward
model using a recurrent network (Gilra & Gerstner, 2017).
As shown in Figure 5, we found that test performance
dropped sharply with learning on the recurrent weights.
Near zero learning on the recurrent weights gave the lowest mean squared test error. Even if we included a feedforward layer between the state input and the recurrent network, with learnable feedforward and recurrent weights,
as for learning the forward model in (Gilra & Gerstner,
2017), still test performance degraded with learning on the
recurrent weights. Note that a recurrent network in the
FOLLOW learning scheme, which has fixed output weights
(Fig. 5A), cannot be transformed into the differential feedforward architecture (Fig. 1B). Even if part of the recurrent network can learn to delay or hold an earlier state in
memory, it always remains connected to the output by fixed

wij (Sj ∗ κ)(t)+
X
γ

keiγ (γ ∗ κ)(t) + bi . (7)

We kept a fixed learning rate on the feedforward weights,
but increased the learning rate on the recurrent weights
5

weights.

feedback with low gain. A proportional-integral-derivative
(PID) feedback should provide even better tracking of the
output without oscillations and offset.

Further, even from among the feedforward network architectures we tried with FOLLOW learning, as shown in
Figure 6, we found that the best performace for a fixed
number of neurons was obtained by the differential feedward network (Fig. 1B). Compared to the other feedforward architectures, we imagine that the differential feedforward network could learn to compute a temporal difference
between the undelayed and delayed states, approximating
the time derivative.

Without the inverse model, the linear negative feedback
loop (be it proportional or PID) on the state variables cannot make the arm follow the desired trajectory, even by tuning feedback gains, because of the complicated non-linear
dynamics of the two-link arm. In our control scheme, the
intuition is that the inverse model learned in the weights of
the network produces an open-loop command that brings
the arm close enough to the desired state, such that linear
negative feedback around this momentary operating point
can bring the arm even further closer.

4. Moving the arm as desired using the
inverse model
Having learned the inverse model, we used it to control
the arm to reproduce a desired state trajectory, for which
the control input was not known and had to be generated by
the network. As shown in Figure 7, the desired state xD
β (t)
was fed to the network which had already been trained as
an inverse model to output the requisite control command
ûγ . This command generated by the network was fed as
torques to the joints of the arm to produce the desired motion. In this simple feed-forward or open loop mode, small
errors in the generated control command will integrate over
time into larger errors in the trajectory of the arm, causing
large deviations from the desired trajectory over time. To
ameliorate this, we closed the loop and injected the difference in the desired state and the true state multiplied by
gain k 0 = 3, back into the network input (Fig. 7). In computing the error, the desired state was delayed by ∆u = 50
ms, since the network was also trained to output the control command with this lag. This feedback loop for the
error in the state variables, fed into the network via the input weights during control, is different from the feedback
loop in the error for control command, fed back via the error feedback weights during learning. The latter feedback
loop is in fact used only during learning and is not present
during control.

5. Discussion
We used a synaptically local, online and stable learning scheme, FOLLOW (Gilra & Gerstner, 2017), to train
a network of heterogeneous spiking neurons with hidden
layers, to infer the continuous-time command needed to
drive a non-linear dynamical system, here a two-link arm,
to produce a desired trajectory. We found that the differential feedforward network architecture performed best in
learning the inverse model, from among a variety of feedforward and recurrent architectures, under the FOLLOW
learning scheme. Under closed-loop, this trained inverse
model was then used to control the arm. We expect that
with proportional-integral-derivative feedback, the control
of the arm will improve further.
Previous methods of training networks with hidden units
incorporate only some of the desirable features of FOLLOW learning. Backpropagation and variants, backpropagation through time (BPTT) (Rumelhart et al., 1986) and
real-time recurrent learning (RTRL) (Williams & Zipser,
1989), are non-local in time or space (Pearlmutter, 1995;
Jaeger, 2005). Synaptically local learning rules in recurrent spiking neural networks have typically not been
demonstrated for learning inverse models and non-linear
motor control (MacNeil & Eliasmith, 2011; Bourdoukan
& Denève, 2015; Gilra & Gerstner, 2017; Alemi et al.,
2017). Reservoir computing methods (Jaeger, 2001; Maass
et al., 2002; Legenstein et al., 2003; Maass & Markram,
2004; Jaeger & Haas, 2004; Joshi & Maass, 2005; Legenstein & Maass, 2007) did not initially learn weights within
the hidden units, while newer FORCE learning methods
that did (Sussillo & Abbott, 2009; 2012; DePasquale et al.,
2016; Thalmeier et al., 2016; Nicola & Clopath, 2016), require weights to change faster than the time scale of the
dynamics, and require different components of a multidimensional error to be fed to different sub-parts of the network. Reward-modulated Hebbian rules have not yet been
used for continuous-time control (Legenstein et al., 2010;

We tested our motor control scheme for drawing a diamond and a star on the wall (Fig. 8). In open loop, i.e.
not feeding back the error in state variables, the command
torque generated by the network enabled the arm to draw a
pattern similar to that desired (Fig. 8 C,D and Supplementary movies 1 and 2). However, some parts of the pattern,
which required large bending of the elbow joint were difficult to reproduce, possibly due to poorer learning at the
limits of the motion, and divergence of the state from the
desired state in open loop over time.
When we closed the loop with gain k 0 = 3 the pattern was reproduced better (Fig. 8 C,D and Supplementary movies 3 and 4). Here, we used only proportional
6

A

B

C

Figure 6. Comparison of feedforward network architectures. A,B,C. In the top panel, the feedforward network architecture used to
learn the inverse model is shown. Weights in red are plastic. Each network has the same number of heterogeneous LIF neurons. Mean
squared error during training with feedback is plotted versus time over approximately 10,000s seconds of learning in the middle panels.
Reference torque component u2 (blue) and inferred torque component û2 (red) given desired state are plotted versus time, during 4 s
of testing without feedback. The mean squared test error without feedback per unit time per state dimension is written above the plot.
Mean squared error during test without feedback is typically larger than that during training due to corrective negative feedback. A.
The differential feedforward network is our default network architecture, with 200 neurons in the undelayed and delayed layers and 500
neurons in the computation layer, with ∆ = 50 ms and ∆u = 50 ms. B. The direct differential feedforward network doesn’t have
intermediate undelayed and delayed hidden layers. Rather the undelayed and delayed states are directly feed on to the computation layer
having 900 neurons. C. The purely feedforward network has 450 neurons in each hidden layer.

differential feedforward network
(ff1)

desired state D
x β + k 0 β
xD
β
+

+

k0

eiβ

delay ∆

+

0β

two-link arm
dynamical system
control input ûγ

(ff2)
ekβ
delay ∆u

(1)
wij

−

(2)

wik

dγi

true state xβ

Figure 7. Schematic for control using inverse model. The desired state trajectory ~
xD (t) is fed into the differential feedforward network
that is already trained as an inverse model. The control input ûγ is read out from the network. The read out control input torques are
delivered to the arm which produces the true state xβ (t), which is already close to the desired one if the inverse model performs well.
To further improve long-time performance, the true state of the arm is compared with the desired state, delayed by ∆u = 50 ms (since
0
the inverse model has learned to infer the control input with this delay), and the error 0β = (xD
β − xβ ) is fed back with gain k = 3.
The non-linear feedforward control by the inverse model brings the arm state close to the desired state, after which the linear negative
feedback control brings the arm state even closer. Twin lines in the connection arrows denote multi-dimensional signals, but their number
is not representative of the dimensionality.

7

C

B

0.7
0.0
0.3

x2D, 4 ; x2, 4

û 1, 2

A

0

time (s)

6

1

tations required for learning.

0

Interesting directions to explore could be to learn and
control more complex dynamical systems like robotic arms
in real-time, to implement the same on neuromorphic hardware, to make the FOLLOW scheme more biologically
plausible, to learn to generate the desired trajectory given a
higher-level goal via reinforcement learning, and to extend
motor control to a hierarchy of levels as in the brain.

1

0

open-loop

time (s)

6

closed loop

6. Acknowledgements

D
open-loop

Financial support was provided by the European Research Council (Multirules, grant agreement no. 268689),
the Swiss National Science Foundation (Sinergia, grant
agreement no. CRSII2 147636), and the European Commission Horizon 2020 Framework Program (H2020) (Human Brain Project, grant agreement no. 720270).

closed loop

Figure 8. Control of two-link arm via inverse model. A. The
elbow and shoulder control torques generated by the learned differential feedforward network under closed loop in the control
architecture (Fig. 7) in reponse to a desired star-drawing trajectory. B. The star-drawing trajectories for the elbow angle (solid)
and angular velocity (dashed) as desired (blue) and as produced
by the arm (red) in response to the torques in A, under closed loop
control of the inverse model. C,D. The network controls the arm
to draw a diamond (C) and a star (D). On the left in blue, is the
desired trajectory, in coordinate space, that the end-point of the
arm is required to trace, starting in the direction of the green arrow. In the middle in red, is the trace produced by the arm under
open-loop control, i.e. no feedback of the error in state back to the
network. To the right, in red is the trace produced under closedloop, with feedback (gain k0 = 3) of the error in state variables.

References

Hoerzer et al., 2014; Kappel et al., 2017). Approaches to
control of non-linear systems seem to use non-local rules
(Sanner & Slotine, 1992; Slotine & Weiping Li, 1987; Slotine & Coetsee, 1986; DeWolf et al., 2016; Zerkaoui et al.,
2009; Hennequin et al., 2014; Song et al., 2016) or use
abstract networks (Berniker & Kording, 2015; Hanuschkin
et al., 2013).

Berniker, Max and Kording, Konrad P. Deep networks
for motor control functions. Frontiers in Computational Neuroscience, 9, 2015. ISSN 1662-5188. doi:
10.3389/fncom.2015.00032.

Abbott, L. F., DePasquale, Brian, and Memmesheimer,
Raoul-Martin. Building functional networks of spiking
model neurons. Nature Neuroscience, 19(3):350–355,
March 2016. ISSN 1097-6256. doi: 10.1038/nn.4241.
Alemi, Alireza, Machens, Christian, Denève, Sophie, and
Slotine, Jean-Jacques. Learning arbitrary dynamics in
efficient, balanced spiking networks using local plasticity rules. arXiv:1705.08026 [q-bio], May 2017.
Bengio, Y., Simard, P., and Frasconi, P. Learning long-term
dependencies with gradient descent is difficult. IEEE
Transactions on Neural Networks, 5(2):157–166, March
1994. ISSN 1045-9227. doi: 10.1109/72.279181.

Bourdoukan, Ralph and Denève, Sophie. Enforcing balance allows local supervised learning in spiking recurrent networks. In Cortes, C., Lawrence, N. D., Lee,
D. D., Sugiyama, M., Garnett, R., and Garnett, R. (eds.),
Advances in Neural Information Processing Systems 28,
pp. 982–990. Curran Associates, Inc., 2015.

Since FOLLOW-based learning of motor control uses
spiking neurons and is local and online, it is biologically
plausible in these aspects, as a candidate learning scheme
for motor control in the brain. However, it violates Dale’s
law in that outgoing synaptic weights from a single neuron in the learned network can be both positive and negative. However, even without obeying Dale’s law, it can
be implemented directly in neuromorphic hardware and incorporated into neuro-robotics for motor control, where the
spike-based coding provides power efficiency, and localility eases hardware implementation and speeds up compu-

Burbank, Kendra S. Mirrored STDP Implements Autoencoder Learning in a Network of Spiking Neurons. PLoS
Computational Biology, 11(12), December 2015. ISSN
1553-734X. doi: 10.1371/journal.pcbi.1004566.
Chow, T. W. S. and Li, Xiao-Dong. Modeling of continuous
time dynamical systems with input by recurrent neural
networks. IEEE Transactions on Circuits and Systems I:
8

Fundamental Theory and Applications, 47(4):575–578,
April 2000. ISSN 1057-7122. doi: 10.1109/81.841860.

neurons and links them to control theoretic inverse models. Frontiers in Neural Circuits, 7:106, 2013. doi:
10.3389/fncir.2013.00106.

Conant, Roger C. and Ashby, W. Ross. Every good regulator of a system must be a model of that system. Intl. J.
Systems Science, pp. 89–97, 1970.

Hennequin, Guillaume, Vogels, Tim P., and Gerstner, Wulfram. Optimal Control of Transient Dynamics in Balanced Networks Supports Generation of Complex Movements. Neuron, 82(6):1394–1406, June 2014. ISSN
0896-6273. doi: 10.1016/j.neuron.2014.04.045.

Dadarlat, Maria C., O’Doherty, Joseph E., and Sabes,
Philip N. A learning-based approach to artificial sensory feedback leads to optimal integration. Nature Neuroscience, 18(1):138–144, January 2015. ISSN 10976256. doi: 10.1038/nn.3883.

Hochreiter, Sepp, Bengio, Yoshua, Frasconi, Paolo, and
Schmidhuber, Jürgen. Gradient Flow in Recurrent Nets:
The Difficulty of Learning Long-Term Dependencies.
2001.

DePasquale, Brian, Churchland, Mark M., and Abbott,
L. F. Using Firing-Rate Dynamics to Train Recurrent
Networks of Spiking Model Neurons. arXiv:1601.07620
[q-bio], January 2016.

Hoerzer, Gregor M., Legenstein, Robert, and Maass, Wolfgang. Emergence of Complex Computational Structures From Chaotic Neural Networks Through RewardModulated Hebbian Learning. Cerebral Cortex, 24(3):
677–690, January 2014. ISSN 1047-3211, 1460-2199.
doi: 10.1093/cercor/bhs348.

DeWolf, Travis, Stewart, Terrence C., Slotine, JeanJacques, and Eliasmith, Chris. A spiking neural model
of adaptive arm control. Proc. R. Soc. B, 283(1843):
20162134, November 2016. ISSN 0962-8452, 14712954. doi: 10.1098/rspb.2016.2134.

Hornik, Kurt, Stinchcombe, Maxwell, and White, Halbert.
Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989. ISSN
0893-6080. doi: 10.1016/0893-6080(89)90020-8.

Eliasmith, Chris. A Unified Approach to Building and
Controlling Spiking Attractor Networks. Neural Computation, 17(6):1276–1314, June 2005. ISSN 0899-7667.
doi: 10.1162/0899766053630332.

Ioannou, P. and Tsakalis, K. A robust direct adaptive controller. IEEE Transactions on Automatic Control, 31
(11):1033–1043, November 1986. ISSN 0018-9286. doi:
10.1109/TAC.1986.1104168.

Eliasmith, Chris and Anderson, Charles H. Neural Engineering: Computation, Representation, and Dynamics
in Neurobiological Systems. MIT Press, August 2004.
ISBN 978-0-262-55060-4.

Ioannou, Petros and Fidan, Barýp. Adaptive Control Tutorial. SIAM, Society for Industrial and Applied Mathematics, Philadelphia, PA, December 2006. ISBN 978-089871-615-3.

Funahashi, Ken-Ichi. On the approximate realization of
continuous mappings by neural networks. Neural Networks, 2(3):183–192, 1989. ISSN 0893-6080. doi:
10.1016/0893-6080(89)90003-8.

Ioannou, Petros and Sun, Jing. Robust Adaptive Control.
Dover Publications, Mineola, New York, first edition,
December 2012. ISBN 978-0-486-49817-1.

Funahashi, Ken-ichi and Nakamura, Yuichi. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801–806,
1993. ISSN 0893-6080. doi: 10.1016/S0893-6080(05)
80125-X.

Jaeger, H. The ”echo state” approach to analysing and
training recurrent neural networks. Technical report,
2001.
Jaeger, Herbert. A Tutorial on Training Recurrent Neural
Networks, Covering BPPT, RTRL, EKF and the ”Echo
State Network” Approach. 2005.

Gilra, Aditya and Gerstner, Wulfram. Predicting non-linear
dynamics by stable local learning in a recurrent spiking
neural network. eLife, 6:e28295, November 2017. ISSN
2050-084X. doi: 10.7554/eLife.28295.

Jaeger, Herbert and Haas, Harald. Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy
in Wireless Communication. Science, 304(5667):78–
80, April 2004. ISSN 0036-8075, 1095-9203. doi:
10.1126/science.1091277.

Girosi, F. and Poggio, T. Networks and the best approximation property. Biological Cybernetics, 63(3):169–
176, July 1990. ISSN 0340-1200, 1432-0770. doi:
10.1007/BF00195855.

Joshi, Prashant and Maass, Wolfgang. Movement Generation with Circuits of Spiking Neurons. Neural Computation, 17(8):1715–1738, August 2005. ISSN 0899-7667.
doi: 10.1162/0899766054026684.

Hanuschkin, Alexander, Ganguli, Surya, and Hahnloser,
Richard. A Hebbian learning rule gives rise to mirror
9

Kappel, David, Legenstein, Robert, Habenschuss, Stefan, Hsieh, Michael, and Maass, Wolfgang. Rewardbased stochastic self-configuration of neural circuits.
arXiv:1704.04238 [cs, q-bio], April 2017.

(9):e22885, September 2011. doi: 10.1371/journal.pone.
0022885.
Meltzoff, Andrew N. and Moore, M. Keith. Explaining
Facial Imitation: A Theoretical Model. Early development & parenting, 6(3-4):179–192, 1997. ISSN 10573593. doi: 10.1002/(SICI)1099-0917(199709/12)6:3/
4h179::AID-EDP157i3.0.CO;2-R.

Khazipov, Rustem, Sirota, Anton, Leinekugel, Xavier,
Holmes, Gregory L., Ben-Ari, Yehezkel, and Buzsáki,
György. Early motor activity drives spindle bursts in the
developing somatosensory cortex. Nature, 432(7018):
758–761, December 2004. ISSN 0028-0836. doi:
10.1038/nature03132.

Morse, A. Global stability of parameter-adaptive control
systems. IEEE Transactions on Automatic Control, 25
(3):433–439, June 1980. ISSN 0018-9286. doi: 10.1109/
TAC.1980.1102364.

Lalazar, Hagai and Vaadia, Eilon. Neural basis of sensorimotor learning: Modifying internal models. Current Opinion in Neurobiology, 18(6):573–581, December 2008. ISSN 0959-4388. doi: 10.1016/j.conb.2008.
11.003.

Narendra, K., Lin, Yuan-Hao, and Valavani, L. Stable
adaptive controller design, part II: Proof of stability.
IEEE Transactions on Automatic Control, 25(3):440–
448, June 1980. ISSN 0018-9286. doi: 10.1109/TAC.
1980.1102362.

Legenstein, Robert and Maass, Wolfgang. Edge of chaos
and prediction of computational performance for neural
circuit models. Neural Networks, 20(3):323–334, April
2007. ISSN 0893-6080. doi: 10.1016/j.neunet.2007.04.
017.

Narendra, Kumpati S. and Annaswamy, Anuradha M. Stable Adaptive Systems. Prentice-Hall, Inc, 1989.
Nicola, Wilten and Clopath, Claudia. Supervised Learning in Spiking Neural Networks with FORCE Training.
arXiv:1609.02545 [q-bio], September 2016.

Legenstein, Robert, Markram, Henry, and Maass, Wolfgang. Input prediction and autonomous movement analysis in recurrent circuits of spiking neurons. Reviews
in the Neurosciences, 14(1-2):5–19, 2003. ISSN 03341763.

Pearlmutter, B. A. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions
on Neural Networks, 6(5):1212–1228, September 1995.
ISSN 1045-9227. doi: 10.1109/72.410363.

Legenstein, Robert, Chase, Steven M., Schwartz, Andrew B., and Maass, Wolfgang. A Reward-Modulated
Hebbian Learning Rule Can Explain Experimentally
Observed Network Reorganization in a Brain Control
Task. Journal of Neuroscience, 30(25):8400–8410, June
2010. ISSN 0270-6474, 1529-2401. doi: 10.1523/
JNEUROSCI.4284-09.2010.

Petersson, Per, Waldenström, Alexandra, Fåhraeus, Christer, and Schouenborg, Jens.
Spontaneous muscle
twitches during sleep guide spinal self-organization. Nature, 424(6944):72–75, July 2003. ISSN 0028-0836. doi:
10.1038/nature01719.
Pouget, Alexandre and Sejnowski, Terrence J. Spatial Transformations in the Parietal Cortex Using Basis Functions. Journal of Cognitive Neuroscience, 9
(2):222–237, March 1997. ISSN 0898-929X. doi:
10.1162/jocn.1997.9.2.222.

Li, Weiwei. Optimal Control for Biological Movement Systems. PhD thesis, University of California, San Diego,
January 2006.
Maass, Wolfgang and Markram, Henry. On the computational power of circuits of spiking neurons. Journal of
Computer and System Sciences, 69(4):593–616, December 2004. ISSN 0022-0000. doi: 10.1016/j.jcss.2004.04.
001.

Pouget, Alexandre and Snyder, Lawrence H. Computational approaches to sensorimotor transformations. Nature Neuroscience, 3:1192–1198, November 2000. doi:
10.1038/81469.
Rumelhart, DE, Hinton, GE, and Williams, RJ. Learning Internal Representations by Error Propagation. In
Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol 1, volume 1. MIT Press
Cambridge, MA, USA, 1986.

Maass, Wolfgang, Natschläger, Thomas, and Markram,
Henry. Real-Time Computing Without Stable States:
A New Framework for Neural Computation Based on
Perturbations. Neural Computation, 14(11):2531–2560,
November 2002. ISSN 0899-7667. doi: 10.1162/
089976602760407955.

Sanner, R. M. and Slotine, J. J. E. Gaussian networks for
direct adaptive control. IEEE Transactions on Neural
Networks, 3(6):837–863, November 1992. ISSN 10459227. doi: 10.1109/72.165588.

MacNeil, David and Eliasmith, Chris. Fine-Tuning and the
Stability of Recurrent Neural Networks. PLoS ONE, 6
10

Sarlegna, Fabrice R. and Sainburg, Robert L. The Roles
of Vision and Proprioception in the Planning of Reaching Movements. Advances in experimental medicine and
biology, 629:317–335, 2009. ISSN 0065-2598. doi:
10.1007/978-0-387-77064-2 16.

Voegtlin, Thomas. Temporal Coding using the Response
Properties of Spiking Neurons. pp. 1457–1464, 2006.
Williams, Ronald J. and Zipser, David. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2):270–280, June
1989. ISSN 0899-7667. doi: 10.1162/neco.1989.1.2.
270.

Sastry, Shankar and Bodson, Marc. Adaptive Control: Stability, Convergence, and Robustness. Prentice-Hall, Inc.,
Upper Saddle River, NJ, USA, 1989. ISBN 978-0-13004326-9.

Wolpert, Daniel M. and Ghahramani, Zoubin. Computational principles of movement neuroscience. Nature
Neuroscience, 3:1212–1217, November 2000. doi: 10.
1038/81497.

Seung, H. Sebastian, Lee, Daniel D., Reis, Ben Y., and
Tank, David W. Stability of the Memory of Eye Position
in a Recurrent Network of Conductance-Based Model
Neurons. Neuron, 26(1):259–271, April 2000. ISSN
0896-6273. doi: 10.1016/S0896-6273(00)81155-1.

Wong, Jeremy D., Kistemaker, Dinant A., Chin, Alvin, and
Gribble, Paul L. Can proprioceptive training improve
motor learning? Journal of Neurophysiology, 108(12):
3313–3321, December 2012. ISSN 0022-3077. doi: 10.
1152/jn.00122.2012.

Siegelmann, H. T. and Sontag, E. D. On the Computational
Power of Neural Nets. Journal of Computer and System
Sciences, 50(1):132–150, February 1995. ISSN 00220000. doi: 10.1006/jcss.1995.1013.

Zerkaoui, Salem, Druaux, Fabrice, Leclercq, Edouard, and
Lefebvre, Dimitri. Stable adaptive control with recurrent neural networks for square MIMO non-linear systems. Engineering Applications of Artificial Intelligence,
22(4–5):702–717, June 2009. ISSN 0952-1976. doi:
10.1016/j.engappai.2008.12.005.

Slotine, J.-J. E. and Coetsee, J. A. Adaptive sliding controller synthesis for non-linear systems. International
Journal of Control, 43(6):1631–1651, June 1986. ISSN
0020-7179. doi: 10.1080/00207178608933564.
Slotine, Jean-Jacques E. and Weiping Li. On the Adaptive
Control of Robot Manipulators. The International Journal of Robotics Research, 6(3):49–59, September 1987.
ISSN 0278-3649. doi: 10.1177/027836498700600303.
Song, H. Francis, Yang, Guangyu R., and Wang, XiaoJing. Training Excitatory-Inhibitory Recurrent Neural
Networks for Cognitive Tasks: A Simple and Flexible Framework. PLOS Comput Biol, 12(2):e1004792,
February 2016. ISSN 1553-7358. doi: 10.1371/journal.
pcbi.1004792.
Sussillo, David and Abbott, L. F. Generating Coherent Patterns of Activity from Chaotic Neural Networks. Neuron, 63(4):544–557, August 2009. ISSN 0896-6273. doi:
10.1016/j.neuron.2009.07.018.
Sussillo, David and Abbott, L.F. Transferring Learning
from External to Internal Weights in Echo-State Networks with Sparse Connectivity. PLoS ONE, 7(5):
e37372, May 2012. doi: 10.1371/journal.pone.0037372.
Thalmeier, Dominik, Uhlmann, Marvin, Kappen,
Hilbert J., and Memmesheimer, Raoul-Martin. Learning
Universal Computations with Spikes. PLOS Comput
Biol, 12(6):e1004895, June 2016. ISSN 1553-7358. doi:
10.1371/journal.pcbi.1004895.
Urbanczik, Robert and Senn, Walter. Learning by the
Dendritic Prediction of Somatic Spiking. Neuron, 81
(3):521–528, February 2014. ISSN 0896-6273. doi:
10.1016/j.neuron.2013.11.030.
11

