Efficient Computation in Adaptive Artificial Spiking
Neural Networks

arXiv:1710.04838v1 [cs.NE] 13 Oct 2017

Davide Zambrano1 , Roeland Nusselder1 , H. Steven Scholte2 & Sander Bohté1
October 16, 2017

Abstract.

Artificial Neural Networks (ANNs) are bio-inspired models of neural computa-

tion that have proven highly effective. Still, ANNs lack a natural notion of time, and neural
units in ANNs exchange analog values in a frame-based manner, a computationally and energetically inefficient form of communication. This contrasts sharply with biological neurons
that communicate sparingly and efficiently using binary spikes. While artificial Spiking Neural
Networks (SNNs) can be constructed by replacing the units of an ANN with spiking neurons
[1, 2], the current performance is far from that of deep ANNs on hard benchmarks and these
SNNs use much higher firing rates compared to their biological counterparts, limiting their efficiency. Here we show how spiking neurons that employ an efficient form of neural coding
can be used to construct SNNs that match high-performance ANNs and exceed state-of-the-art
in SNNs on important benchmarks, while requiring much lower average firing rates. For this,
we use spike-time coding based on the firing rate limiting adaptation phenomenon observed
in biological spiking neurons. This phenomenon can be captured in adapting spiking neuron
models, for which we derive the effective transfer function. Neural units in ANNs trained with
this transfer function can be substituted directly with adaptive spiking neurons, and the resulting
Adaptive SNNs (AdSNNs) can carry out inference in deep neural networks using up to an order
of magnitude fewer spikes compared to previous SNNs. Adaptive spike-time coding additionally allows for the dynamic control of neural coding precision: we show how a simple model of
arousal in AdSNNs further halves the average required firing rate and this notion naturally extends to other forms of attention. AdSNNs thus hold promise as a novel and efficient model for
neural computation that naturally fits to temporally continuous and asynchronous applications.
Introduction While the currently best-performing SNNs use high firing rates (on average hundreds of Hertz) to cover the dynamic range of corresponding analog neurons [1, 2], in biology,
real neurons use on average 1-5Hz[3] and sensory neurons are known to adaptively control the
1

number of spikes that are used to efficiently cover large dynamic ranges [4]. This adaptive behaviour can be captured with fast spike-triggered adaptation in Leaky-Integrate-and-Fire neuron
models, or corresponding Spike Response Models (SRMs)[5, 6, 7] including the Adaptive Spiking Neuron models (ASN)[6]. ASNs can implement adaptive-spike coding as a neural coding
scheme that maps analogue values to sequences of spikes, where the thresholding mechanism
carries out an online analog-to-digital conversion of the analog signal computed in the neuron
unit.
Adaptive spike-time coding is illustrated in Fig. 1a: expressed as an SRM, the input to
a neuron j consists of an external input Vinj,j (t), and a series of spikes from input neurons i
impinging at times ti each contributing a postsynaptic potential (PSP) weighted by synaptic
efficacy wij . The PSP is modelled as a normalised kernel κ(t − ti ) multiplied by the height of
the spike h. These contributions result in the neuron’s activation Sj (t):
Sj (t) = Vinj,j (t) +

XX

wij hκ(t − ti ).

(1)

ti

i

The activation Sj (t) corresponds to the membrane potential of a spiking neuron absent spiking. A spike emitted by neuron j at time tj resets the membrane potential by subtracting a
scaled refractory kernel ϑ(tj )η(t), this kernel is added to the total refractory response Ŝj (t)
that is computed as the sum of scaled refractory kernels; Ŝj (t) thus approximates the rectified
activation [Sj (t)]+ . A spike is emitted when the the membrane potential – the difference between Sj (t) and Ŝj (t) – exceeds half the threshold ϑ(t) (as in [8], the threshold ϑ is redefined
for convenience). Spike-triggered adaptation is incorporated into the model by multiplicatively
increasing the variable threshold ϑ(t) at the time of spiking with a decaying kernel γ(t):
ϑj (t) = ϑ0 +

X

mf ϑ(tj )γ(t − tj ),

(2)

tj

where ϑ0 is the resting threshold and the multiplicative parameter mf controls the speed of the
firing rate adaptation. This adaptive spiking mechanism effectively maps an activation Sj to
a normalised average contribution y(Sj ) to the next neuron’s activation Sk as a rectified halfsigmoid-like transfer function (Fig. 1a, inset):
*
yj = f (Sj ) =

+
X

κ(t − tj ) ,

(3)

tj

We derive an analytical expression for the shape of the transfer function f (S) to map spiking
neurons to analog neural units (see Methods). The use of exponentially decaying kernels for
η(t), γ(t) and κ(t) allows the neuron model to be computed with simple dynamical systems
where each impinging spike adds an impulse response function of height h multiplied by the
weight associated with the connection.
2

a

y
(t)

Sj (t)

3

Sk (t)

y(t)

2
1
0

b

c
σLow FR

0

2

S

3

4

5

High
Low

300
200

-1

100

-2

0

1

y(t)

-1

0

3

σHigh FR

0

-2

400

1

Firing Rate

1

0

1

2

3

4

5

3

4

5

High
Low

2
1

0

Time (ms)

0

500

0

1

2

S

Figure 1: a, Adaptive spike-time coding. Spikes are generated when the difference between
the activation S(t) and the refractory response Ŝ(t) exceeds 12 ϑ(t). Emitted spikes contribute
a sum of PSPs to the target neuron’s potential. Inset: effective transfer function. b, Example
of encoding a step-function activation: the resultant normalised postsynaptic contribution y(t)
(blue line) is plotted for two different values of the resting threshold ϑ0 = [0.5, 0.1]. The
default neural coding precision (σLow FR ) increases for a lower threshold ϑ0 corresponding to a
higher firing rate (σHigh FR ). c, Top: two values of ϑ0 result in two different firing rate curves;
bottom: the effective transfer function for two values of ϑ0 , the same approximated value can be
represented with different precisions (standard-deviation across the average), and thus different
firing rates, by controlling spike height h.
The speed of adaptation, mf , and the spike height h together control the precision of the
spike-based neural coding, where neural coding precision is measured as the standard-deviation
of y(t) around the mean response to a fixed input. As illustrated in Fig. 1b, a same-but-moreprecise spike-time encoding can be realised by changing the adaptation parameters mf , ϑ0 to
increase the firing rate for a given stimulus intensity, while simultaneously reducing the impact
3

of spikes on target neurons by decreasing h. An ASN can thus map different stimulus-to-firingrate curves (Fig. 1c, top) to the same transfer function but with different neural coding precision
(Fig. 1c, bottom).
We construct adaptive SNNs – AdSNNs– comprised of ASN neurons using adaptive spikecoding similar to the approach pioneered in [2]. First, ANNs are constructed with analog neural
units that use the derived half-sigmoid-like transfer function, both for fully connected feedforward ANNs and for various deep convolutional neural network architectures. We train these
ANNs for standard benchmarks of increasing difficulty (SONAR, IRIS, MNIST, CIFAR-10/100,
and the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC 2012) benchmarks).
Corresponding AdSNNs are then obtained by replacing the ANNs’ analog units with ASNs
(illustrated in Fig. 2).

Output

Deep AdSNN

Deep ANN
0

1

2

3

4

5

6

7

8

9

ʘ∑

Fully Connected

ʘ∑

Fully Connected

Convolution

0

Input

ASN fun

2

3

4

5

6

7

8

9

∑

0

500

Time (ms)

∑
0

500

Time (ms)

feature maps
ʘ∑

Convolution

ASN fun

1

ASN fun
activation

filters
ʘ∑
pixels

ASN fun

∑

0

spikespost
∑
spikespre

0

500

Time (ms)

500

Time (ms)

Figure 2: Adaptive Spiking Neural Network conversion schematic. Left. During training of a
deep ANN, the output of a convolutional or fully connected layer is passed through the ASN
transfer function. In ANNs, every layer performs a series of weighted sums of the inputs as
each analog input is multiplied by its analog weight. Right. For classification, the analog ANN
units are converted in ASNs to obtain an SNN. All the layers perform similarly, but now binary
spikes are conveyed across the network and corresponding weights are simply added to the next
ASN.
For suitable choices of adaptation parameters (Table SI1), the AdSNNs exactly match performance to the original ANNs as measured on the test set (Table 1). Since we trained high-

4

Table 1: Performance(%), Matching Firing Rate (FR) (Hz) and Matching Time (MT) (ms). (*): for
CIFAR-10, [9] report performance of 83.54 @ 143Hz. Current (Poisson) SNN performance and firing rate
is compared against trained ANN and AdSNN performance. State of the art is denoted with bold font; no
current SNN state of the art exists.

Current SNN

ANNs

AdSNNs

DataSet
Perf.

FR

Perf.

Perf.

FR

MT

IRIS

-

-

98.67

98.67 ± 2.8e−15

51

269

SONAR

-

-

89.42

89.89 ± 1.1

26

119

MNIST

99.12[9]

9.47[9]

99.56

99.56 ± 4.6e−3

67

291

CIFAR-10

87.86[10]∗

-

89.90

89.86 ± 4.2e−2

68

372

CIFAR-100

55.13[9]

-

64.53

64.21 ± 2.4e−2

59

507

ILSVRC-2012

51.8[9]

93[9]

62.98

62.97 ± 4.8e−2

66

347

Table 2: High precision AdSNNs (from Table 1) Matching Time (MT) (ms) to achieve current (Poisson) SNN performance. Performance(%), Firing Rate (FR) (Hz) and Matching Time (MT) (ms) are also
provided for low precision AdSNNs that have similar current (Poisson) SNN performance.

AdSNNs

Current SNN
DataSet
Perf.
MNIST

Low FR AdSNNs

Table1

99.12[9]
[10]∗

CIFAR-10

87.86

CIFAR-100

55.13[9]

ILSVRC-2012

51.8[9]

FR

MT

Perf.

FR

MT

9.47[9]

131

99.12 ± 4.14e−2

10

209

-

276

88.52 ± 2.9

22

304

-

286

58.08 ± 9.5

16

328

93[9]

256

53.77 ± 5.4

12

338

performance ANNs, the AdSNNs exceed previous state-of-the-art SNN performance on all
benchmarks while requiring substantially lower average firing-rates, in the range of 24-68 Hz.
Note that on some benchmarks, the AdSNNs exceed the ANNs performance, presumably because the AdSNNs compute an average from sampled neural activity[9] that correctly separates
some additional inputs. As any SNN, the time-based communication in AdSNNs incurs latency,
measured as the time required between onset of the stimulus and the time when the output neurons are able to classify at the level of the network’s analog counterpart. For AdSNNs, this
latency (MT) is of order 300ms, and mainly depends on the PSP decay time (50ms here); faster
decay times result in lower latency, at the expense of increased firing rates (see Fig. SI1a).
However, state-of-the-art accuracies are already reached after about 200ms (Table 2).

5

Table 3: AdSNNs versus Arousal AdSNNs. Performance(%), Matching Firing Rate (FR) (Hz) and Matching Time (MT) (ms).

AdSNNs

Arousal AdSNNs

DataSet
Perf.

FR

MT

Perf.

FR

MT

IRIS

98.67 ± 2.8e−15

51

269

98.67 ± 0

18

484

SONAR

89.89 ± 1.1

26

119

90.25 ± 0.3

15

393

MNIST

−3

99.56 ± 4.6e

67

291

99.56 ± 0

12

498

CIFAR-10

89.86 ± 4.2e−2

68

372

89.88 ± 4.5e−2

34

546

CIFAR-100

−2

64.21 ± 2.4e

59

507

64.57 ± 0.12

50

691

ILSVRC-2012

62.97 ± 4.8e−2

66

347

63.02 ± 0.1

42

575

We further find that AdSNNs exhibit a gradual and graceful performance degradation when
the neural coding precision is decreased, by changing the ASN adaptation parameters such that
the firing rate is lowered while increasing h (Fig. 3a): performance equal to previously reported
state-of-the-art can be reached with even lower firing rates (10-22Hz, Table 2). Increasing the
PSP decay time further lowers the required firing rate to achieve AdSNN performance matching
ANNs (Fig. SI1b), at the expense of increased latency (Fig. SI1a)1 .
The tuneable relationship between firing rate and neural coding precision can be exploited
to further increase efficiency by selectively manipulating this trade off as a particular form of
attention. It is well known that for stable sensory inputs attention in the brain manifests as
enhanced firing in affected neurons [12]. One purported effect of this mechanism is to improve
neural coding precision on demand, for instance in specific locations, for a brief amount of time,
and only if needed [13, 14]. Such attention would allow the brain to process information at a
low default precision when possible and increase firing rate only when necessary, potentially
saving a large amount of energy.
We implement attention in the form of arousal affecting all neurons in the network simultaneously. Arousal is engaged selectively based on classification uncertainty: the neural coding
precision is increased from a low base level only for samples deemed uncertain, as illustrated
in Fig. 3b. Uncertain inputs are identified by accumulating the two highest valued outputs for
50ms after a network-dependent fixed waiting time (dashed vertical line in Fig. 3b). Arousal
is engaged only if the averaged difference between these two outputs does not exceed a hard
threshold as determined from the training set; engaging arousal causes a brief deterioration of
1 We

excluded [11] as there a binary neural network is simulated without notion of time while requiring many more

binary neurons as compared to similarly performing ANNs.

6

80

70

70

40

60

15

b

10

56 Hz

15

0

0
-5

10

0 100 200 300 400 500

56 Hz

5

Low fidelity

High fidelity

0

15

0

5

-5
-10

12 Hz

10

5

IL
S
20 VR
12 C

20

Firing Rate [Hz]

0

80

10

00

60

ILSVRC-2012

-1

40

20

C

20

30

-1

CIFAR-100

40

IF
AR

45

50

C

50

60

IS

55

40

IF
AR

60

20

AR

65

50
0
64
62
60
58
56
54
52
0

IR

60

IS
T

40

N

20

CIFAR-10

M

85
0

60

60

MNIST

N

90

Full High FR
Selective Arousal

80

SO

95

40
0

c

90

Firing Rate [Hz] Selectivity [%]

100

Final spiking accuracy [%]

a

-10

15

4

10

9

5

0

0 100 200 300 400 500

S
12 Hz

56 Hz

4
9

0

-5

-5

-10

-10

0 100 200 300 400 500

Time (ms)

0 100 200 300 400 500

Time (ms)

Figure 3: a, accuracy on the test sets of four datasets (MNIST, CIFAR-10 and CIFAR-100,
ILSVRC-2012) as a function of average firing rate: lower coding precision decreases accuracy.
b, Classifying with attention. Ease of classification is measured as the distance between the
internal value S of the winning output neuron and the second highest output neuron (line-plots).
Top row: easy example that is correctly classified both at low precision (right) and high precision
(left). Bottom row: ambiguous samples can be disambiguated by applying arousal to increase
precision in the network. c, Efficiency improvement when Arousal is applied to hard-to-classify
images only: the same classification accuracy is reached using a significantly lower average
firing rate over the test set. Cyan bars designate percentage of inputs selected by the Arousal
criterion.
classification accuracy before quickly settling to higher performance (Fig. SI2). Using this simple model of attentional modulation, the number of spikes required for overall classification is
effectively halved (Fig. 3c), while latency increases as the selected inputs require additional
time for classification (see Table 3). The uncertainty based arousal is also engaged more or less
frequently depending on the accuracy of the model (blue markers in Fig. 3c), and the benefit is
thus greatest for networks with the highest absolute accuracy.

7

Discussion.

A number of recent studies have suggested that spiking neurons implement an

efficient analog-to-digital conversion similar to the mechanisms proposed here [15, 6, 8]. While
population coding is a popular concept to explain how pools of spiking neurons can approximate
analog signals with arbitrary precision [16], the results presented here show that firstly, the
required neural coding precision in many deep neural networks can be satisfied with a single and
plausible spiking neuron model at reasonable firing rates; and secondly, neural coding precision
can further be increased or decreased by manipulating the firing rate inversely with a form of
global synaptic efficacy modulation. This provides an alternative explanation for the observed
attentional modulation of firing rates, and more detailed location-based or object-based attention
algorithms can be studied to increase neural efficiency further.
As presented, the half-sigmoid-like derived transfer function holds for isomorphic spikes
that can be communicated with a binary number. A rectified linear (ReLU) transfer function
can be constructed by scaling the impact of individual spikes on postsynaptic targets with the
presynaptic adaptation magnitude at the time of spiking. Using such a transfer function slightly
improves performance and speeds up convergence, at the expense of effectively communicating an analog value with each spike (not shown). From a biological perspective, such neural
communication would require a tight coupling between neural adaptation and phenomena like
synaptic facilitation and depression [17], which at present has not been examined in this context.
From a computer science perspective, the efficiency penalty in terms of bandwidth may be limited as spike-based neuromorphic simulators like SpiNNaker already use sizable addressing bits
for each spike [18]; the computationally simple addition of spikes to the target neuron however
is replaced by a conventional multiply-add operation.
AdSNNs explicitly use the time-dimension for communication and implicitly exploit temporal correlations in signals for sparse spike-time coding; in contrast, ANNs applied to temporal
problem domains sequentially and synchronously sample their inputs in a time-stepped manner,
recomputing the network for each successive timestep. The actual extraction of computational
efficiency from sparsely active SNNs is a separate challenge. Sparse activity and computationally cheap connection updates are accompanied by a more complex and state-based neuron
model that is updated more frequently. Networks with a high fan-in fan-out architecture, like
the brain, benefit most from this trade-off; current deep learning architectures in contrast are
characterised by a low degree of fan-in fan-out, except for the last layers which are typically
fully connected. Hybrid analog/spiking neural network approaches may be most efficient for
the implementation of these architectures. Additionally, like other state-based neural networks,
and in contrast to feedforward ANN architectures, networks of adapting spiking neurons require
per-neuron local memory to store state information such as potential and adaptation values. The

8

availability of sufficient local memory is thus necessary to best extract efficiency from sparse
spiking activity. Since current GPU-based deep learning accelerators are lacking in this regard,
at least for the large state-based neural networks considered, neuromorphic hardware seems the
most suitable approach for the implementation of large SNNs.

Methods
Adaptive Spiking Neurons. In the ASN, the kernel κ(t) is computed as the convolution of a
spike-triggered postsynaptic current (PSC) with a filter φ, decaying exponentially respectively
with time constants τφ and τβ ; and the input signal Vinj,j (t) is similarly computed from a current
injection Iinj,j (t). The adaptation kernel γ(t) decays with time-constant τγ .
The AdSNNs’s are created by converting standard Deep Neural Networks[2] trained with a
mathematically derived transfer function f (S) of the ASN (full derivation in SI), defined as the
function that maps the activation S to the average post-synaptic contribution. This has the form:


h


− c0 + h/2 ,
f (S) = max 0,
2
−
1
exp cc13 ·S+c
·S+c4
where,
c1 = 2 · mf · τγ2 ,
c2 = 2 · ϑ0 · τη · τγ ,
c3 = τγ · (mf · τγ + 2 · (mf + 1) · τη ),
c4 = ϑ0 · τη · (τγ + τη ),
h

c0 =
exp



c1·ϑ0 /2+c2
c3·ϑ0 /2+c4



,
−1

are constants computed from the neuron parameters setting, and h defines the spike size. Here,
by normalising f (S) to 1 when S = 1, h becomes a scaling factor for the network’s trained
weights, allowing communication with binary spikes.
Adaptive Spiking Neural Networks (AdSNNs). Analog units using f (S) as their transfer
function, AANs, in trained ANNs can be replaced directly and without modification with ASNs.
In the presented results, the adaptation kernel decays with τγ = 15ms, the membrane filter with
τφ = 5ms, the refractory response with τη = 50ms and the PSP with τβ = 50ms, all roughly
corresponding to match biological neurons, and ϑ0 = mf . Batch Normalization (BN)[19] is
used to avoid the vanishing gradient problem[20] for saturating transfer functions like halfsigmoids and to improve the network training and regularisation. After training, the BN layers
9

are removed and integrated into the weights’ computation [10]. A BN-AAN layer is also used
as a first layer in all the networks to convert the inputs into spikes. When converting, biases are
added to the post-synaptic activation. Max and Average Pooling layers are converted by merging
them into the next ASN-layer: the layer activation S is computed from incoming spikes, then
the pooling operator is applied and the ASN-layer computes spikes as output. The last ASN
layer acts as a smoothed read-out layer with τφ = 50ms, where spikes are converted into
analog values for classification. The classification is performed as in the ANN network, usually
using SoftMax: at every time-step t the output with highest value is considered the result of the
classification.
ANN training. We trained ANN with AANs on widely used datasets: for feedforward ANNs,
IRIS and SONAR; and for deep convolutional ANNs: MNIST, CIFAR-10, CIFAR-100 and
ILSVRC-2012. All the ANNs are trained using Keras2 with Tensorflow3 as its backend. We
used categorical cross-entropy as a loss function with Adam[21] as the optimiser, except for
ILSVRC-2012 where we used Stochastic Gradient Decent with Nesterov (learning rate = 1e−3,
decay = 1e − 4 and momentum = 0.9). Consistent with the aim of converting high performance
ANNs into AdSNNs, for each dataset, we selected the model at the training epoch where it
performed best on the test set.
We trained a [4 − 60 − 60 − 3] feedforward ANN on the IRIS dataset: IRIS is a classical
non-linearly separable toy dataset containing 3 classes – 3 types of plants – with 50 instances
each, to be classified from 4 input attributes. Similarly, for the SONAR dataset[22] we used a
[60−50−50−2] ANN to classify 208 entries of sonar signals divided in 60 energy measurements
in a particular frequency band in two classes: metal cylinder or simple rocks. We trained both
ANNs for 800 epochs and obtained competitive performance.
The deep convolutional ANNs are trained on standard image classification problems with
incremental difficulty. The simplest is the MNIST dataset[23], where 28×28 images of handwritten digits have to be classified. We used a convolutional ANNs composed of [28×28 −
c64×3 − m2 − 2×(c128×3 − c) − m2 − d256 − d50 − 10], where cN×M is a convolutional
layer with N feature maps and a kernel size of M×M , mP is a max pooling layer with kernel
size P×P , and dK is a dense layer with K neurons. Images are pre-normalised between 0 and
1, and the convolutional ANN was trained for 50 epochs.
The CIFAR-10 and CIFAR-100 data sets[24] are harder benchmarks, where 32×32 colour
images have to be classified in 10 or 100 categories respectively. We use a VGG-like architecture[25]
2 https://keras.io/
3 https://www.tensorflow.org/

10

with 12 layers: [32×32 − 2×(c64×3) − m2 − 2×(c128×3) − m2 − 3×(c256×3) − m2 −
3×(c512×3) − m2 − d512 − 10] for CIFAR-10 and [32×32 − 2×(c64×3) − m2 − 2×(c128×3) −
m2 − 3×(c256×3) − m2 − 3×(c1024×3) − m2 − d1024 − 100] for CIFAR-100. Dropout[26]
was used in the non-pooling layers (0.5 in the top fully-connected layers, and 0.2 for the first
500 epochs and 0.4 for the last 100 in the others). Images were converted from RGB to YCbCr
and then normalised between 0 and 1.
The ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)[27] is a large-scale image classification task with over 15 million labeled high-resolution images belonging to roughly
22, 000 categories. The 2012 task-1 challenge was used, a subset of ImageNet with about 1000
images in each of 1000 categories. We trained a ResNet-18 architecture in the Identity-mapping
variant [28] for 100 epochs and the top-1 error rate is reported. As in[25], we rescaled the images to a resolution of 256×256 pixels and then performed random cropping during training and
centre cropping for testing.
AdSNN evaluation.

The AdSNNs are evaluated in simulations with 1ms timesteps, where

inputs are persistently presented for 500ms (identical to the method used in [2]). The Firing
Rate (FR) in Table 1 is computed as the average number of spikes emitted by a neuron, for
each image, in this time window. The time window is chosen such that all output neurons
reach a stable value; we defined the Matching Time (MT) as the time to which 99% of the
maximum classification accuracy is reached for each simulation. From the MT to the end of the
time interval, the standard deviation of the accuracy is computed to evaluate the stability of the
network’s response. Each dataset was evaluated for a range of ϑ0 , mf values of [0.03, 1.0] and
the minimum firing rate needed to match the ANN performance is reported. All the AdSNNs
simulations are run on MATLAB in a modified version of the MatConvNet framework4 .
Arousal. For Arousal, we highlight uncertain inputs by increasing firing-rate and corresponding precision. The network is simulated with ϑ0 set to ϑ0−lp , the standard low-precision parameter; if the input is selected by the arousal mechanism, this parameter is set to high precision
value: ϑ0−hp (and mf is changed identically). Selection is determined by accumulating the
winning and the 2nd-highest outputs for 50ms starting from a pre-defined tsa specific for each
dataset. If the difference between these two outputs exceeds a threshold θA , the input is not
highlighted – θA is estimated by observing those images that are not correctly classified when
the precision is decreased on the training set. The Arousal method selects more images than
needed: we defined Selectivity as the proportion of highlighted images (Table SI1 ). In addition,
4 http://www.vlfeat.org/matconvnet/

11

θA increases linearly with the accumulation time interval as θA = p1 ·(t−tsa )+p2 , while Selectivity decreases exponentially. We report results for the minimum firing rate recorded for each
dataset (Fig. 3c), which is obtained at a specific ϑ0−lp : in fact, starting from very low precision
leads to higher Selectivity, which in turn results in a higher average firing rate. The parameter
ϑ0−hp is chosen as the lowest precision needed to match the ANN performance. Table SI1 reports the values of Selectivity, tsa , ϑ0−lp , ϑ0−hp , p1 , p2 for each dataset. Note that, since deeper
networks need more time to settle to the high precision level, we extended the simulation time
for these networks (see Table 1).

Supplementary Information
To convert a trained Artificial Neural Network (ANN) into an Adaptive Spiking Neural Network
(AdSNN), the transfer function of the ANN units needs to match the behaviour of the Adaptive
Spiking Neuron (ASN). The ASN transfer function is derived for the general case of τη 6= τγ
using an approximation of the ASN behaviour.
Derivation of the ASN activation function

We consider a spiking neuron with activation

S(t) that is constant over time, and the refractory response Ŝ(t) approximates S(t) using a
variable threshold ϑ(t). Whenever S − Ŝ(t) > 0.5·ϑ(t), the neuron emits a spike of fixed height
h to the synapses connecting to the target neurons, and a value of ϑ(tf ) is added to Ŝ, with tf the
time of the spike. At the same time, the threshold is increased by mf ϑ(tf ). The post-synaptic
current (PSC) in the target neuron is then given by I(t), which is convolved with the membrane
filter φ(t) to obtain the contribution to the post-synaptic potential; a normalized exponential
filter φ(t) with short time constants τφ will just smooth the high-frequency components of I(t).
We derive the transfer function that maps the activation S to the PSC I of the target neuron. We
recall the ASN model here, elaborating the SRM to include the current-to-potential filtering:

 i
XX
ts − t
,
(4)
PSC:
I(t) =
wi exp
τβ
i
i
ts

activation:
threshold:
refractory response:

S(t) = (φ ∗ I)(t),


X
ts − t
ϑ(t) = ϑ0 +
mf ϑ(ts ) exp
,
τγ
ts


X
ts − t
Ŝ(t) =
ϑ(ts ) exp
,
τη
t

(5)
(6)
(7)

s

where tis denotes the timing of incoming spikes that the neuron receives and ts the timing of
outgoing spikes.

12

Since the variables of the ASN decay exponentially, they converge asymptotically. For a
given fixed size current injection, we consider a neuron that has stabilised around an equilibrium,
that is Ŝ(t) and ϑ(t) at the time of a spike always reach the same values. Let these values be
denoted as Ŝl and ϑl respectively. Then, ϑ(tf ) = ϑl and Ŝ(tf ) = Ŝl for all tf . The PSC I(t)
also always declines to the same value, Il , before it receives a new spike. Setting t = 0 for the
last time that there was a spike, we can rewrite our ASN equations, Equations (4), (5), (6) and
(7), for τβ = τη and 0 < t < tf to:
Ŝ(t) = Ŝl e

− τtη

+ ϑl e

− τtη

ϑ(t) = ϑ0 + (ϑl − ϑ0 )e
I(t) = Il e

− τtη

− τtη

+ he

,

− τtγ

+ mf ϑl e

− τtγ

,

.

The transfer function f (S) of the ASN is a function of the value of S; f (S) should be a bit
larger than Il since that is the lowest value of I(t), and we are interested in the average value of
I(t) between two spikes: f (S) = Iaverage .
Since we are in a stable situation, the time between each spike is fixed; we define this time
as te . Thus, if the last spike occurred at t = 0, the next spike should happen at t = te . This
implies that Ŝ(t), ϑ(t) and I(t) at t = te must have reached their minimal values Ŝl , ϑl and Il
respectively.
To obtain the activation function f (S), we solve the following set of equations:
Ŝ(te ) = Ŝl ,
ϑ(te ) = ϑl ,
I(te ) = Il ,
and by noting that the neuron only emits a spike when S − Ŝ(t) > 21 ϑ, we also have:
S − Ŝl =

1
ϑl .
2

We first notice:
− τteη

he

1−e

− τteη

= Il .

(8)

We now want an expression for ϑl :
ϑ0 + (ϑl − ϑ0 )e
ϑ0 − ϑ0 e

e
− τtγ

e
− τtγ

+ mf ϑl e

e
− τtγ

e
− τtγ

= ϑ l − mf ϑ l e

= ϑl ,

− ϑl e

e
− τtγ

.

We can rewrite this to:
ϑ0

1−e

e
− τtγ

1 − (mf + 1)e
13

e
− τtγ

= ϑl .

(9)

Using equations S − Ŝl = 12 ϑl and Ŝ(te ) = Ŝl , we get:
1
1
− te
− te
(S − ϑl )e τη + ϑl e τη = S − ϑl ,
2
2
− τteη

e

(2S + ϑl ) = 2S − ϑl .

Inserting Equation 9 gives:
e

− τteη

2S + ϑ0

1−e

e
− τtγ

1 − (mf + 1)e

!

1−e

= 2S − ϑ0

e
− τtγ

e
− τtγ

1 − (mf + 1)e

e
− τtγ

.

This can be rewritten to:
e

− τteη



2S(1 − (mf + 1)e

e
− τtγ

) + ϑ0 (1 − e

e
− τtγ


− te
) = 2S(1 − (mf + 1)e τγ )
− ϑ0 (1 − e

(2S + ϑ0 )e

− τteη

− (2S(mf + 1) + ϑ0 )e

e
− τtγ

e

− τteη

e
− τtγ

),

= 2S − ϑ0
e
− τtγ

− 2S(mf + 1)e
+ ϑ0 e
(2S + ϑ0 )e

− τteη

− (2S(mf + 1) + ϑ0 )e

−te ( τ1γ

+ τ1η

)

e
− τtγ

,

+ (2S(mf + 1) − ϑ0 )e

e
− τtγ

= 2S − ϑ0 .

(10)

Approximation of the AAN activation function In the general case of τη 6= τγ , a (second
order) Taylor series expansion can be used to approximate the exponential function:
ex ≈ 1 + x +

x2
,
2

for x close to 0. We can use this in our previous equation:
(2S + ϑ0 )(1 −

1
1
te + 2 t2e )
τη
2τη

1
1 1
1
1
+ )te + ( + )2 t2e )
τγ
τη
2 τγ
τη
1 2
1
+ (2S(mf + 1) − ϑ0 )(1 − te + 2 te )
τγ
2τγ

− (2S(mf + 1) + ϑ0 )(1 − (

= 2S − ϑ0 .

14

We need a few steps to isolate te :
(2S + ϑ0 )(−

1
1
te + 2 t2e )
τη
2τη

1
1
1 1
1
+ )te + ( + )2 t2e )
τγ
τη
2 τγ
τη
1
1 2
+ (2S(mf + 1) − ϑ0 )(− te + 2 te ) = 0,
τγ
2τγ
1
1
(2S + ϑ0 )(− + 2 te )
τη
2τη
1
1
1 1
1
− (2S(mf + 1) + ϑ0 )(−( + ) + ( + )2 te )
τγ
τη
2 τγ
τη
1
1
+ (2S(mf + 1) − ϑ0 )(− + 2 te ) = 0,
τγ
2τγ
− (2S(mf + 1) + ϑ0 )(−(

((2S + ϑ0 )

1
1 1
1
− (2S(mf + 1) + ϑ0 ) ( + )2
2τη2
2 τγ
τη
1
+ (2S(mf + 1) − ϑ0 ) 2 )te
2τγ
1
1
1
= (2S + ϑ0 ) − (2S(mf + 1) + ϑ0 )( + )
τη
τγ
τη
1
+ (2S(mf + 1) − ϑ0 ) ,
τγ

(−S · mf

1
1
1
− (2S(mf + 1) + ϑ0 )
− ϑ0 2 )te
τη2
τγ τη
τγ
1
1
= −2S · mf
− 2ϑ0 ,
τη
τγ

(−S · mf τγ2 − (2S(mf + 1) + ϑ0 )τγ τη − ϑ0 τη2 )te
= −2S · mf τγ2 τη − 2ϑ0 τγ τη2 .
This leads to our expression for te :
2τγ τη (S · mf τγ + ϑ0 τη )
.
S · τγ (mf τγ + 2(mf + 1)τη ) + ϑ0 τγ τη + ϑ0 τη2

te =

We now insert this expression in Equation 8 and get:
h

Il (S) =
e

te
τη

−1

h

=
exp



2τγ (S·mf τγ +ϑ0 τη )
S·τγ (mf τγ +2(mf +1)τη )+ϑ0 τγ τη +ϑ0 τη2



.
−1

To make sure that our activation function f (S) is 0 at S = ϑ0 /2 we choose our activation
function to be:
f (S) = Il (S) − Il (

ϑ0
h


)=
− c,
2mf τγ2 S+2ϑ0 τη τγ
2
exp τγ (mf τγ +2(mf +1)τη )S+ϑ0 τγ τη +ϑ0 τ 2 − 1
η

for S >

ϑ0
2

and f (S) = 0 for S <=

ϑ0
2

with c =

15

Il ( ϑ20 ).

(11)

Parameters used in the Arousal attention method
Table SI4: Parameters used in for the Arousal attention method.
Selectivity(%)

tsa (ms)

ϑ0−lp

ϑ0−hp

p1

p2

IRIS

32.00

150

0.80

0.17

2.1864

−342.2727

SONAR

44.23

150

1.30

0.35

3.7591

−525.0000

MNIST

1.13

200

0.60

0.12

2.1121

−405.7576

CIFAR-10

48.07

250

0.15

0.05

7.7262

−1.87e+03

CIFAR-100

70.50

350

0.10

0.03

7.0000

−2.42e+03

LSVRC-2012

76.64

350

0.15

0.05

2.5536

−895.1786

DataSet

Supplementary results
1

b

Classification Error

25
50
75
100
150

200
150

0.5

0
0

250

Firing Rate (Hz)

a

100

200

Time (ms)

50
0

400

25 50 75 100

(ms)

150

Figure SI4: Effects of τη on MNIST. a. The classification error over time is shown for increasing
values of τη s, [25, 50, 75, 100, 150]ms. Note that changing τη changes the transfer function
shape, and thus different networks were trained. The plotted results are obtained with ϑ0 = 0.05.
MT visibly increases for longer τη s. b. Networks’ firing rates. Longer τη s require less spikes to
approximate a signal.

16

tsa

0.02

MNIST Low Precision
MNIST Arousal

0.01

Classification Error

0
100

150

200

250

300

350

400

450

0.2

500

CIFAR10 Low Precision
CIFAR10 Arousal

0.15
0.1
200

250

300

350

400

450

500

550

600

0.42

650

700

CIFAR100 Low Precision
CIFAR100 Arousal

0.4
0.38
0.36
300

350

400

450

500

550

600

0.42

650

700

ImageNet Low Precision
ImageNet Arousal

0.4
0.38
0.36
300

400

500

600

Time (ms)

700

800

900

1000

Figure SI5: Classification error over time. The effect of the Arousal method on the classification
error is reported for MNIST, CIFAR-10, CIFAR-100, and the Imagenet LSVRC-2012. The vertical line denotes the moment in time, tsa , where the outputs start being accumulated. Selection
for Arousal is then determined 50ms later. The increase of the firing rate on selected images
causes a brief loss of accuracy, after which a lower classification error is reached.

17

References
[1] Cao, Y., Chen, Y. & Khosla, D. Spiking deep convolutional neural networks for energyefficient object recognition. International Journal of Computer Vision 113, 54–66 (2015).
[2] Diehl, P. et al. Fast-classifying, high-accuracy spiking deep networks through weight and
threshold balancing. In IEEE International Joint Conference on Neural Networks (IJCNN),
1–8 (2015).
[3] Attwell, D. & Laughlin, S. An energy budget for signaling in the grey matter of the brain.
J. Cerebral Blood Flow & Metabolism 21, 1133–1145 (2001).
[4] Fairhall, A., Lewen, G., Bialek, W. & de Ruyter van Steveninck, R. Efficiency and ambiguity in an adaptive neural code. Nature 412, 787–792 (2001).
[5] Gerstner, W. & Kistler, W. Spiking Neuron Models: Single Neurons, Populations, Plasticity (Cambridge University Press, 2002).
[6] Bohte, S. Efficient Spike-Coding with Multiplicative Adaptation in a Spike Response
Model. In Advances in Neural Information Processing (NIPS), vol. 25, 1844–1852 (2012).
[7] Pozzorini, C., Naud, R., Mensi, S. & Gerstner, W. Temporal whitening by power-law
adaptation in neocortical neurons. Nature Neuroscience 16, 942–948 (2013).
[8] Yoon, Y. LIF and Simplified SRM Neurons Encode Signals Into Spikes via a Form of
Asynchronous Pulse Sigma-Delta Modulation. IEEE Transaction on Neural Networks
and Learning Systems (TNNLS) 1–14 (2016).
[9] Hunsberger, E. & Eliasmith, C. Training spiking deep networks for neuromorphic hardware. preprint arXiv:1611.05141 (2016).
[10] Rueckauer, B., Lungu, I.-A., Hu, Y. & Pfeiffer, M. Theory and tools for the conversion of
analog to spiking convolutional neural networks. preprint arXiv:1612.04052 (2016).
[11] Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences 201604850 (2016).
[12] Roelfsema, P., Lamme, V. & Spekreijse, H. Object-based attention in the primary visual
cortex of the macaque monkey. Nature (1998).
[13] Saproo, S. & Serences, J. T. Spatial attention improves the quality of population codes in
human visual cortex. Journal of neurophysiology 104, 885–895 (2010).

18

[14] Friston, K. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience 11, 127–138 (2010).
[15] Boerlin, M. & Denève, S. Spike-based population coding and working memory. PLoS
computational biology 7, e1001080 (2011).
[16] Denève, S. & Machens, C. K. Efficient codes and balanced networks. Nature Neuroscience
19, 375–382 (2016).
[17] Abbott, L. & Regehr, W. G. Synaptic computation. Nature 431, 796 (2004).
[18] Furber, S. B. et al. Overview of the spinnaker system architecture. IEEE Transactions on
Computers 62, 2454–2467 (2013).
[19] Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML),
448–456 (2015).
[20] Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and
problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based
Systems 6, 107–116 (1998).
[21] Kingma, D. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 (2014).
[22] Gorman, R. & Sejnowski, T. Analysis of hidden units in a layered network trained to
classify sonar targets. Neural Networks 1, 75–89 (1988).
[23] Lecun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278–2324 (1998).
[24] Krizhevsky, A. Learning multiple layers of features from tiny images. Master’s thesis,
University of Toronto, Canada (2009).
[25] Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image
recognition. arXiv preprint arXiv:1409.1556 (2014).
[26] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: A
simple way to prevent neural networks from overfitting. The Journal of Machine Learning
Research 15, 1929–1958 (2014).
[27] Russakovsky, O. et al. ImageNet Large Scale Visual Recognition Challenge. International
Journal of Computer Vision (IJCV) 115, 211–252 (2015).
19

[28] He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–
778 (2016).

Acknowledgement
DZ is supported by NWO project 656.000.005.

20

