Bio-Inspired Multi-Layer Spiking Neural Network
Extracts Discriminative Features from Speech Signals

arXiv:1706.03170v1 [] 10 Jun 2017

Amirhossein Tavanaei and Anthony Maida
The Center for Advanced Computer Studies, Bio-inspired AI Lab
The School of Computing and Informatics
University of Louisiana at Lafayette
Lafayette, LA 70504, USA
{tavanaei,maida}@louisiana.edu

Abstract
Spiking neural networks (SNNs) enable power-efficient implementations due to
their sparse, spike-based coding scheme. This paper develops a bio-inspired SNN
that uses unsupervised learning to extract discriminative features from speech
signals, which can subsequently be used in a classifier. The architecture consists
of a spiking convolutional/pooling layer followed by a fully connected spiking
layer for feature discovery. The convolutional layer of leaky, integrate-and-fire
(LIF) neurons represents primary acoustic features. The fully connected layer is
equipped with a probabilistic spike-timing-dependent plasticity learning rule. This
layer represents the discriminative features through probabilistic, LIF neurons. To
assess the discriminative power of the learned features, they are used in a hidden
Markov model (HMM) for spoken digit recognition. The experimental results show
performance above 96% that compares favorably with popular statistical feature
extraction methods. Our results provide a novel demonstration of unsupervised
feature acquisition in an SNN.
Keywords: Bio-inspired multi-layer framework, spiking network, speech recognition, unsupervised feature extraction.

1

Introduction

Multi-layer neural network (NN) learning extracts signal features by using a hierarchy of non-linear
elements [1]. Such networks extract increasingly complex, discriminative, and independent features
and serve as the basis for deep learning architectures [2, 3]. The power of a multi-layer neural
architecture can be used for a challenging problem like automatic speech recognition (ASR). Speech
signal characteristics are highly variable in time and frequency. The feature extraction method must
convert the speech signal to discriminative features to support the recognition task. Recently, multilayer neural architectures have outperformed previous ASR models. Examples are convolutional
neural networks (CNNs) [4], deep CNNs [5], deep NNs [6], and deep recurrent NNs [7]. Despite
the high performance of these architectures, training and then using them is expensive from a
power consumption viewpoint. The quest to meet a power-efficient framework can be accomplished
by a spike-based neuromorphic platform. Spiking neural networks (SNNs) [8, 9, 10] provide an
appropriate starting point for this. After introducing the power of SNNs as a third generation neural
network [8], a number of studies have concentrated on biologically motivated approaches for pattern
recognition [11, 12, 13]. An SNN architecture consists of spiking neurons and interconnecting
synapses undergoing spatio-temporally local learning in response to the stimulus presentation.
.

3

SPIKING CONVOLUTIONAL LAYER

Word recognition functionality in the auditory ventral stream of the human brain is enabled by a
multi-layer, biological SNN. Multi-layer SNNs have performed comparably with conventional NNs
in visual pattern recognition tasks [14, 15, 16]. Also, SNNs have been employed for the spoken
word recognition tasks such as using synaptic weight association training [17], extracting spike
signatures from speech signals [18], and Gaussian mixture model (GMM) implementation using
SNNs [19]. Representing speech signal features through temporal spike trains has attracted much
interest. Verstraeten et al. developed a high performance reservoir-based SNN for spoken digit
recognition problem [20]. The reservoir is not trained and it maps the inputs to a higher dimensional
space to be linearly separable. Dibazar et al. proposed a feature extraction method using a dynamic
synaptic neural network for isolated word recognition [21]. In another study, Loiselle et al. utilized
the cochlear Gammatone filter bank to generate spike trains in French spoken word recognition based
on rank-order coding with an SNN [22]. However, these studies did not develop a multi-layer SNN
to extract discriminative speech characteristics in a hierarchy of feature discovery neural layers.
This paper develops a multi-layer spiking neural network consisting of a convolutional layer, a
pooling layer, and an unsupervised feature-discovery layer. The convolutional layer extracts primary
acoustic features through the spike trains emitted from the feature map neurons. The pooling layer
reduces size of the feature maps by a novel max-pooling operation based on Mel-scaled patch sizes.
The feature discovery layer receives presynaptic spikes from the pooling layer and undergoes learning
via a probabilistic spike-timing-dependent plasticity (STDP) rule. Neurons in this layer represent
the final features extracted from a speech signal. During the training phase, weights connecting the
pooling layer and the feature discovery layer are adjusted based on the speech signal characteristics.
The features extracted by the SNN are used for training and evaluating a hidden Markov model
(HMM) [23] to handle temporal behavior of the speech signal. The proposed network offers a novel
way to discover discriminative features for spoken digits while implementing a bio-inspired model,
potentially enabling low-power consumption on appropriate hardware [24].

2

Network Architecture

Our multi-layer SNN, shown in Fig. 1a, consists of two spiking neural layers which follow an
input layer. The input layer’s stimulus is the magnitude Fourier transform of a speech frame (M
values). This layer converts the stimuli to spike trains using a Poisson process over T = 40 ms. The
convolutional layer maps the presynaptic spike trains to a higher dimensional feature space using
convolutional filters. The pooling layer reduces the feature map dimension, according to Mel-scaled
frequencies, and then transfers the new spike trains to the fully connected feature discovery layer for
unsupervised training. After training, the neuronal activities in the feature discovery layer represent
the frame’s feature vector that may be input to a classifier (e.g., an HMM classifier).

3

Spiking Convolutional Layer

The convolutional layer receives presynaptic spike trains from the input layer and generates D
independent feature maps. Each feature map contributes to a convolutional filter acquired by shifted
p-point difference-of-Gaussian (DoG) (p = 7 in this network) filters. Therefore, seven shifted DoG
filters are convolved with the presynaptic spike trains to generate D = 7 feature maps. An LIF
neuron in a feature map integrates the convolution results for a particular signal patch (p = 7 points)
over T = 40 time steps and fires a spike when reaching threshold. The neuron’s membrane potential,
U (t), is reset to the resting potential (zero) upon firing. Eq. 1 calculates the membrane potential of
neuron m in feature map k at time t.
k
dUm
(t)
k
k
+ Um
(t) = Im
(t)
dt
k
k
if Um
(t) ≥ θconv , then Um
(t) = 0

(1a)
(1b)

k
Im
is computed by
k
Im
(t)

=

p
X

Wik · sm,i (t)

i=1

2

(2)

5

(a)

FEATURE DISCOVERY LAYER

(b)

Figure 1: (a): Multi-layer SNN architecture. Each rectangle shows a frame’s feature vector in each
layer. The convolutional layer produces seven 1-D feature maps of spiking neurons. (b): RC circuit
implementing the LIF neuron with R = 1 Ω, C = 1 mF, and threshold, θ = θconv .
k
The threshold, θconv , is set to 0.4 in the experiments. The Im
(t) value determines the convolution of
k
the filter, W , and the presynaptic spike train, sm , in T = 40 time steps. This value is considered
as an injected current in the RC circuit (Fig. 1b), which simulates the LIF neuron. The length of a
feature map is same as the input vector.

4

Mel-Scaled Pooling Layer

The human hearing mechanism resolves frequencies non-linearly with higher resolution at lower
frequencies. The Mel scale makes the features close to the hearing model. In the pooling layer, the
stride value (pooling window size) is computed based on the Mel-scaled bandwidths. Eq. 3 converts
the frequency scale to the Mel scale. Same stride value l = 2 is used for the frequencies in the range
0 to 1000 Hz. Then, the stride values increase corresponding to the feature frequency. The number
of pooling windows (N ) is analogous to the number of Mel-spaced filter banks (20-40) selected
for computing the Mel-scaled frequency cepstral coefficients (MFCCs). The pooling layer selects a
neuron with the highest activity in a pooling window (max pooling). Activities of the neurons in a
feature map can be characterized by their spike rate. This layer summarizes and extracts effective
signal characteristics.
fmel = 1125 ln 1 +

f 
700

(3)

The new pooling strategy and the Mel-scaled bandwidths can also be employed by conventional
CNNs for extracting acoustic features.

5

Feature Discovery Layer

The feature discovery layer, containing H neurons and equipped with lateral inhibition, is fully
connected to the pooling layer. It learns and extracts the signal features based on the provided
Mel-scaled pooling features. This layer conveys D × N spike trains emitted from the pooling layer
to H probabilistic LIF neurons. The feature discovery layer is shown in the last layer of Fig. 1a.
5.1

Neuron Model

The stochastic LIF neurons in the feature discovery layer fire based on both their membrane potential
and their firing posterior probability (imposed by a softmax function). A neuron fires if its membrane
3

5.2

Learning

6

EXPERIMENTS AND RESULTS

potential reaches the threshold, θh , (following Eq. 1) and its posterior probability is greater than
0.5. In the experiments, θh is set to 3. The posterior probability of neuron h firing at time t given
input spike vector yt and weights W is given by Eq. 4. The posterior probability determines a
softmax function in which the mutual exclusivity is not enforced. This neuron model implements a
winners-take-all competition (inhibition) in the feature discovery layer. Although there are no direct
inhibitory connections between neurons, the softmax function imposes an overall inhibition upon the
neural column extracting independent features. The neural column shown in Fig. 1a illustrates that
because of the lateral inhibitions, only a small subset of the neurons can fire at the same time. The
inhibition constrains the model to extract independent features in response to the input.
T

eWh ·yt
P (zh = 1|W, yt ) = PH W T ·y
t
j
j=1 e
5.2

(4)

Learning

Spike-timing-dependent plasticity (STDP) is a learning rule used in SNNs that increases the weight
of a synapse if the presynaptic spike occurs briefly before the postsynaptic spike (LTP) and decreases
the synaptic weight otherwise (LTD) [25]. We use a probabilistic version of the STDP rule provided
by Tavanaei et al. [26] (Eq. 5) to learn the final acoustic features.

a+ e−whi , if i fires in [t − , t]
∆whi =
(5)
−a− ,
otherwise.
In the above, a+ = 10−3 and a− = 0.75 × 10−3 are, respectively, LTP and LTD learning rates.
When a postsynaptic neuron h fires, STDP is triggered for the incoming weights. LTP occurs if the
presynaptic neuron i fires briefly (e.g. within  = 5 ms) before h fires, while the weight change
magnitude is controlled by the current weight. Weights remain in the range (0, 1). The probability
derivations and justifications are given in [26]. Weights are initialized by sampling uniformly in the
range (0, 1).
5.3

Final Feature Representation

A speech frame is built from H neurons in the feature discovery layer. The elements of the vector are
the neurons’ activities in response to received spike trains via trained weights, W . More specifically,
the accumulated membrane potentials of the neurons are used for the feature values. Samples with H
attributes obtained by the feature discovery layer are then used to train the HMM classifier.

6

Experiments and Results

This investigation uses the proposed SNN to discover and extract discriminative features from the
spoken digits 0 to 9 selected in the Aurora dataset [27] that is recorded at the sampling rate of 8000
Hz. Each speech signal (spoken digit) is divided into 25 ms frames with 50% temporal overlap,
resulting in about 60 frames per digit. The frames are converted to the frequency domain by using the
magnitude of the Fourier transform. Each frame is represented by 100 sample values (0 to 4000 Hz).
DoG convolutional filters are obtained by the difference of two Gaussian functions with variances
σ 2 = 1 and σ 2 = 6 (Eq. 6). Seven filters produce D = 7 feature maps containing 100 spiking
neurons. Each map is converted to a pooled map by the max-operation over the Mel-scaled pooling
windows. The pooling windows are shown in Fig. 2a with the stride values 2 through 10. In the
experiments, we use 28 values to represent each speech frame. The stride values for the sample points
1 to 26 (0 to 1040 Hz) are set to 2 (13 feature values). The last stride value is 10. Fig. 2b shows neural
spike rates of three selected pooled maps extracted from the spoken digit ‘one’. The pooled layer
conveys the speech frame characteristics with only 7 × 28 = 196 values.
DoG = N (x, µ = 0, σ 2 = 1) − N (x, µ = 0, σ 2 = 6),
x = {−3, −2, ..., 3}

(6)

The feature discovery layer extracts the final feature vectors for the speech frames using its trained
synaptic weights. Training the network involved cycling through a dataset of 1000 spoken digits
4

6

(a)

(b)

EXPERIMENTS AND RESULTS

(c)

Figure 2: (a): Mel-scaled stride values spanning 0-4000 Hz. (b): Neural spike rates of three selected
pooled maps extracted from the spoken digit ‘one’. Neurons emit zero (dark blue) to 39 (dark red)
spikes. (c): Synaptic weights of the feature discovery layer connecting the neurons in a pooled map
to 30 LIF neurons. The synaptic weights fall in the range 0 (dark blue) to 1 (dark red).

Table 1: Accuracy rates of the HMMs trained and evaluated by the features extracted by the multi-layer
SNN. S: number of states. G: number of GMMs.
HMM Variations G = 4
G=8
G = 16 G = 32
S=5
62.88
75.59
85.95
90.97
S = 10
71.91
87.63
92.31
95.32
S = 15
84.95
91.30
96.32
96.32

for 10 iterations. Fig. 2c shows an example synaptic weight set connecting a pooled map and the
feature discovery neurons. Most neurons show higher resolution at lower frequencies. The feature
vectors extracted from the temporal frames are used to train and evaluate the HMM. In the first
experiment, H = 30 feature discovery neurons (as feature vector dimension) were used to train 12
HMMs with {5, 10, 15} states and {4, 8, 16, 32} GMMs embedded in each state. The testing set was
sampled from the spoken digits that were not used for the training. Table 1 shows the accuracy rates
reported for the spoken digit recognition task. Performances higher than 95% were achieved by the
HMMs with 10 and 15 states and 32 GMMs. These accuracy rates show that the SNN, which uses
unsupervised learning, has extracted discriminative features from the speech signals.
The next experiment evaluates: 1) the number of extracted features, H, (number of neurons in the
feature discovery layer); and, 2) applying ∆ (differential, D) and ∆∆ (acceleration, A) features
to use more dynamic information. The HMM with ten states and 32 GMMs is selected for this
experiment. Fig. 3 shows that the model with H = 30 probabilistic neurons in the feature discovery
layer outperforms the other architectures. Additionally, the ∆ features improved the accuracy rate.
However, the ∆∆ features did not improve performance in the networks with more than H = 20
output neurons. This could happen because the HMM might be overtrained by the high dimensional
feature vectors.
Our SNN extracted discriminative features that were classified at 96% accuracy. This accuracy
rate is comparable with [17] (95.25% accuracy) that uses synaptic weight association training in a
neural layer. Although the reservoir approach [20] has performed better (near perfect) on spoken
digit recognition, the question of training a reservoir to scale to larger problems remains unresolved.
Additionally, our multi-layer SNN provides an environment to support a deep learning architecture in
a spiking platform that compares with conventional deep networks. Table 2 shows the performance of
spoken digit recognition models using spiking neural networks and recent conventional approaches.
The proposed model compares favorably with the high performance methods, such as conventional
CNN (with > 99% accuracy), while using a power-efficient, biologically plausible framework.
5

REFERENCES

Figure 3: Performance of the networks with 10 through 50 spiking neurons in the feature discovery
layer for different sets of features. ∆ (D) and ∆∆ (A) features are concatenated to the extracted
features (F).
Table 2: Spoken digit recognition models using SNNs (Top) and non-SNN (Bottom) architectures
evaluated on the Aurora [27], TIDigits [28], and TI46 [29] spoken digit datasets. Aurora is based on
a version of the TIDigits dataset, but downsampled at 8 kHz, and is roughly comparable to the TI46
dataset.
Model
Method
Dataset Acc. (%)
Wade et al. [17]
Synaptic weight association (SNN)
TI46
95.25
Tavanaei et al. [18]
Single layer SNN and SVM classifier Aurora
91
Verstraeten et al. [20]
Reservoir computing (LSM)
TI46
>97.5
Our Model
Spiking CNN and HMM classifier
Aurora
96
Dao et al. [30]
Structured sparse representation
Aurora
95
Van Doremalen et al. [31] Hierarchical Temporal Memory
TIDigits 91.6
Neil et al. [32]
MFCC and Deep RNN
TIDigits 96.1
Groenland et al. [33]
CNN and deep shifting methods
TI46
>99

7

Conclusion

A multi-layer SNN was developed to give a novel demonstration of unsupervised feature extraction
and discovery for speech processing. The SNN receives input in the form of spike trains from
incoming speech signals, extracts Mel-scaled convolutional features, and then extracts higher-level
discriminative features. The convolutional layer extracts primary acoustic features to make the
network layer stack-admissible for the next layers. The pooling layer summarizes the primary
features such that the lower frequencies had higher resolution using Mel-scaled pooling windows.
As far as we know, this is a novel feature of the network. The last layer is the feature discovery
layer equipped with the probabilistic STDP learning rule. It is trained to discover more complex,
independent features using stochastic LIF neurons. The classification results reported by the HMM
(with 96% accuracy for the spoken digit recognition problem) showed that the network architecture
can discover effective, discriminative acoustic features. Furthermore, the multi-layer network uses
sparse, spike-based coding which can be implemented by appropriate, power-efficient VLSI chips.
The initial promising results obtained by unsupervised learning in this model offer promise of
scaling to larger problems, such as developing a deep spiking network for the large vocabulary
speech recognition (LVSR) problem. Our future work concentrates on a multi-layer SNN for feature
extraction followed by a spike-based sequential feature classifier.

References
[1] Yann LeCun. Learning invariant feature hierarchies. In European Conference on Computer
Vision, pages 496–505. Springer, 2012.
[2] Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine
Learning, 2(1):1–127, 2009.
6

REFERENCES

REFERENCES

[3] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444,
2015.
[4] Ossama Abdel-Hamid, Abdel-Rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong
Yu. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio,
Speech, and Language Processing, 22(10):1533–1545, 2014.
[5] Tara N Sainath, Abdel-rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep
convolutional neural networks for LVCSR. In 2013 IEEE International Conference on Acoustics,
Speech and Signal Processing, pages 8614–8618. IEEE, 2013.
[6] Geoffrey Hinton, Li Deng, Dong Yu, et al. Deep neural networks for acoustic modeling in speech
recognition: The shared views of four research groups. IEEE Signal Processing Magazine,
29(6):82–97, 2012.
[7] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep
recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and
Signal Processing, pages 6645–6649. IEEE, 2013.
[8] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models.
Neural networks, 10(9):1659–1671, 1997.
[9] Samanwoy Ghosh-Dastidar and Hojjat Adeli. Spiking neural networks. International Journal
of Neural Systems, 19(04):295–308, 2009.
[10] Nikola Kasabov, Kshitij Dhoble, Nuttapod Nuntalid, and Giacomo Indiveri. Dynamic evolving
spiking neural networks for on-line spatio-and spectro-temporal pattern recognition. Neural
Networks, 41:188–201, 2013.
[11] Peter U Diehl and Matthew Cook. Unsupervised learning of digit recognition using spiketiming-dependent plasticity. Frontiers in computational neuroscience, 9, 2015.
[12] Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, and Timothée Masquelier. Bio-inspired
unsupervised learning of visual features leads to robust invariant object recognition. Neurocomputing, 205:382–392, 2016.
[13] Yoshua Bengio, Thomas Mesnard, Asja Fischer, Saizheng Zhang, and Yuhuai Wu. Stdpcompatible approximation of backpropagation in an energy-based model. Neural Computation,
29(3):555–577, 2017.
[14] Timothée Masquelier and Simon J Thorpe. Unsupervised learning of visual features through
spike timing dependent plasticity. PLoS Comput Biol, 3(2):e31, 2007.
[15] Simei Gomes Wysoski, Lubica Benuskova, and Nikola Kasabov. Fast and adaptive network of
spiking neurons for multi-view visual pattern recognition. Neurocomputing, 71(13):2563–2575,
2008.
[16] Michael Beyeler, Nikil D Dutt, and Jeffrey L Krichmar. Categorization and decision-making in a
neurobiologically plausible spiking network using a STDP-like learning rule. Neural Networks,
48:109–124, 2013.
[17] John J Wade, Liam J McDaid, Jose A Santos, and Heather M Sayers. SWAT: a spiking neural
network training algorithm for classification problems. IEEE Transactions on Neural Networks,
21(11):1817–1830, 2010.
[18] Amirhossein Tavanaei and Anthony S Maida. A spiking network that learns to extract spike
signatures from speech signals. Neurocomputing, 240:191–199, 2017.
[19] Amirhossein Tavanaei and Anthony S. Maida. Training a hidden markov model with a Bayesian
spiking neural network. Journal of Signal Processing Systems, pages 1–10, 2016.
[20] David Verstraeten, Benjamin Schrauwen, and Dirk Stroobandt. Reservoir-based techniques
for speech recognition. In The 2006 IEEE International Joint Conference on Neural Network
Proceedings, pages 1050–1053. IEEE, 2006.
7

REFERENCES

REFERENCES

[21] A. A. Dibazar, D. Song, W. Yamada, and T. W. Berger. Speech recognition based on fundamental
functional principles of the brain. In IEEE International Joint Conference on Neural Networks,
volume 4, pages 3071–3075, 2004.
[22] Stéphane Loiselle, Jean Rouat, Daniel Pressnitzer, and Simon Thorpe. Exploration of rank
order coding with spiking neural networks for speech recognition. In Proceedings. 2005 IEEE
International Joint Conference on Neural Networks, 2005., volume 4, pages 2076–2080. IEEE,
2005.
[23] Lawrence R Rabiner. A tutorial on hidden Markov models and selected applications in speech
recognition. Proceedings of the IEEE, 77(2):257–286, 1989.
[24] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks
for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54–66,
2015.
[25] Yang Dan and Mu-Ming Poo. Spike timing-dependent plasticity: from synapse to perception.
Physiological Reviews, 86(3):1033–1048, 2006.
[26] Amirhossein Tavanaei, Timothee Masquelier, and Anthony S. Maida. Acquisition of visual
features through probabilistic spike-timing-dependent plasticity. In Proceedings. IEEE International Joint Conference on Neural Networks., pages 307–314, 2016.
[27] Hans-Günter Hirsch and David Pearce. The Aurora experimental framework for the performance
evaluation of speech recognition systems under noisy conditions. In ASR2000-Automatic Speech
Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW),
2000.
[28] R Leonard. A database for speaker-independent digit recognition. In Acoustics, Speech, and
Signal Processing, IEEE International Conference on ICASSP’84., volume 9, pages 328–331.
IEEE, 1984.
[29] Gearge R Doddington and Thomas B Schalk. Computers: Speech recognition: Turning theory
to practice: New ICs have brought the requisite computer power to speech technology; an
evaluation of equipment shows where it stands today. IEEE spectrum, 18(9):26–32, 1981.
[30] Minh Dao, Yuanming Suo, Sang Peter Chin, and Trac D Tran. Structured sparse representation
with low-rank interference. In Signals, Systems and Computers, 2014 48th Asilomar Conference
on, pages 106–110. IEEE, 2014.
[31] Joost Van Doremalen and Lou Boves. Spoken digit recognition using a hierarchical temporal
memory. In INTERSPEECH, pages 2566–2569, 2008.
[32] Daniel Neil and Shih-Chii Liu. Effective sensor fusion with event-based sensors and deep
network architectures. In Circuits and Systems (ISCAS), 2016 IEEE International Symposium
on, pages 2282–2285. IEEE, 2016.
[33] Koen Groenland and Sander Bohte. Efficient forward propagation of time-sequences in convolutional neural networks using deep shifting. arXiv preprint arXiv:1603.03657, 2016.

8

