Role of zero synapses in unsupervised feature learning
Haiping Huang

arXiv:1703.07943v4 [q-bio.NC] 10 Jan 2018

RIKEN Brain Science Institute, Wako-shi, Saitama 351-0198, Japan
(Dated: January 11, 2018)
Synapses in real neural circuits can take discrete values including zero (silent or potential)
synapses. The computational role of zero synapses in unsupervised feature learning of unlabeled
noisy data is still unclear, thus it is important to understand how the sparseness of synaptic activity
is shaped during learning and its relationship with receptive field formation. Here, we formulate this
kind of sparse feature learning by a statistical mechanics approach. We find that learning decreases
the fraction of zero synapses, and when the fraction decreases rapidly around a critical data size, an
intrinsically structured receptive field starts to develop. Further increasing the data size refines the
receptive field, while a very small fraction of zero synapses remain to act as contour detectors. This
phenomenon is discovered not only in learning a handwritten digits dataset, but also in learning
retinal neural activity measured in a natural-movie-stimuli experiment.
PACS numbers: 02.50.Tt, 87.19.L-, 75.10.Nr

INTRODUCTION

Sparsity in either neural activity or synaptic connectivity plays an important role in sensory information processing across brain areas [1, 2]. Sparsity constraints imposed on neural activity in a sparse coding model [3]
reproduce Gabor-like filters (edge detectors), which resemble receptive fields of simple cells in the mammalian
primary visual cortex. The sparse representation was
also applied to deep belief networks to model hierarchical representations of natural image statistics [4], which
capture higher order features at deeper levels of the cortical hierarchy. In addition, from a perspective of optimal
information storage, there must exist a large fraction of
silent or potential synapses [5], consistent with the existence of these synapses in cortex and cerebellum [6].
These (zero) synapses are vital for plasticity during learning [5]. Therefore, the sparse representation in synaptic
connectivity is also appealing in optimal neural computation.
Most artificial neural networks are trained by supervised learning, which requires a large library of images
prelabeled with categories. However, unsupervised learning gives humans and non-human animals the ability to
make sense of the external world by themselves, without
any additional supervision. Thus unsupervised learning
aims at extracting regularities in sensory inputs without
specific labels. To figure out what learning algorithms
may be used for modeling the external world in an unsupervised way is important for designing machine intelligence. However, understanding computational mechanisms of unsupervised learning in sensory representation
is extremely challenging [7].
Although zero synapses were observed in real neural
circuits, their computational role in concept formation
during unsupervised learning remains unclear, and lacks
a simple model to explain. Furthermore, previous theoretical efforts focused on random models of neural net-

works [8–10], where the distribution of synaptic values is
prefixed. Here, we propose a simple model of unsupervised learning with zero synapses, where a two-layered
neural network is introduced to learn the synaptic values
from sensory inputs, which is thus more practical than
random models. The bottom layer is composed of visible neurons receiving sensory inputs, while the top layer
contains only one hidden neuron in response to specific
features in the inputs. The bottom layer is connected to
the top layer by synapses. Note that there do not exist lateral connections in the bottom layer. This kind of
neural network is called a one-bit restricted Boltzmann
machine (RBM) [11, 12]. Binary synapses were also experimentally observed in real neural circuits [13, 14]. The
one-bit RBM with binary synapses has been studied as
a toy model of unsupervised feature learning [12], and is
analytically tractable at the mean-field level [15].
To study the computational role of zero synapses, we
model the connections between bottom and top layers
in the one-bit RBM by ternary synapses, which take
discrete values of {0, ±1}. Given a sensory input, the
ternary synaptic connections provide a hidden feature
representation of the input. One configuration of ternary
synapses forms a feature map and is also called the receptive field of the hidden neuron at the top layer.

PROBLEM SETTING AND MEAN-FIELD
METHOD

In this study, we use the one-bit RBM defined above
to learn specific features in sensory inputs, which are
raw unlabeled data. The machine is required to internally create concepts about the inputs. This process
is thus called unsupervised learning. Here, the sensory
inputs are given by handwritten digits taken from the
MNIST dataset [16]. Each image from this dataset has
28×28 pixels, specified by an Ising-like spin configuration
σ = {σi = ±1}N
i=1 where N is the input dimensionality.

2
A collection of M images is denoted by {σ a }M
a=1 . The
number of synapses is the same as the input dimensionality. Synaptic values are characterized by ξ, where each
component takes one of the ternary values. The one-bit
RBM is thus hdescribed by ithe Boltzmann distribution
P
P (σ, s) ∝ exp √βN i ξi σi s , where s = ±1 denotes the
activity of the hidden neuron, β denotes an inverse temperature, and the synaptic strength is scaled by the factor
√1 to ensure that the corresponding statistical mechanN
ics model has an extensive free energy. After marginalization of the hidden activity, one obtains the distribution
of the visible activity as:


cosh √βN ξ T σ
.

P (σ|ξ) = P
(1)
√β ξ T σ
σ cosh
N
As an inference model, for any given input, the one-bit
RBM has 3N possible synaptic configurations to describe
the sensory input. However, the machine will choose one
of these potential candidates as the feature map. This
process is naturally modeled by Bayes’ rule:
Q
a
a P (σ |ξ)
P
Q
)
=
P (ξ|{σ a }M
a=1
a
ξ
a P (σ |ξ)
 
(2)


βξi
1 Y
β T a Y −M ln cosh √
N
=
e
cosh √ ξ σ
,
Z a
N
i
where Z is the partition function of the model, and a
uniform prior for ξ is assumed. β serves as the inversetemperature like parameter to control learning
noise.
√
The synaptic scaling also requires β ≪ N , and under this condition, the last product in Eq. (2) becomes
Q −γξ2
αβ 2
M
i , where γ ≡
ie
2 with α ≡ N . It is clear that
from a Bayesian viewpoint, introducing zero synapses
amounts to some sort of Gaussian-like regularization but
with discrete support. The Bayesian method is able to
capture uncertainty in learned parameters (synaptic values here) and thus avoids over-fitting [17]. It may be able
to reduce the necessary data size for learning as well [12].
In what follows, we compute the maximizer of the
posterior marginals estimator ξˆi = arg maxξi Pi (ξi ) [18],
where the data dependence of the probability is omitted.
Hence, the task is to compute marginal probabilities, e.g.,
Pi (ξi ), which is a computationally hard problem due to
the interaction among data constraints (the product over
a in Eq. (2)). However, by mapping the original model
(Eq. (2)) onto a graphical model [12], where data constraints and synaptic values are treated as factor (data)
nodes and variable nodes respectively, one can estimate
the marginal probability by running a message passing
algorithm as we shall explain below. The key assumption is that synapses on the graphical model are weakly
correlated, which is called the Bethe approximation [19]
in physics. We first define a cavity probability Pi→a (ξi )

of ξi with data node a removed. Under the weak correlation assumption, Pi→a (ξi ) satisfies a self-consistent
equation:
1 −γξi2 Y
µb→i (ξi ),
(3a)
e
Pi→a (ξi ) =
Zi→a
b∈∂i\a

 Y
X
β
cosh √ ξ T σ b
µb→i (ξi ) =
Pj→b (ξj ),
N
{ξ |j∈∂b\i}
j∈∂b\i
j

(3b)

where Zi→a is a normalization constant, ∂i\a denotes
neighbors of feature node i except data node a, ∂b\i denotes neighbors of data node b except feature node i, and
the auxiliary quantity µb→i (ξi ) indicates the probability
contribution from data node b to feature node i given the
value of ξi [19]. Products in Eq. (3) stem from the weak
correlation assumption.
In the thermodynamic limit, the sum inside the hyperbolic cosine function in Eq. (3b), excluding the idependent term, is a random variable following a nor2
mal distribution with mean
P Gb→i and variance Ξb→i [20],
where Gb→i = √1N j∈∂b\i σjb mj→b and Ξ2b→i ≃
P
1
2
). The cavity magnetization
j∈∂b\i (m̂j→b − mj→b
N
P
is defined as mj→b =
ξj ξj Pj→b (ξj ), while the second moment
of
the
feature
component ξj is defined by
P
m̂j→b ≡ ξj ξj2 Pj→b (ξj ). Thus the intractable sum over
all ξj (j 6= i) can be replaced by an integral over the normal distribution. Furthermore, because ξi is a ternary
variable, Pi→a (ξi ) can be parametrized by cavity fields
ξi hi→a +(ξ 2 −1)gi→a

i
e
.
hi→a and gi→a , as Pi→a (ξi ) = ehi→a
+e−hi→a +e−gi→a
Combining this representation with Eq. (3), we have the
following iterative learning equations:

ζi→a
tanh hi→a ,
(4a)
1 + ζi→a


X
β
tanh−1 tanh βGb→i tanh √ σib , (4b)
=
N
b∈∂i\a
X Y
x
(4c)
eub→i ,
= e−γ

mi→a =
hi→a
ζi→a

x=±1 b∈∂i\a

i
h
βσb
βσb
where uxb→i ≡ ln cosh √Ni (1 + x tanh βGb→i tanh √Ni ) ,
and 2 cosh hi→a egi→a = ζi→a . mi→a can be interpreted
as the message passing from feature node i to data node
a, while uxb→i can be interpreted as the message passing from data node b to feature node i, depending on x.
ζi→a
Note that the prefactor 1+ζ
in Eq. (4a) is the cavi→a
ity probability of non-zero synapses, i.e., 1 − Pi→a (0),
and this is also m̂i→a according to the definition. In this
sense, the sparsity P
of synapses is described by a single
1
parameter ρ ≡ N1
i Pi (0), where Pi (0) = 1+ζi . The
potential feature (synaptic
configuration)
is inferred by
P
Q
x
computing ζi = e−γ x=±1 b∈∂i eub→i as well as mi =


P
β
−1
ζi
b
√
, in
tanh
βG
tanh
σ
tanh
tanh
b→i
b∈∂i
1+ζi
N i

3
which the magnetization mi is related to Pi (1) via
i (0)
.
Pi (1) = 1+mi −P
2
If the weak correlation assumption is self-consistent,
starting from randomly initialized messages, the learning
equations will converge to a fixed point corresponding to
a thermodynamically dominant minimum of the Bethe
free energy function (− β1 ln Z) [19]. In the following part,
we study how the learned feature map and the fraction
of zero synapses change with data size. In particular, we
focus on when the machine develops an internal concept
about the input handwritten digits and what the computational role of zero synapses is for feature selectivity
and receptive field formation.

(a)

We use the above mean field theory to study unsupervised feature learning with zero synapses. In the following simulations, β = 0.5 unless otherwise stated. For
simplicity, we consider only the 0 and 1 digits, because
other combinations of two different digits yield similar
results. We first study how the receptive field of the hidden neuron develops during the learning, as the number
of training images increases. As shown in Fig. 1 (a), when
the data is severely scarce, there is no apparent structure
in the feature map. When the number of training images
increases up to around 100, an intrinsically structured
feature map starts to develop. Nevertheless, there are
still a large fraction of zero synapses. As learning proceeds, the intrinsic structure concentrates more on the
center of the feature map, indicating that the machine
has already created an internal perception of external
stimuli. This kind of perception has been shown to have
an excellent discriminative capability on different stimuli by a precision-recall analysis [12]. This is because
the distribution of the weighted sum of inputs the hidden neuron receives develops two well-separated peaks
for two different digits.
Then, we study the computational role of zero
synapses. As shown in Fig. 1 (b), the sparsity level of
synapses ρ decreases with the data size M . Around
some critical M , the sparsity decreases abruptly, suggesting that a structured feature map is beginning to develop. Here, learning indeed induces the fraction of zero
synapses to decrease [5], since some zero synapses are
required to adopt non-zero values for capturing characteristics in the input data. When the data size is further
increased, the feature map is refined, and the sparsity decreases more slowly than around the critical region. At
large values of M , a small fraction of zero synapses are
still maintained. The zero synapses at this stage seem to
form an approximate boundary between active and inactive regions in the feature map (see the last feature map
in Fig. 1 (a)). Therefore, the zero synapses behave like
contour detectors. This effect is predicted by our model,

mcav

RESULTS

M
FIG. 1: (Color online) Learning behavior in a handwritten
digits dataset (digits 0 and 1). (a) Formation of receptive fields during learning. From left to right and top to
bottom, the corresponding data size increases as M = 2k
(k = 2, 3, . . . , 10). The colors black, white and gray indicate
inactive (ξi = −1), active (ξi = 1), and zero (ξi = 0) synapses
respectively. The gray border separating subfigures does not
refer to synapses. (b) The fraction ρ of zero synapses (left
axis) and the overall strength of cavity messages (mcav , right
axis) as a function of data size. Each marker in the plot is
averaged over ten random selections of training images with
equal number. The asymptotic curve of ρ in the small limit
of mcav is also shown.

but its neurobiological counterpart is still unclear and
deserves tested in future experiments of feature learning.
In particular, the monotonic behavior of the sparsity level of synapses is intimately related to the overall strength Pof cavity messages, which is defined as
2
1
mcav ≡ MN
(i,a) tanh hi→a . The model has originally
symmetry, since Eq. (2) is invariant under the transformation of ξ → −ξ. This symmetry can be spontaneously broken, as indicated by mcav (Fig. 1 (b)).
Around some critical M , cavity messages start to polarize without maintaining trivial (null) values any more,

4

(a)

FIG. 2: (Color online) Receptive field formation when only
images of various digit 0 are learned without any label information. From left to right and top to bottom, the corresponding data size increases as M = 2k (k = 2, 3, . . . , 10).

which is accompanied by the rapid decrease of the number of zero synapses. Moreover, the asymptotic behavior
of ρ in the small limit of the message strength can be
4 2
derived as ρasympt = 31 [1 − β 3ǫ ω M 2 ], where ǫ denotes
the small strength (mj→b = ǫ, ∀(j, b)), P
and ω denotes
the imageP
statistics expressed as ω = N1 i fi2 in which
1
b b
fi = MN
b,j σi σj . This asymptotic behavior captures
well the trend of ρ when M is small (Fig. 1 (b)).
Next, we study how the machine creates a perception
of only one digit such as 0, as learning proceeds. Receptive field formation is displayed in Fig. 2. Around
M = 64, a rough structure of receptive field starts to
emerge from the learning process, and the structure becomes more apparent as more data is added. Meanwhile,
the fraction of zero synapses decreases. Some of them become active, while some become inactive, refining the developed receptive field or feature map. The belief about
the stimulus image is continuously updated with more
sensory inputs. Around M = 512, a clear concept about
digit 0 is created by the unsupervised learning via combining likelihood and prior (see Eq. (2)). Interestingly, a
very small fraction (4.2%) of zero synapses remain and
serve as contour detectors. These zero synapses specify
the boundary between active and inactive regions in the
feature map.
Next, we study the effect of the inverse-temperature
β on the receptive field formation. β can be thought of
as a scalar tuning the global contrast level of the input
image [21]. By increasing β, one observes a qualitative
change of the feature map (Fig. 3 (a)), from an activesynapses-dominated phase (in the center of the feature
map) at small β to a zero-synapses-dominated phase at
high β. Surprisingly, the zero-synapses-dominated phase
still maintains the discriminative capability to distin-

β
(b)

FIG. 3: (Color online) (a) Feature maps at different inversetemperatures β. From left to right and from top to bottom,
β = 0.5, 2.0, 5.0 and 10.0. All feature maps are obtained when
M = 512. (b) The fraction of zero synapses versus the inversetemperature at different values of M . Each marker in the plot
is averaged over ten random selections of training images with
equal number.

guish different stimuli. Note that, at large β, the free
energy ceases to be extensive, which can be seen from
the last product of the second equality in Eq. (2). The
qualitative change results from the competition between
data constraints and biases introduced by zero synapses
(Eq. (2)). The critical β is determined by the value from
which ρ ceases to decrease and starts to increase. As
observed in Fig. 3 (b), βc decreases as M increases.
Finally, we apply the computational framework to
model retinal neural activity. We study a dataset composed of about 280 × 103 spike patterns of 160 retinal
ganglion cells. The neural activity was measured during a natural-movie-stimuli experiment on the salamander retina (data courtesy of Michael J. Berry [22]). The
retina is an early visual system performing decorrela-

cav

5

M
FIG. 4: (Color online) The fraction ρ of zero synapses (left
axis) and the overall strength of cavity messages (mcav , right
axis) as a function of the data size of neural activity. Each
marker in the plot is the average over ten random selections
of neural spike patterns with equal number.

tion computation of redundant visual inputs [23]. The
downstream brain areas may directly model the structure of population activity from the upstream area (such
as retina), without any reference to external sensory inputs [24]. Thus it is important to test our theory on
this kind of unsupervised learning of retinal neural activity. In Fig. 4, we observe similar behavior of the sparsity of synapses as found in learning a handwritten digits dataset. The learned feature map has a spontaneous
symmetry breaking at some critical data size, where the
sparsity of synapses changes rapidly as well. After the
spontaneous symmetry breaking, the feature map has
two possible phases: synapses are either all-active or allinactive, and the fraction of zero synapses becomes nearly
zero. The hidden neuron in our model can be thought of
as a unit in a downstream circuit along the ventral visual
pathway, and the polarization of its receptive field does
not show any intrinsic structures similar to those we already observed in learning a handwritten digits dataset.
This may be because the retina circuit is at the bottom
level of the visual hierarchy, while the concept of the visual input can only be formed at the higher level of the
cortical hierarchy [25].

DISCUSSION

In conclusion, we build a physics model of sparse unsupervised feature learning based on the one-bit RBM,
and in this model, the sparseness of synaptic activity is
automatically learned from the noisy data. A rapid decrease of the number of zero synapses signals concept
formation in the neural network, and the remaining zero
synapses refine the learned concept by serving as con-

tour detectors. In addition, zero synapses are sensitive
to the contrast level of sensory inputs. These predictions
may guide future neurobiological experiments. In particular, the fact that the number of zero synapses acts as
an indicator of concept formation is intimately related to
the spontaneous symmetry breaking in the model. These
findings may also have implications on promising deep
neuromorphic computation with discrete synapses [26].
It would be very interesting, yet challenging, to generalize the current framework to neural networks with
multiple hidden neurons, and furthermore with hierarchical multi-layered architectures.
Previous studies showed that parallel retrieval of memory is possible by a random dilution of connections in
random RBMs [8, 9], which may have connections to our
current findings, in the sense that zero synapses offer
the possibility to simultaneously recall multiple patterns.
Furthermore, our findings on unsupervised learning with
zero synapses are consistent with results reported in [5],
where supervised learning in a perceptron model of cerebellar Purkinje cells was studied. An intuitive explanation is that, learning stretches the synaptic-weight distribution, pushing synapses towards their limit values
(either 0 in the perceptron model [5] or ±1 here) [27].
A recent work derived the paramagnetic-spin-glass transition line in a generalized RBM with spin and weight
priors interpolating between Gaussian and binary distributions [28], which may connect to our results of spontaneous symmetry breaking and concept formation in real
data analysis.

Acknowledgments

I thank Taro Toyoizumi for his comments on silent
synapses, Jack Raymond and James Humble for careful reading the manuscript, and Adriano Barra for
drawing my attention to his previous works. This research was supported by AMED under Grant Number
JP15km0908001.

[1] P. Lennie, Current Biology 13, 493 (2003).
[2] H. Huang and T. Toyoizumi, Phys. Rev. E 93, 062416
(2016).
[3] Olshausen Bruno A. and Field David J., Nature 381, 607
(1996).
[4] H. Lee, C. Ekanadham, and A. Y. Ng, in Advances in
Neural Information Processing Systems 20, edited by
J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (Curran Associates, Inc., 2008), pp. 873–880.
[5] N. Brunel, V. Hakim, P. Isope, J.-P. Nadal, and B. Barbour, Neuron 43, 745 (2004).
[6] B. Barbour, N. Brunel, V. Hakim, and J.-P. Nadal,
Trends in Neurosciences 30, 622 (2007).

6
[7] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol,
P. Vincent, and S. Bengio, J. Mach. Learn. Res. 11, 625
(2010).
[8] E. Agliari, A. Barra, A. Galluzzi, F. Guerra, and
F. Moauro, Phys. Rev. Lett. 109, 268101 (2012).
[9] P. Sollich, D. Tantari, A. Annibale, and A. Barra, Phys.
Rev. Lett. 113, 238106 (2014).
[10] J. Tubiana and R. Monasson, Phys. Rev. Lett. 118,
138301 (2017).
[11] P. Smolensky (MIT Press, Cambridge, MA, USA, 1986),
chap. Information Processing in Dynamical Systems:
Foundations of Harmony Theory, pp. 194–281.
[12] H. Huang and T. Toyoizumi, Phys. Rev. E 94, 062310
(2016).
[13] C. C. H. Petersen, R. C. Malenka, R. A. Nicoll, and J. J.
Hopfield, Proc. Nat. Acad. Sci. 95, 4732 (1998).
[14] D. H. O’Connor, G. M. Wittenberg, and S. S.-H. Wang,
Proc. Nat. Acad. Sci. 102, 9679 (2005).
[15] H. Huang, Journal of Statistical Mechanics: Theory and
Experiment 2017, 053302 (2017).
[16] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Proceedings of the IEEE 86, 2278 (1998).
[17] D. J. C. MacKay, Neural Comput. 4, 448 (1992).
[18] H. Nishimori, Statistical Physics of Spin Glasses and In-

[19]
[20]
[21]
[22]

[23]
[24]
[25]
[26]
[27]
[28]

formation Processing: An Introduction (Oxford University Press, Oxford, 2001).
M. Mézard and A. Montanari, Information, Physics, and
Computation (Oxford University Press, Oxford, 2009).
H. Huang and T. Toyoizumi, Phys. Rev. E 91, 050101
(2015).
G. Orban, P. Berkes, J. Fiser, and M. Lengyel, Neuron
92, 530 (2016).
O. Marre, D. Amodei, N. Deshmukh, K. Sadeghi, F. Soo,
T. E. Holy, and M. J. Berry, J. Neurosci. 32, 14859
(2012).
Pitkow Xaq and Meister Markus, Nat Neurosci 15, 628
(2012).
A. R. Loback, J. S. Prentice, M. L. Ioffe, and M. J. Berry,
II, ArXiv e-prints 1610.06886 (2016).
J. J. DiCarlo, D. Zoccolan, and N. C. Rust, Neuron 73,
415 (2012).
A. Ardakani, C. Condo, and W. J. Gross, ArXiv e-prints
1611.01427 (2016).
Nicolas Brunel, personal communication.
A. Barra, G. Genovese, P. Sollich, and D. Tantari, ArXiv
e-prints: 1702.05882 (2017).

