arXiv:1710.09875v1 [] 26 Oct 2017

Phase Transitions in Image Denoising via Sparsely
Coding Convolutional Neural Networks

Jacob Carroll
Department of Physics & Center for Soft Matter and Biological Physics
Virginia Tech
Blacksburg, VA 24061
jac21934@vt.edu
Nils Carlson
Department of Computer Science and Engineering
New Mexico Tech
Socorro NM, 87801

Garrett T. Kenyon
CCS-3, Information Sciences
Los Alamos National Laboratory
Los Alamos, NM, 87545

Abstract
Neural networks are analogous in many ways to spin glasses, systems which are
known for their rich set of dynamics and equally complex phase diagrams. We
apply well-known techniques in the study of spin glasses to a convolutional sparsely
encoding neural network and observe power law finite-size scaling behavior in
the sparsity and reconstruction error as the network denoises 32×32 RGB CIFAR10 images. This finite-size scaling indicates the presence of a continuous phase
transition at a critical value of this sparsity. By using the power law scaling relations
inherent to finite-size scaling, we can determine the optimal value of sparsity for
any network size by tuning the system to the critical point and operate the system
at the minimum denoising error.

1

Introduction

Spin glasses and neural networks are very analogous and draw many parallels in their dynamics.
Generally, a spin glass is a model of disordered magnetism. The simplest model of a spin glass, the
Ising model, is a network of N "spins" {σi } which take on the discrete values, connected by a weight
matrix Jij ∈ R that represents the strength of connection between the spins. The dynamics of these
systems is determined by the values of randomly chosen Jij , which are generally time independent
(3).
The similarity of these spin glass systems with neural networks is of interest to us because spin
glasses have been a focus of research in statistical physics for the last fifty years, and a large library
of machinery and techniques has been developed to deal with them. We would like to apply this
machinery to the field of neural networks.
For this paper we used PetaVision, a high performance neural simulation toolbox (1), to construct
sparsely coding convolutional neural networks and examine the relationship between the network’s
efficiency and sparsity. Interesting behavior in the efficiency of the networks as the sparsity was varied
led us to analyze the finite-size scaling of the network, a technique more commonly used in the study
of spin glasses, and discovered power law relationships that indicate a continuous (second-order)
phase transition is occurring in the networks as sparsity is varied.

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

2

Neural network

We used two networks in our simulation, both built using PetaVision. The first network was a
sparse auto-encoder network that trained the filter kernels of a convolutional layer using a Locally
Competitive Algorithm, as defined by Rozell et al. (7), as it attempted to iteratively converge on a
sparse representation of different input images. The second network (see Figure 1) used the same
sparsely encoding convolutional layer that was trained by the autoencoder to denoise images that had
very high Gaussian noise added to them.
The input for both networks were images from the CIFAR-10 image set (6).The image set was divided
into two parts. The first 50,000 images were used for training the filter kernels of the sparsely coding
convolutional layer for different levels of sparsity. Then, 10,000 additional images had very high
Gaussian noise added to them and were denoised by the denoising network for each level of sparsity
used in training.
We observed a distinct minimum in the percent reconstruction error of the noisy images as the sparsity
of the network was varied that displayed behavior analogous to continuous phase transitions seen in
spin glasses (see figure 2) (3; 9). With this as our motivation we investigated the presence of a phase
transition in our system.
Input Layer

Noise Layer

Input Error
Layer

Sparsely Coding Input Reconstruction
Convolutional Layer
Layer

Figure 1: A schematic of the denoising network. The Input Error Layer computes the difference
between the Noise Layer and the Input Reconstruction Layer, an alternative implementation of lateral
inhibition in LCA (8). Feature learning utilizes a local Hebbian rule to implement stochastic gradient
descent.

3

Phase transitions and finite-size scaling

A phase of a system is defined as a subspace of the microscopic system parameters where the system’s
dynamics obey the same macroscale laws and relations everywhere in that subspace. The space of
system parameters can have many phases, and the system can transition between them as system
control parameters change. The point of transition between two (or more) phases is known as the
critical point.
Phase transitions have been subject of significant study in Condensed Matter Physics, and it is well
established that the occurrence of a continuous phase transition1 is accompanied by a singularity
at the critical point in one or more system parameters when the system is of infinite size (9). It is
impossible to achieve infinite system sizes computationally, but this theory can be expanded to finite
systems where these singularities become truncated and rounded. These minima or maxima that the
singularities turn into at finite system sizes follow very specific relations with system size:
Location of Minima ∼ L−1/ν
−γ/ν

Height of Minima ∼ L

,

(1)
(2)

where L is the linear system size. This behavior is known as finite-size scaling (9; 2). The exponents
ν and γ two examples of "critical exponents". The critical exponents describe the behavior of the
system as it approaches the critical point (9; 2). Thus we can identify a phase transition in our network
by the existence and behavior of minima and maxima in the space of system parameters as we vary
the system size, which in our case will be the number of neurons in the convolutional layer. The
exponents we record, ν̄ and γ̄, will be proportional to γ and ν through some effective dimension of
our system.
1

A continuous (second-order) phase transition has a continuous change in the dynamics of the system as it
transitions between phases, while first-order phase transitions are discontinuous.

2

4

Results

The parameters of the system that we are interested in are the fraction of active neurons and the
average percent reconstruction error of our noisy images:
10,000
X ksi − ŝi k2
1
Perr =
,
(3)
10, 000 i=1
ksi k2
where Perr is the average percent reconstruction error, si is the ith original image before it has
Gaussian noise added to it, ŝi is the ith reconstruction of the noised image taken from the sparsely
coding convolutional layer (1; 7; 8).

213 neurons
214 neurons
215 neurons
216 neurons
217 neurons

0.26

0.28

●

0.24

Percent Reconstruction Error

The fraction of active neurons is controlled by a parameter λ, as described Rozell et al. (7), that
behaves monotonically with the sparsity of active neurons and inversely with the fraction of active
neurons. Through λ we can control the fraction of active neurons and observe how the average percent
reconstruction error behaves as the fraction of active neurons is varied. We observed a minimum in
Perr occur as we varied the fraction of active neurons for many different system sizes. These results
are summarized in Figure 2.

●

●

●
●
●

●

●
●

●

0.22

●● ●

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Fraction Active Neurons

Figure 2: Average percent active error vs. fraction active neurons

0.03

0.04

0.05

101.91x−0.756

●

0.01

0.02

Fraction Active Neurons

0.2225
0.2215
0.2205

●

0.2195

Minimum Percent Error

10−0.621x−0.00745

214

215

216

217

214

System Size (Number of Neurons)

215

216

217

System Size (Number of Neurons)

(a) Height of minima vs. system size

(b) Location of minima vs system size

Figure 3: The power law behavior of the minimum average percent reconstruction error (a), and the
fraction of active neurons at that minimum (b). We report ν̄ = 1.32 ± 0.04 and γ̄ = 0.0099 ± 0.0095.
We measured the shift in height and location of the minima in Perr as the system size was varied,
and plot each on a log-log plot (see Figures 3 (a), and 3 (b)). We observe power law behavior in both
3

the location and height of the minima as the system size is varied. This satifies the finite-size scaling
requirements as defined in equations 1 and 2. This finite-size scaling behavior indicates a continuous
phase transition is occurring as the sparsity of the network is varied.

5

Discussion

The existence of phase transitions in neural networks is not unique to this sparsely coding convolutional system. The auto-associative network proposed by Hopfield (5) was shown by Hertz et al. (4)
to display a first-order phase transition in its memory capacity. If the number of patterns recorded by
the network exceeds a "critical fraction" of the network size, the output of the network is maximally
disordered (4).
We propose a similar mechanism is responsible for the observed continuous phase transition of our
sparsely coding convolution network, where the fraction of active neurons is analogous to the "critical
fraction" of learned patterns in the auto-associative network. If our network’s fraction of active
neurons is too far above the "critical fraction", the network will have the freedom to reconstruct the
noise in the image, while if the fraction of active neurons is too low, the network will only reconstruct
image components for which it has learned strong priors. These two different regions of dynamics
form our "phases". The existence of a phase transition in the average percent reconstruction error
of the network as the fraction of active neurons is varied guarantees the persistence of the power
law behavior seen in Figure 3 (b). This power law behavior allows us to predict the optimal fraction
of active neurons for any system size, which in turn can be tuned to through the parameter λ, as
described by Rozell et al. (7), to ensure that any sparsely coding convolutional network is operating
at the optimal level of sparsity.
The critical behavior of the network allows us to always achieve the minimum denoising error by
operating the network at this critical value of sparsity.
Acknowledgments
We gladly acknowledge helpful discussions with Uwe C. Täuber.
This work was supported by the Los Alamos National Laboratory under contract DE-AC5206NA25396.
Computations were performed using the Darwin Computational Cluster at Los Alamos National
Laboratory.

References
[1] Petavision. URL github.com/PetaVision/OpenPV.
[2] J. Cardy. Scaling and renormalization in statistical physics. Cambridge University Press, 1996. ISBN
0521499593.
[3] V. Dotsenko. An introduction to the theory of spin glasses and neural networks. World Scientific Lecture
Notes in Physics. World Scientific Publishing Company, 1995. ISBN 9810218737.
[4] J. A. Hertz, A. S. Krogh, and R. G. Palmer. Introduction to the theory of neural computation. Addison-Wesley
Publishing Company, 1991. ISBN 0201515601.
[5] J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities.
Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982. URL http://www.pnas.org/
content/79/8/2554.abstract.
[6] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[7] C. J. Rozell, D. H. Johnson, R. G. Baraniuk, and B. A. Olshausen. Sparse coding via thresholding and local
competition in neural circuits. Neural Computation, 20:2526–2563, 2008.
[8] P. F. Schultz, D. M. Paiton, W. Lu, and G. T. Kenyon. Replicating kernels with a short stride allows sparse
reconstructions with fewer independent kernels. arXiv preprint arXiv:1406.4205, 2014.
[9] U. C. Täuber. Critical dynamics: a field theory approach to equilibrium and non-equilibrium scaling
behavior. Cambridge University Press, 2014. ISBN 9780521842235.

4

