On the organization of grid and place cells: Neural de-noising via
subspace learning

arXiv:1712.04602v1 [q-bio.NC] 13 Dec 2017

David M. Schwartz∗1 and O. Ozan Koyluoglu†2
1

Department of Electrical and Computer Engineering, University of Arizona
2
University of California, Berkeley

Abstract
Place cells in the hippocampus are active when an animal visits a certain locations (referred
to as place fields) within an environment and remain silent otherwise. Grid cells in the medial
entorhinal cortex (MEC) respond at multiple locations, with firing fields that exhibit a hexagonally symmetric periodic pattern. The joint activity of grid and place cell populations, as a
function of location, forms a neural code for space. An ensemble of codes, for a given set of parameters, is generated by selecting grid and place cell population and tuning curve parameters.
For each ensemble, codewords are generated by stimulating a network with a discrete set of
locations. In this manuscript, we develop an understanding of the relationships between coding
theoretic properties of these combined populations and code construction parameters. These
observations are revisited by measuring the performances of biologically realizable algorithms
(e.g. neural bit-flipping) implemented by a network of place and grid cell populations, as well
as interneurons, which perform de-noising operations. Simulations demonstrate that de-noising
mechanisms analyzed here can significantly improve fidelity of this neural representation of
space. Further, patterns observed in connectivity of each population of simulated cells suggest
the existence of heretofore unobserved neurobiological phenomena.

1

Introduction
Place cells are spatially modulated neurons with bivariate Gaussian tuning curves centered on

particular locations in the environment, and have been identified in the hippocampus [1]. Grid
cells are spatially modulated neurons that exhibit a peak firing rate at a periodic and hexagonally
symmetric distribution of locations in the envirionment, and are found in the Entorhinal Cortex
(EC) of rats, mice, bats, and humans [2–4]. Grid cells are clustered in discrete modules wherein
cells share grid scale. Anatomically, both cell types share a dorsoventral organization, with cells
possessing wider receptive fields distributed towards the ventral end [5, 6]. It is known that the rat
grid cell network requires communication from the hippocampus to maintain grid-like activity [7],
∗
†

dmschwar@email.arizona.edu
ozan.koyluoglu@berkeley.edu

1

and that a significant improvement in accuracy of the rodent place cell representation is tightly
correlated with the emergence of the grid cell network [8]. However, the mechanisms by which these
networks communicate and how each may bolster the other’s accuracy are unknown.
Associative memories are a class of biologically implementable content addressable memory
consisting of networks of neurons, a learning rule, and in some instances, a separate recall process [9].
This means that they can be exploited to stabilize the states of their constituent neurons to match
a previously memorized network state if enough of the network already lies in this state. The
information capacity of the simplest of these constructions is quite limited:

n
2 log n

bits, for a network

of n binary neurons [10] . However, recent advances by Salavati et al. take advantage of sparse
neural coding and non-binary neurons to design an associative memory with information storage
capacity exponential in the number of neurons [11]. Sparse connectivity confers the memory network
with other performance improvements: infrequent spiking implies reduced energy costs and faster
convergence to a stable state. In communications, this principle is leveraged by LDPC codes, whose
power (in coding and decoding complexity) depends on sparsity of the code’s parity check matrix
(an object analogous to our matrix of synaptic weights that form the de-noising network).
Grid cell population activity forms a dense modulo code, in which (in the absence of noise)
information about position at any grid scale may be present in the activity of each module [2].
Biologically reasonable decoders for the grid cell position code were proposed and tested in [12],
and we replicated these results by implementing the same grid cell population decoder, though it
is not the focus of this paper. Place cell activity forms a relatively sparse code (for enough cells,
and a sufficiently large environment), thus combining populations of grid and place cells realizes
position codes that are sparser than the grid cell component of the code. Our results indicate that
there are reasonable (in the biological sense) grid and place cell network parameters that result in
population codes for position with a wide range of densities. Nature provides myriad circumstances
in which many neural computations (e.g. object recognition, acoustic source localization, and self
location) must be executed robustly in the presence of neural noise if the organism is to survive.
We propose a de-noising mechanism for populations of grid and place cells, in the form of the
associative memories described in [11], [13], and [14], which takes advantage of coding theoretic
properties of these populations to eliminate the negative impacts of external noise. In particular,
we propose a de-noising algorithm that relies on the biological organization of grid cells into discrete
modules, and observe that after learning, average connectivity between place cells and grid modules
decreases with increasing place cell size for each module.
Redundancy in receptive field (RF) population codes is known to confer improvements in decoding accuracy when a small tolerance to error is introduced (expressed in this case, in the stimulus
space to which we decode) [15]. We believe we are the first to investigate coding theoretic impacts
of redundancy in grid cells. We investigate the impact of this redundancy on decoding accuracy by
comparing de-noising and decoding performance across codes of varying redundancies. We demonstrate that a maximum likelihood (ML) estimator reliably decodes position from population activity
with small position estimation error in the presence of bounded noise. Overall, our work shows
that the biological organization of grid cells into modules may be necessary for optimal estimation
2

of self location.
This paper is organized sectionally. In section 1 we introduce a few key concepts and present
the main results. Section 2 describes the theoretical framework upon which our model is built.
Section 3 presents results of all coding theoretic analysis and experimentation. Section 4 describes
the learning algorithms implemented in our de-noising network, as well as the outcomes of these
learning algorithms. In section 5, we describe the de-noising processes and address their biological
plausibility. Additionally, we present the results of performance tests of the ML position estimator,
with and without de-noising. Section 6 consists primarily of discussion of these results.

2

Theoretical framework

2.1

A hybrid code
(b) 2D Hybrid code

(a) 1D Hybrid code
ventral

module m=1

8
6
4
2
50

100

150

200

position (cm)

8
6
4
2
0
0

50

100

150

position (cm)

p1

10

pP
firing rate

firing rate

gM,JM

gM,1

position (cm)

pP
8
6
4
2
0
200

g2,J2

g2,1

200

position (cm)

p1

g1,J1

g1,1

10

module m=M

firing rate

10

0
0

gM,JM

gM,1
firing rate

firing rate

g2,J2

g2,1

ventral

module m=1

module m=M

g1,J1

g1,1

dorsal

firing rate

dorsal

250

300

350

400

position (cm)

position (cm)

We consider a population of place and grid cells, a total of N neurons. There are M grid
cell modules, each module, m, containing Jm neurons. Each grid cell is denoted as Gm,j ∼
Poisson(gm,j ), for m ∈ {1, · · · , M } and j ∈ {1, · · · , Jm }.

Place cells are denoted as Pi ∼

Poisson(pi ), for i ∈ {1, . . . , P }. The activity of this population, as a function of the (1D) location `, is represented by

xi (`) =




gm,j (`),

i=

m−1
P



pi−M J (`),

i>

k=1
M
P

Jk + j, i ≤

M
P

Jm

m=1

Jm

m=1

where the location dependent mean firing rates of the grid cells, gm,j (`), are given by the following
periodic Gaussian tuning curve

gm,j (`) = fmax exp

−(mod(λm /2 + ` − φj , λm ) − λm /2)2
2
2σm


.

(1)

Here, λm and σm are, respectively, the period and tuning widths of grid cells in module m, and
φj is a random spatial phase. Each pi (`) is modeled as a thresholded Gaussian distribution over
3

[0, L], where L is the length of a 1-dimensional track, with mean, µ, a random variable uniformly
distributed over [0, L], standard deviation, σ, chosen randomly and uniformly from [0.9λ1 , 1.1λM ].
We also consider a population of grid and place cells receptive to position in two dimensions.
In this paradigm, each grid cell’s tuning curve follows a two-dimensional distribution resembling a
von-Mises density function,
" 3
#
X
3
fmax
4
exp
gm,j (s) =
cos( √ u(θk − θm,j ) · (s − cm,j ) + ) − 1 ,
Z
2
λm 3
k=1

(2)

where u(θk − θm,j ) is a unit vector in the direction of θk − θm,j , s ∈ [0, L] × [0, L] is the position
stimulus, cm,j , θm,j , and λm are the grid cell’s spatial phase offset, orientation offset, and scaling
ratio. Grid cell orientations were taken to be ideal values about which the measurements presented
in [6] fluctuate, θm,j ∈ {−60◦ , 0◦ , 60◦ }. Z is a normalizing constant (≈ 2.857399), and fmax is
the grid cell’s maximum firing rate. In two dimensions, place cells have bivariate Gaussian tuning
σ 2 ρσ σ 
curves, with mean ξ ∈ [0, L] × [0, L], correlation, ρ ∈ [− 12 , 21 ], and covariance ρσ 1σ σ12 2 , where
1 2

2

σ1 and σ2 are chosen from the same range as the standard deviation in the 1D case.
A hybrid codebook: C codewords, of length N = P +

M
P

Jm , are generated by quantizing

m=1

positions on a linear track of length L, and represents the states of the grid and place cells when
stimulated with these positions. That is, in 1D, the hybrid code is given by C = {x(`) : ` ∈
{0, ∆L, 2∆L, · · · , (C − 1)∆L}}, where ∆L = L/C is the distance between sequential quantized
positions. In 2D, C is constructed by choosing locations, l, from the vertices of a square lattice
imposed on the plane, with unit area equal to (∆L)2 , and total area equal to L2 . Code rate, r =

C
N

(number of represented locations per neuron), measures spatial information resolution, i.e., for a
fixed L, a higher code rate, r, is obtained by lowering ∆L.

2.2

De-noising network

Two high capacity associative memory designs are considered to test the hybrid code’s resilience
to noise. In each case, the memory network is a bipartite graph consisting of N pattern neurons (i.e.
grid and place cells) and ni interneurons. In the unclustered design, all interneurons are connected
to a random set of pattern neurons. In the clustered configuration, the interneurons were split into
M distinct clusters of n interneurons per cluster, with each cluster connected to a distinct grid
module. Each cluster’s interneurons were connected randomly to pattern neurons, chosen from a
set consisting of every grid cell in the corresponding module, and every place cell.

4

(a)
module m=1
g1,1

g1,J1

g1,2

module m=M
gM,1 gM,2

c1

gM,JM

cnc-1

p1

p2

pP

p2

pP

cnc

(b)
module m=1
g1,1

c1,1

g1,J1

g1,2

c1,n

module m=M
gM,1 gM,2

cM,1

gM,JM

p1

cM,n

Figure 2: (a) Structure of an un-clustered de-noising network - considered as a baseline for comparison to to the neurophysiologically inspired systematic clustering scheme (b) Structure of a
systematically clustered de-noising network
We also consider a foil to this systematic clustering scheme: Grid and place cells are randomly
assigned to clusters. Figures 2a and 2b depict the general connectivity structure of the un-clustered
and clustered designs, respectively. In both the clustered and unclustered configurations, a neurally
plausible modified version of Oja’s subspace learning rule was applied to learn the code, i.e. a
sparse connectivity matrix is found such that the weights of connections from interneurons to
pattern neurons lie orthogonal to the code space (i.e., the space spanned by C) [16]. This way,
interneuron connectivity converges to the parity structure of the code and may be utilized in denoising operations.

3

Coding theoretic results
We begin our investigation of coding theoretic properties of the hybrid code by defining two

measures of redundancy for grid cells: µp and µo . Define µp , be a hybrid code configuration’s
spatial phase multiplicity, as the number of grid cells with the same phase in the same module.
In [17], it is revealed that there may be a highly non-uniform distribution of phases among grid cells.
Replication of phases allows us to investigate coding theoretic repercussions of this phenomenon.
 
We also set µo = J31 , the code’s orientation multiplicity (i.e. minimum number of replications
of a orientation offset in the first module). These replications are the main sources of grid cell
redundancy under consideration. We compare the systematic distributions of grid cell orientation
and phase to uniformly random choices for each. Inspired by [18], in order to investigate previously
unexplored properties of the grid cell component of the hybrid code, for each of these schemes, we
consider two distributions of grid cells to modules: uniform and non-uniform. In the non-uniform
5

case, Jm ∝

1
λm .

A scaling ratio of λ defines the scale of module m as λm = λ1 (λ)m−1 . Neural

recordings show that the smallest scale is λ1 ≈ 40cm (the value used here) [6].
We construct a codebook matrix, C, by placing elements of C in its rows. We computed
C) as a function of the grid scaling ratio. Normalized rank
normalized rank of the code, R = rank(
N
is an indicator of a code’s density, expressed as the fraction of possible code space occupied by a
particular code. The grid cell code is known to be dense [2]. This is especially pronounced when all
orientations and phases are chosen randomly (uniformly from [0, 2π] and [0, L]×[0, L] respectively),
where for all choices of other parameters, the hybrid code achieves full rank at low rate. When
only phases are chosen randomly, but orientations are chosen in accordance with the results in [6],
rank is also large at small rates (but not necessarily 1). When orientations are chosen randomly,
but phases are uniformly and deterministically distributed, there are hybrid codes with low ranks
at plausible rates. When µp = 1, a hybrid code with no place cells achieves the largest normalized
rank. Further, since place cells communicate redundant information, their inclusion also reduces
rank, which is precisely the trend observed in Figure 3. However, this appears to reverse when
µp > 1, for a sufficiently small number of participating place cells. This occurs because rendering
grid cells redundant lowers the rank of the grid-only hybrid code. Consequently, including place
cells increases rank, until the information contributed by the place cells reaches its maximum, at
which point the inclusion of additional place cells only lowers rank. In contrast to the uniformly
allocated grid cells, non-uniformity shifts the rank curve up, though the effect is less pronounced as
µp increases. In all cases, for a fixed rate, code rank has little dependence on λ. Figure 4 illustrates
the influence of µo on rank. Increasing µo decreases rank, even for less dramatic increases than
those shown. This is really because increases in µo correspond to decreases in code rate, which
lowers rank for most hybrid codes considered. Error bars (measuring SEM) are included due to the
stochastic nature of instantiating certain hybrid code parameters (e.g. ξ, which is always chosen
uniformly randomly from the set of quantized locations).
1

R

µp = 1
µp = 5
0.5

0
0

200

400

600

800

1000

P

Figure 3: Code rank (R) vs. number of place cells, P , for the hybrid code in 2D, with a uniform
allocation of grid cells; here (and in any other plot containing them) error bars show standard error
of the mean

6

µp = 1

1

0.6

0.6
M
M
M
M
M
M
M
M

0.2

0
1.2

1.4

=
=
=
=
=
=
=
=

4,
4,
4,
4,
4,
4,
4,
4,

J
J
J
J
J
J
J
J

1.6

=
=
=
=
=
=
=
=

20,
20,
20,
20,
30,
30,
30,
30,

P
P
P
P
P
P
P
P

=
=
=
=
=
=
=
=

R

0.8

R

0.8

0.4

µp = 5

1

0
10
20
100
0
10
20
100

0.4

0.2

0
1.2

1.8

λ

1.4

1.6

1.8

λ

Figure 4: Code rank (R) vs. scaling ratio (λ) for the hybrid code in 2D, a non-uniform allocation
of grid cells to modules for r = 1, for µo ∈ {6, 10}
We also computed rank, R, as a function of code rate, r. When phases are chosen randomly,
low rank is difficult to obtain at all but the smallest of code rates tested (r ∈ [0, 1] and µp > 1 may
result in low ranks if enough place cells are included). This indicates that biologically reasonable
redundancy reduces dimensionality so low ranks are achievable even at high rate. Non-uniformity
of the grid cell distribution (to modules) shifts this curve upwards in all cases except that in which
phases are chosen randomly, while orientations are chosen deliberately. Later, it will be shown that
this low dimensionality is important in constructing sparse and readily de-noisable representations
of space. Figure 5 demonstrates that without the redundancy introduced by increasing µp > 1, a
hybrid code that encodes in 90 neurons more than 90 locations in a 9 m2 environment has full rank.
However when µp > 1, there is a stark drop in the maximum rank achieved. Increasing code rate
further after the rank has saturated at one plateau eventually results in a marginal increase in rank.
As shown, when µp > 1, one may encode orders of magnitude more locations while maintaining
low dimensionality, with relatively few cells. This trend is observed in all configurations when
appropriate replication of grid phase is ensured. Therefore both dense and sparse hybrid codes
may be instantiated by proper choice of population size and redundancy parameters.

7

1

M
M
M
M
M
M

0.9
0.8
0.7

=
=
=
=
=
=

4,
4,
4,
4,
4,
4,

J
J
J
J
J
J

=
=
=
=
=
=

20,
20,
20,
20,
20,
20,

P
P
P
P
P
P

=
=
=
=
=
=

10, µp = 1
10, µp = 4
10, µp = 5
100, µp = 1
100, µp = 4
100, µp = 5

R

0.6
0.5
0.4
0.3
0.2
0.1
0
-1

-0.5

0

0.5

1

1.5

log10 (r)

2

2.5

3

3.5

Figure 5: Code rank (R) vs. log of code rate (log10 (r)) for the hybrid code in 2D, for µo = 6

20

20

d

10

= 1 , uniform
= 5 , uniform
= 1, non-uniform
= 5 , non-uniform

15

d

µp
µp
µp
µp

15

5

10
5

0

0
5

10

15

5

r

10

15

r

Figure 6: Minimum distance (d) vs r for the hybrid code in 1 and 2 dimensions, with M = 4,
J = 20, and P = 10, with deliberately organized grid cell phases and orientations
The code’s resilience to neural noise can be assessed by the minimum pairwise (euclidean)
distance between codewords, (d). Higher d (i.e., larger distances between codewords) corresponds
a more noise tolerant neural representation of space. We computed d as a function of rate, r, for
different grid scales (Fig. 6). For any λ, the there is a trade-off between d and r, which highlights
the dynamic nature of the code. Since rank tends to increase, with rate, this is also a tradeoff
between d and R. When the rate is low (a low resolution of location is targeted), d is larger, so
noise tolerance is larger. For a fixed r, a lower rank yields a stronger minimum distance, and lower
rates give more desirable tradeoffs between rank and d. For a typical hybrid codebook, for any
code rate, d exhibits little dependence on λ. The hybrid code for two-dimensional space performs
no worse than the hybrid code for one-dimensional space, and generates a better d for large code
8

rates in all configurations. Further, in 2D, the code with non-uniformly allocated grid cells has
significantly smaller d for a fixed r, while in 1D, this code achieved the largest d with µp = 1.
Thus, in 2D, for a fixed r (i.e. for codes of the same rate), the code with uniformly allocated
grid cells should exhibit measurably better de-noising performance. We confirm this prediction by
simulating the de-noising process and collecting statistics presented in Figures 10 through 14. In
contrast to a uniform distribution of grid cells to modules, non-uniformity shifts the curve down and
to the left (though in 1D, this shift is up and to the right). This is no surprise since non-uniform
allocation decreases the total number of grid cells. Increasing µp results in marginal decreases in d,
though d is less sensitive to increases in phase redundancy than decreases in population size (i.e.
codeword length).
For environments of a fixed size, x2max cm2 , and a hybrid code with N neurons, varying code rates
implies quantizations of space with varying unit width (∆L =

x√
max
).
C√

Thus the spatial sampling period, ∆L is inversely proportional to

Since rate, r =

C
N,

∆L =

x
√max .
Nr

r. In order to ensure we probed

reasonable code rates, we estimate the typical perceivable spatial period of a rat by considering its
running speed (ranging from .1 to 100

cm
s ),

and average ISI of 150ms, which bounds neural sampling

periods for space, implying that ∆L should lie somewhere in [0.15, 15]cm. Code rates considered in
this work assume ∆L < 15cm. To satisfy curiosity, and probe rate dependent phenomena at even
greater rates, the smallest ∆L considered is 0.0022cm.

4

Code construction via subspace learning

4.1

Code learning algorithms

Before we can use the de-noising system to correct corrupted codewords, it must learn (i.e.
adapt its weights for) the hybrid code. This process is complete when the interneurons may be
read to determine if the states of the pattern neurons map to a valid codeword. Formally, this
amounts to finding a connectivity matrix, W (Wi,j is the synaptic weight between interneuron i
and pattern neuron j), whose rows are approximately perpendicular to the code space. A procedure
to procure such a matrix is outlined in [16], and improved in [11]. These algorithms begin with
a random set of vectors, and for each, seeks a nearby vector orthogonal to C (i.e. a vector onto
which each element of C has minimal projection). We implement this in Algorithm 1 (a derivation
of this algorithm can be found in the appendix). In the clustered design, Algorithm 1 is applied
to each cluster’s local connectivity matrix. Note that here, all arithmetic on the synaptic weights,
Wi,j is performed in R, while arithmetic on states of neurons (i.e. their firing rates), is quantized
to the nearest integer in [0, Q − 1]. The maximum firing rate, Q − 1, is identical for all neurons.
With each update, w ← w − αt (y(x −

yw
)
kwk2

+ ηΓ(w, θ)), where θ is a sparsity threshold, η is a

penalty coefficient, y = xT w is the scalar projection of x onto w, and αt is the learning rate at
iteration t. Γ is a sparsity enforcing function, approximating the gradient of a penalty function,
m
P
g(w) =
tanh(σwk 2 ), which, for appropriate choices of σ, penalizes non-sparse solutions early in
k=1

the learning procedure [11].
9

Algorithm 1 Neural Learning
Input: set of C patterns, C, stopping point, 
Output: learned weights matrix, W
1:
2:

for rows, w, of W do
for t ∈ {1, ..., Tmax } do

3:

50·α0
αt ← max{ 50+log
(t) , 0.005}

4:

θt ←

5:

for c ∈ C do

6:

y ←c·w

7:

if kck >  then

10

θ0
t

αt ←

8:

α0
kck2

end if

9:

w ← Dale(update(c, w, αt , θt , η))

10:
11:

end for

12:

if kCw0 k <  then
break

13:
14:

end if

15:

t←t+1

16:

end for

17:

for components, wi of w do

18:

if |wi | ≤  then
wi ← 0

19:
20:
21:
22:

end if
end for
end for
As in [11], to speed up learning, we approximate Γ = ∇g with
(
wt : |wt | ≤ θt
Γ(wt , θt ) =
0 : otherwise

This update rule is a an improved approximation to Oja’s Hebbian learning algorithm [16], with
advantages in both biological plausibility and computational complexity. For connections of fixed
type (i.e. inhibitory vs excitatory), Oja’s rule alone is biologically dubious without the inclusion
of many interneurons to manage this change in type. Dale’s Principle states that real synaptic
connections change type rarely, if ever [19]. In accordance with this principle, our update rule does
not allow weights to change sign. This is accomplished after the updated weights are determined: If
the sign has changed after applying the update, set the new weight to a value just above (resp. just
below) zero if the previous weight was positive (resp. negative). Thus, when learning is complete,
these weights will be small in magnitude and are thresholded to zero.
In Algorithm 1, line 11 terminates the learning process if the sum of the projections of the
current weight vector on each pattern is no more than  away from zero, that is, if the current
10

weight vector is approximately orthogonal to the code space. Lines 17-21 perform a thresholding
operation that maps to zero any weight sufficiently small in magnitude. This is primarily to suppress
numerical errors and promote consistency, as in Line 11, we use  as a small positive constant. Note
that since the weights processed on each iteration are independent of those in other iterations, this
algorithm can be readily parallelized so that each interneuron learns its weights simultaneously.

4.2

Code learning results

Typical learned connectivity matrices and their associated degree distributions are found in
Figures 7 and 8. These demonstrate that for a typical hybrid code, the clustered network has
a sparser connectivity, with less variability in its sparsity compared to the un-clustered network.
This is because clustering enforces a tighter limit on the number of pattern neurons to which an
interneuron may connect. We simulated an ensemble of 4 modules of 20 grid cells each, together
with 20 place cells, which produced the following connectivity matrices and associated degree
distributions. Interestingly, in both cases, there are place cells (i.e. pattern neurons with index
exceeding 80) that are left unconnected to grid modules via interneurons. An illustration of the
learned weights matrix corresponding to a randomly clustered de-noising network was omitted, as

1

.5

10

10

20

20

interneurons

interneurons

normalized connection strength

it is sparser, but otherwise very similar to that of the un-clustered weights image.

30
40
50
60

30
40
50
60

0

pattern neurons 80

100

80

pattern neurons

100

Figure 7: Image of typical weights matrices learned by the de-noising network for a hybrid code

frequency of occurrence

with M = 4, J = 20 and P = 20, with modular clustering on the right

1
un-clustered
clustered

.8
.6
.4
.2
0
0

0.1

0.2

0.3

0.4

0.5

0.6

sparsity (degree/N)

Figure 8: Degree distributions of the connectivity matrices shown in Figure 7
11

Figure 9 depicts the average connection strength between place cells and grid modules, where
P
the connection strength between place cell p and grid module m is defined as n1i (
|wi,j wi,p |),
(i,j)

where i indexes interneurons, and j indexes grid cells in module m. Note here that connectivity
does not imply direct synaptic connection, but effective connectivity through interneurons. Results
were obtained from configurations with M = 4, J = 20 and P = 20; place cells in legend are
ordered by increasing receptive field size; connectivities depicted are averaged over 50 networks.
Place cells are ordered by increasing size of receptive field. This trend appears for any µp > 1. In the
modularly clustered case, average connectivity (between place cells and all grid modules) appears to
decrease with increasing place cell size, as compared to a random clustering which produces nearly
the same connectivity for each place cell. This phenomenon was not observed when grid phases or

average connection strength

grid orientations were chosen randomly. It does not appear in the un-clustered configuration.
1

modular clustering
random clustering

0.8
0.6
0.4
0.2
0
2

4

6

8

10

12

14

16

Place cell index (ordered by place field size)

18

20

Figure 9: Average connectivities between place cells (x-axis) and grid modules for configurations
with M = 4, J = 20, and P = 20; place cell indices are ordered from smallest to largest receptive
field size; grid cell phases were uniformly distributed across the environment and grid orientations
in agreement with the data presented in [6]; connectivities depicted are averaged over 50 networks

5

De-noising and decoding

5.1

De-noising and decoding algorithms

We implemented a Bit Flipping style neural de-noising process, which we applied to both the
clustered and un-clustered de-noising networks. For all configurations (clustered and un-clustered,
and for a fixed maximum number of de-noising iterations, the bit flipping algorithm has proven
to perform no worse that winner-take-all. Moreover, since it requires only the implementation
of additional parallel thresholding operations for each pattern neuron, its implementation is no
less biologically feasible. The goal of this algorithm is to recover the correct activity pattern,
x, which has been corrupted by noise, and as such, is currently (and errantly) represented by
a noisy version, xn = x + n, where n is this noise pattern. Since each weight vector is nearly
perpendicular to every pattern, for a matrix of weights, W , xn W 0 reveals inconsistencies in xn ,
which the de-noising algorithm seeks to correct in the feedback stage 1 . The clustered de-noising
1

To see this, consider that xn W 0 = (x + n)W 0 = xW 0 + nW 0 ≈ 0 + nW 0

12

process begins with Algorithm 3, in which each cluster attempts to detect errant pattern neurons.
If no errors are detected, the process is complete. Otherwise, algorithm 2 is invoked for each cluster
that detected errant neurons. Note that this de-noising mechanism differs from error correction
methods presented in [2] and [12] in that information contributed by place cells only reaches grid
cells through interneurons, and place information contributed by grid cells at module i only reaches
other modules through interneurons if connectivity (i.e. the hybrid code) allows.
In order to quantify the information content of the population, we estimated the location encoded by the population using a maximum likelihood decoder in 4 different schemes. Joint hybrid
decoding utilizes information from all cells. Grid (resp. place) only decoding utilizes information
from only grid (resp. place) cells. Grid decoding conditioned on place response performs decoding
using only information provided by the grid cells, however, the only candidate locations considered
for the estimate are those that are not impossible given the place cell activity.
Algorithm 2 Modular Recall
Input: local weights for this cluster, W , maximum number of iterations, Tmax , noisy subpattern,
x, feedback threshold, φ
Output: denoised subpattern, d
1:

d←p

2:

while t < Tmax do

3:

y ← xW 0

4:

r ← y0 W

5:

if kyk <  then

6:

break;

7:

end if

8:

f←

9:

for each pattern neuron,j do

|y0|·|W |
m
P
|W |
i=1

10:

if fj ≥ φ then fj = sign(xj )

11:

else fj = 0

12:

end if

13:

end for
d←d+f

14:

end while

13

Algorithm 3 Sequential de-noising
Input: local weights, Wi , for each cluster, i ∈ {1, ..., M }, noisy pattern, xn , stopping threshold, 
Output: denoised pattern, xd
1:

x d ← xn

2:

while t < Tmax and a cluster has an unsatisfied constraint do
for each cluster, i ∈ {1, ..., M } do

3:
4:

x ← subpattern corresponding to cluster i

5:

d ← Modular Recall(x, Wi )

6:

if |dWi | ≤  then
xd (cluster i’s subpattern indices) ← d

7:

end if

8:
9:

end for

10:

t←t+1
end while

11:

5.2

De-noising and Decoding Results

To measure the denoising performance of this system, we first perturb the states (i.e. firing
rates) of the grid and place cells by incrementing or decrementing randomly and clipping to the
boundaries of [0, Q − 1]. A pattern error occurs if after de-noising, any entry of the de-noised
pattern differs from the corresponding component of the original pattern. A symbol error occurs
each time any symbol of the de-noised pattern differs from the corresponding symbol of the correct
pattern. For identical populations of grid and place cells (M = 4, J = 20, and P = 10), in pattern
error rate, the clustered network dramatically outperforms the un-clustered (when the grid cells
have sufficient redundancy), and the modular clustering scheme always outperforms the random
clustering scheme. Figure 10 depicts pattern error rate (Ppe ) for a clustered hybrid code, with
varying phase multiplicity and biologically consistent orientation multiplicity. All other configurations (that is, those configurations that did not employ clustering in de-noising, or had randomly
selected phases or randomly selected orientations) had 100 percent pattern error rate even for a
small number of initial errors. This shows that for a small number of initial errors, the full pattern
of population activity corresponding to the correct location may be recovered, but that for many
noise-induced errors, this is rarely possible. That only the modularly clustered de-noising network
is able to achieve low Ppe (and only when µp is large enough) shows that the biological organization
of grid cells into discrete modules, is important for high quality self location in the presence of noise.
Further, clustering is the only way to acieve such a small Ppe , since no non-clustered de-noising
network consistently reduces Ppe below 0.99. It is no surprise that for a small number of errors, the
modularly clustered de-noising mechanism achieves a better Ppe when de-noising hybrid codes with
uniform allocations of grid cells to modules (as compared to non-uniform allocations of grid cells
to modules), as Figure 6 demonstrates that such codes tend to have a larger minimum distance
at any rate probed. This result also demonstrates that whether or not grid cells are distributed
uniformly to modules has a smaller impact on Ppe than µp .
14

pattern error rate

1

modular clustering, µp = 1
modular clustering, µp = 5
random clustering, µp = 1
random clustering, µp = 5
non-uniform, modular clustering, µp = 1
non-uniform, modular clustering, µp = 5

0.5

0
2

4

6

8

10

12

14

16

18

20

initial number of errors (E)

Figure 10: Pattern error rate vs. initial number of errors for a clustered hybrid code with M = 4,
J = 20, P = 10, with deliberately chosen spatial phases and orientations; all other pattern error
rates tested (i.e. those with random redundancy parameters or those with a non-clustering denoising
network) have Ppe = 1 for any initial number of errors

log symbol error rate

0
-1
modular clustering, µp = 1
modular clustering, µp = 5
random clustering, µp = 1
random clustering, µp = 5
no clustering, µp = 1
no clustering, µp = 5
log10(E) − log10(N)

-2
-3
-4
2

4

6

8

10

12

14

initial number of errors (E)

16

18

20

Figure 11: Symbol error rate vs. initial number of errors for clustered and non-clustered hybrid
codes; here, each code utilizes a uniform distribution of grid cells to modules, and deliberately
chosen spatial phases and orientations
Figure 11 shows symbol error rates of hybrid codes for several configurations with deliberately
chosen grid cell phases and orientations. This demonstrates that generally, clustered de-noising networks do not offer improved symbol error rate, Pse , compared to their un-clustered counterparts.
However, for a small initial number of errors, when the grid cells exhibit sufficient redundancy
in their phases, a randomly clustered de-noising network is only outperformed by a modularly
clustered network. Figure 12 shows Pse for a hybrid code with deliberately chosen phases and
orientations, de-noised by a modularly clustered network. Consistent with the observations on
pattern error rate, hybrid codes with grid cells uniformly allocated to modules achieve better Pse .
This may result directly from the fact that d is larger for such codes. On the other hand, it is
possible that the smaller population sizes resulting from a non-uniform allocation (thus allowing
fewer opportunities for grid cell redundancy) are behind both the improvement in denoising performance and the larger minimum distance. Plotted in both Figures 11 and 12 is a red, solid curve,
of errors
). This curve is a threshold between regions of desirable and unacceptable
log10 ( initial number
N

15

Pse (i.e. log10 (Pse ) for a network that performs no de-noising). To see this, consider a de-noising
network that does not change the initial number of errors, E. For this network, Pse =

E
N,

so

log10 (Pse ) = log10 (E) − log10 (N ). Surprisingly, figure 11 shows that for a small initial number of
errors, configurations with µp = 1 have log10 (Pse ) above this threshold, that is, they increase the
number of symbol errors! Figure 12 shows that this observation is independent of uniformity of the
distribution of grid cells to modules.

log symbol error rate

0
-1
-2
uniform, µp = 1
uniform, µp = 5
non-uniform, µp = 1
non-uniform, µp = 5
log10(E) − log10(N)

-3
-4
2

4

6

8

10

12

14

initial number of errors (E)

16

18

20

Figure 12: Symbol error rate vs. initial number of errors for uniform and non-uniform clustered
hybrid codes; here, each de-noising network employs the modular clustering scheme

×105
joint hybrid decoding
grid decoding conditioned on place response
grid only decoding
place only decoding

MSE of decoding (cm2 )

4

3

2

1

0
1

2

3

initial number of errors (E)

4

5

Figure 13: MSE of decoding after de-noising for a hybrid code with M = 4, J = 20, P = 10, and
µp = 5, for position in 2D, and deliberately chosen grid cell parameters
Figure 13 shows MSE of different decoding processes after de-noising for a Hybrid code with
M = 4, J = 20, P = 10, and µp = 5, for position in 2D, and deliberately chosen grid cell
parameters. This plot demonstrates that an ideal observer decoder which considers information
from all cells outperforms all others for any initial number of errors. This disparity may, in part,
be accounted for by the difference between the number of grid cells and the number of place cells.
Figure 14 shows MSE of joint hybrid decoding after de-noising for a hybrid code with µp = 5,
16

with position in 2D, for the configuration that achieved the best error correction performance in
both Ppe and Pse . This plot demonstrates that the code with grid cells distributed to modules
uniformly, with a modularly clustered de-noising network achieves the best decoding performance,
outperforming its non-uniformly distributed analogue. Since in 2D the code with a non-uniform
allocation of grid cells to modules had a smaller minimum distance (compared to the same code
with a uniform allocation of grid cells to modules), this result confirms our hypothesis that codes
with larger d may be de-noised more effectively. Further, this demonstrates (in a natural metric
of the stimulus space) that in the most redundant hybrid code considered, a modularly clustered
de-noising network is far superior to a randomly clustered or un-clustered one. These results are
unsurprising given the Ppe and Pse data presented earlier.
×105
uniform, modular clustering
uniform, random clustering
uniform, no clustering
non-uniform, modular clustering
non-uniform, random clustering
non-uniform, no clustering

3.5

MSE of decoding (cm2 )

3

2.5

2

1.5

1

0.5

0
1

2

3

4

5

initial number of errors (E)

Figure 14: MSE of decoding after neural de-noising (cm2 ) vs initial number of errors for a hybrid
code with µp = 5 and µo = 6.

6

Discussion
We demonstrated that both dense and sparse hybrid codes may be constructed by proper choice

of grid and place cell parameters. We also showed that in the presence of neural noise, the activity
of only some of these configurations may be consistently de-noised. It is somewhat surprising that
choosing µp > 1 produces a more noise resilient code (as shown in the de-noising performance
results) as the 2D hybrid codes with uniformly allocated grid cells and largest d are those with
unique spatial phases (i.e. µp = 1) (Figure 6). Further, this observation demonstrates that the
hybrid code for space may trade off efficiency of encoding, R, for improvements in de-noising
17

performance by distributing grid cells to modules non-uniformly [18]. Hybrid codes of widely
varying R, d, and r may be instantiated by choosing appropriate parameters for the populations
of grid and place cells, a fact that showcases the code’s adaptability. This means that the grid and
place cells may be chosen for participation in neural computations, whose performance relies on
assumptions other than those presented here, which insist on a low dimensional code space, and a
sparse connectivity matrix.
We demonstrate that the chosen de-noising network architecture performs quite well for hybrid
codes that fit its assumptions regarding rank, and poorly for those that do not. Additionally, we
assessed average connectivity between place cells of varying receptive field sizes and modules of
grid cells by analyzing the learned connectivity matrix. This analysis demonstrates that our model
place cells of smaller receptive field size are more strongly connected to grid modules, and that
they are most strongly connected to grid modules of the smallest scale. Such a result presents a
physiologically testable hypothesis. While difficult, two photon microscopy has been successfully
employed to accurately image the microscopic structure of nervous tissue [20]. One way to estimate
connection strength between real neurons is to count the number of boutons expressed on the presynaptic neurons, assuming that weight should be proportional to this number, though there are
certainly easier ways to estimate connection strength [21]. Thus, if groups of place cells connected
via interneurons to several distinct grid modules may be identified, this theoretical prediction can
be confirmed or refuted. Another interesting experiment is made possible by recent advances in
optogenetics, which enable single cell resolution of network activity for a population of inoculated
cells (e.g. a collection of grid cells, as in [22]). While technically challenging due to the physical
separation of each population in the brain, it should be possible to image simultaneous activity
of grid and place cells at high temporal precision [23]. From these measurements, for a set of
quantized locations, simultaneous firing rates may be estimated [24]. Then, the rank, rate, and
minimum distance of this empirical codebook may be computed to offer insight about limits of
noise tolerance of real spatial navigation circuitry. Of particular interest is discovering the extent
to which neural noise transiently varies such attributes for grid and place cells in real brains, and
how these coding theoretic properties adapt (if at all) to changes in speed, context, and other
variables.
In Figures 10, 12, 13, and 14, we demonstrate the differences in performance of each network
structure, and of the various decoding algorithms. The universal improvements from place only
decoding to joint hybrid decoding show that highly accurate position estimation can be significantly
more difficult without both populations of cells. The discrepancy between ’grid only decoding’
and ’grid decoding conditioned on place response’ shows that even utilizing place cell information
indirectly (by eliminating places deemed impossible given the state of the place cell population)
yields a sizable improvement in decoding accuracy when there are many place cells, or when place
cells are less noisy than grid cells. That the modular clustering network universally outperforms
the corresponding randomly clustered network implies that the physiological organization of grid
cells by their scale may provide a computational advantage in de-noising and decoding. This may
be because the un-clustered network is essentially a randomly clustered network that does not
18

take advantage of synergistic cluster computing. In any cluster, both grid cells and place cells
are able to correct each others’ errant activity. However, under modular clustering, in order for
a grid cell in module i to correct the activity of a grid cell in a different module j, the activity
of each neuron in module i must be correct so that the activity of place cells (connected to both
modules i and j) will contradict and correct the erroneous activity. It should be noted here that
the de-noising interneurons are a hypothetical construct and need not reside in the hippocampus,
or MEC. Furthermore, this work is not intended to convince readers of the necessity or existence
of these cells, only to demonstrate tangible coding theoretic advantages to their implementation
alongside this neural code for position.
Further development on neural codes for space include studying coding theoretic properties of
more complete navigational codes including head direction cells, boundary vector cells, and time
cells [25–27]. Even with these classes of neuron, the hybrid code might be unable to encode and
de-noise path information, unless there is a higher level structure that provides the encoded activity
in the correct sequence. One strong candidate solution for this is to include so called hippocampal
time cells. Just as place cells code for distinct locations on paths through space, time cells encode
ordered moments in a temporally ordered sequence of events, precisely the information, which,
when coupled with location, allows for the encoding of paths [28].

References
[1] A. D. Ekstrom, M. J. Kahana, J. B. Caplan, T. A. Fields, E. A. Isham, E. L. Newman, and
I. Fried, “Cellular networks underlying human spatial navigation,” Nature, vol. 425, no. 6954,
pp. 184–188, 2003.
[2] I. R. Fiete, Y. Burak, and T. Brookings, “What grid cells convey about rat location,” The
Journal of Neuroscience, vol. 28, no. 27, pp. 6858–6871, 2008.
[3] M. Fyhn, T. Hafting, M. P. Witter, E. I. Moser, and M.-B. Moser, “Grid cells in mice,”
Hippocampus, vol. 18, no. 12, pp. 1230–1238, 2008.
[4] M. M. Yartsev, M. P. Witter, and N. Ulanovsky, “Grid cells without theta oscillations in the
entorhinal cortex of bats,” Nature, vol. 479, no. 7371, pp. 103–107, 2011.
[5] B. A. Strange, M. P. Witter, E. S. Lein, and E. I. Moser, “Functional organization of the
hippocampal longitudinal axis,” Nat Rev Neurosci, vol. 15, no. 10, pp. 655–669, Oct. 2014.
[Online]. Available: http://dx.doi.org/10.1038/nrn3785
[6] H. Stensola, T. Stensola, T. Solstad, K. Frøland, M.-B. Moser, and E. I. Moser, “The entorhinal
grid map is discretized,” Nature, vol. 492, no. 7427, pp. 72–78, 2012.
[7] T. Bonnevie, B. Dunn, M. Fyhn, T. Hafting, D. Derdikman, J. L. Kubie, Y. Roudi, E. I.
Moser, and M.-B. Moser, “Grid cells require excitatory drive from the hippocampus,” Nature
neuroscience, vol. 16, no. 3, pp. 309–317, 2013.
19

[8] L. Muessig, J. Hauser, T. J. Wills, and F. Cacucci, “A developmental switch in place cell
accuracy coincides with grid cell maturation,” Neuron, vol. 86, no. 5, pp. 1167–1173, 2015.
[9] J. J. Hopfield, “Neural networks and physical systems with emergent collective computational
abilities,” Proceedings of the national academy of sciences, vol. 79, no. 8, pp. 2554–2558, 1982.
[10] R. J. McEliece, E. C. Posner, E. R. Rodemich, and S. S. Venkatesh, “The capacity of the
hopfield associative memory,” Information Theory, IEEE Transactions on, vol. 33, no. 4, pp.
461–482, 1987.
[11] A. H. Salavati, K. R. Kumar, and A. Shokrollahi, “Nonbinary associative memory with exponential pattern retrieval capacity and iterative learning,” Neural Networks and Learning
Systems, IEEE Transactions on, vol. 25, no. 3, pp. 557–570, 2014.
[12] M. Stemmler, A. Mathis, and A. V. Herz, “Connecting multiple spatial scales to decode the
population activity of grid cells,” Science Advances, vol. 1, no. 11, p. e1500816, 2015.
[13] A. Karbasi, A. H. Salavati, A. Shokrollahi, and L. R. Varshney, “Noise facilitation in associative
memories of exponential capacity,” Neural computation, 2014.
[14] A. Karbasi, A. H. Salavati, and A. Shokrollahi, “Iterative learning and denoising in convolutional neural associative memories,” in Proceedings of The 30th International Conference on
Machine Learning, 2013, pp. 445–453.
[15] C. Curto, V. Itskov, K. Morrison, Z. Roth, and J. L. Walker, “Combinatorial neural codes
from a mathematical coding theory perspective,” 2012.
[16] E. Oja and T. Kohonen, “The subspace learning algorithm as a formalism for pattern recognition and neural networks,” in Neural Networks, 1988., IEEE International Conference on.
IEEE, 1988, pp. 277–284.
[17] D. Wennberg, “The distribution of spatial phases of grid cells,” 2015.
[18] N. W. Mosheiff, H. Agmon, A. Moriel, and Y. Burak, “An efficient coding theory for a dynamic
trajectory predicts non-uniform allocation of grid cells to modules in the entorhinal cortex,”
arXiv preprint arXiv:1601.02948, 2016.
[19] J. Eccles, “From electrical to chemical transmission in the central nervous system,” Notes and
records of the Royal Society of London, vol. 30, no. 2, pp. 219–230, 1976.
[20] K. Svoboda and R. Yasuda, “Principles of two-photon excitation microscopy and its applications to neuroscience,” Neuron, vol. 50, no. 6, pp. 823–839, 2006.
[21] G.-q. Bi and M.-m. Poo, “Synaptic modifications in cultured hippocampal neurons:
Dependence on spike timing, synaptic strength, and postsynaptic cell type,” Journal
of Neuroscience, vol. 18, no. 24, pp. 10 464–10 472, 1998. [Online]. Available:
//jneurosci.org/content/18/24/10464
20

http:

[22] C. Sun, T. Kitamura, J. Yamamoto, J. Martin, M. Pignatelli, L. J. Kitch, M. J. Schnitzer,
and S. Tonegawa, “Distinct speed dependence of entorhinal island and ocean cells, including
respective grid cells,” Proceedings of the National Academy of Sciences, vol. 112, no. 30, pp.
9466–9471, 2015. [Online]. Available: http://www.pnas.org/content/112/30/9466.abstract
[23] B. F. Grewe, D. Langer, H. Kasper, B. M. Kampa, and F. Helmchen, “High-speed in vivo
calcium imaging reveals neuronal network activity with near-millisecond precision,” Nature
methods, vol. 7, no. 5, pp. 399–405, 2010.
[24] L. Theis, P. Berens, E. Froudarakis, J. Reimer, M. R. Rosón, T. Baden, T. Euler, A. S. Tolias,
and M. Bethge, “Benchmarking spike rate inference in population calcium imaging,” Neuron,
vol. 90, no. 3, pp. 471–482, 2016.
[25] C. Lever, S. Burton, A. Jeewajee, J. O’Keefe, and N. Burgess, “Boundary vector cells in the
subiculum of the hippocampal formation,” The journal of neuroscience, vol. 29, no. 31, pp.
9771–9777, 2009.
[26] D. M. Salz, Z. Tiganj, S. Khasnabish, A. Kohley, D. Sheehan, M. W. Howard, and H. Eichenbaum, “Time cells in hippocampal area ca3,” Journal of Neuroscience, vol. 36, no. 28, pp.
7476–7484, 2016.
[27] J. S. Taube, R. U. Muller, and J. B. Ranck, “Head-direction cells recorded from the postsubiculum in freely moving rats. i. description and quantitative analysis,” The Journal of
neuroscience, vol. 10, no. 2, pp. 420–435, 1990.
[28] C. J. MacDonald, K. Q. Lepage, U. T. Eden, and H. Eichenbaum, “Hippocampal ”time cells”
bridge the gap in memory for discontiguous events,” Neuron, vol. 71, no. 4, pp. 737–749, 2011.
[29] E. Oja and J. Karhunen, “On stochastic approximation of the eigenvectors and eigenvalues of
the expectation of a random matrix,” Journal of mathematical analysis and applications, vol.
106, no. 1, pp. 69–84, 1985.
[30] L. Xu, A. Krzyzak, and E. Oja, “Neural nets for dual subspace pattern recognition method,”
International Journal of Neural Systems, vol. 2, no. 03, pp. 169–184, 1991.

7

Appendix

7.1

Subspace learning

In [29], the authors propose an algorithm that is capable of computing a basis for the null space
of a random matrix, A, which is assumed to be the expected value of sample matrices, At . The
update rule for the matrix whose columns are the resulting basis vectors is
W̃t = Wt−1 + At−1 Wt−1 αt−1

21

(3)

Wt = W̃t Rt−1 ,

(4)

where αt is a diagonal (and compatible) matrix of gain factors. As in [29], equations 3 and 4 may
be re-written as operations on column vectors, wt .
w̃t = wt−1 + αt−1 At−1 wt−1

(5)

w̃t
,
(6)
kw̃t k
in which αt is the gain factor corresponding to the current column. This number may be equivalently
wt =

understood as a learning rate. Indeed in [30], the authors show that for appropriate choices of At ,
the update rule is a form of anti-Hebbian learning. In [29] the authors prove convergence of this
algorithm to the eigenvectors of A corresponding to the largest eigenvalues. Further, when At is
replaced by −At , wt converges to the eigenvectors of A corresponding to the smallest eigenvalues.
In [29], it is demonstrated that by combining equations 5 and 6, expanding as a power series in αt ,
and ignoring second (and higher) order terms, we arrive at
wt = wt−1 + αt−1 (At−1 wt−1 −

wtT At−1 wt−1
wt−1 ).
T w
wt−1
t−1

(7)

The authors of [11] choose At = (xTt xt )Pxt = xt xTt , the product of projections onto the space
spanned by xt , and define yt = xTt wt = wtT xt . In [29], it is mentioned that this update rule finds
eigenvectors corresponding to the largest eigenvalue of At , or those corresponding to the smallest
eigenvalues of −At , when this matrix is used instead. Since At is a projection matrix, it has
rank 1. Thus it has one eigenvector with non-zero eigenvalue, xt , and dim(x) − 1 eigenvectors
with eigenvalue 0. Each of these eigenvectors, v, is guaranteed to be perpendicular to x because
At v = 0v = 0, that is, the v’s projection onto x has magnitude 0. By choosing xt ∈ C, with the
aforementioned choice for At , this algorithm should compute vectors approximately perpendicular
to the code space.
Now, we may rewrite equation 7 as
wt = wt−1 − αt−1 xt−1 xTt−1 wt−1 + αt−1
= wt−1 − αt−1 yt−1 xt−1 + αt−1

T x
T
wt−1
t−1 xt−1 wt−1
wt−1
kwt−1 k2

2
yt−1
wt−1 .
kwt−1 k2

(8)

To obtain a sparse basis for null(C), one may add to equation 8 a regularizing term that penalizes
non-sparse solutions. In particular, using ηΓ(wt−1 , θt−1 ), as considered in [11], to arrive at
wt = wt−1 − αt−1 (yt−1 (xt−1 −

7.2

yt−1 wt−1
)) − αt−1 ηΓ(wt−1 , θt−1 ).
kwt−1 k2

(9)

Choices of parameters

In learning, normalized weights are initialized randomly with degree d4 loge (n)e, where n is
the length of the weight vector. We used, θ0 = 0.031,  = C10−3 , η = 0.075, and α0 = 0.95. In
de-noising, we set φ = 0.95.
22

