arXiv:1606.01552v1 [cs.NE] 5 Jun 2016

CBMM Memo No. 049

June 7, 2016

View-tolerant face recognition and Hebbian
learning imply mirror-symmetric neural tuning to
head orientation
by
Joel Z. Leibo1 , Qianli Liao1 , Winrich Freiwald1,2 , Fabio Anselmi1,3 , Tomaso Poggio1
1: Center for Brains, Minds, and Machines and McGovern Institute for Brain Research at MIT, Cambridge, MA, USA
2: Laboratory of Neural Systems, The Rockefeller University, New York, NY, USA
3: Istituto Italiano di Tecnologia, Genova, Italy

Abstract:

The primate brain contains a hierarchy of visual areas, dubbed the ventral stream, which
rapidly computes object representations that are both specific for object identity and relatively robust
against identity-preserving transformations like depth-rotations [33, 32, 23, 13]. Current computational
models of object recognition, including recent deep learning networks, generate these properties through
a hierarchy of alternating selectivity-increasing filtering and tolerance-increasing pooling operations, similar to simple-complex cells operations [46, 8, 44, 29]. While simulations of these models recapitulate
the ventral stream’s progression from early view-specific to late view-tolerant representations, they fail
to generate the most salient property of the intermediate representation for faces found in the brain:
mirror-symmetric tuning of the neural population to head orientation [16]. Here we prove that a class of
hierarchical architectures and a broad set of biologically plausible learning rules can provide approximate
invariance at the top level of the network. While most of the learning rules do not yield mirror-symmetry
in the mid-level representations, we characterize a specific biologically-plausible Hebb-type learning rule
that is guaranteed to generate mirror-symmetric tuning to faces tuning at intermediate levels of the architecture.

This work was supported by the Center for Brains, Minds and Machines
(CBMM), funded by NSF STC award CCF - 1231216.

1

The ventral stream rapidly computes image representations that are simultaneously tolerant of identitypreserving transformations and discriminative enough to support robust recognition. The ventral stream
of the macaque brain contains discrete patches of cortex that support the processing of images of faces
[52, 53, 28, 2]. Face patches are selectively interconnected to form a face-processing network [36]. Face
patches are arranged along an occipito-temporal axis (from the middle lateral (ML) and middle fundus (MF)
patches, through the antero-lateral face patch (AL), and culminating in the antero-medial (AM) patch [51]
(Fig. 1-A)) along which response latencies increase systematically from ML/MF via AL to AM, suggesting
sequential forward-processing [16].
Face patches differ qualitatively in how they represent identity across head orientations [16]. Neurons in
the ML/MF patches are view-specific, while neurons in AM approach view-invariance. Furthermore, spatial
position and size invariance increase from ML/MF to AL, and further to AM [16]. These properties of the
face-processing network replicate the general trend of the ventral stream as summarized in [32, 43, 13]
and conform to the concept of a feedforward processing hierarchy.

A

B

ML

AL

AM

orientation

Figure 1: Schematic of the macaque face-patch system [36, 54, 16]. (A) Side view of computer-inflated
macaque cortex with six areas of face-selective cortex (red) in the temporal lobe together with connectivity
graph (orange). Face areas are named based on their anatomical location: PL, posterior lateral; ML;
middle lateral; MF, middle fundus; AL, anterior lateral; AF, anterior fundus ; AM, anterior medial (3), and
have been found to be directly connected to each other to form a face-processing network [36]. Recordings
from three face areas, ML, AL, AM, during presentations of faces at different head orientations revealed
qualitatively different tuning properties, schematized in B. (B) Prototypical ML neurons are tuned to head
orientation, e.g., as shown, a left profile. A prototypical neuron in AL, when tuned to one profile view,
is tuned to the mirror-symmetric profile view as well. And a typical neuron in AM is only weakly tuned
to head orientation. Because of this increasing invariance to in-depth rotation, increasing to invariance to
size and position (not shown) and increased average response latencies from ML to AL to AM, it is thought
that the main AL properties, including mirror-symmetry, have to be understood as transformations of ML
representations, and the main AM properties as transformations of AL representations [16].

Several hierarchical models of object recognition [17, 40, 43, 9] and face recognition [8, 29, 14] feature
a progression from view-specific early processing stages to view-invariant later processing stages similar
to ML/MF and AM, respectively. Simulations have shown that view-based models can achieve an AM-like
representation by successively pooling the responses of view-tuned units like those found in the early
processing stage ML/MF. The theoretical underpinnings of this property are described in the Appendix
(section 1.1).
Neurons in the intermediate face area AL, but not in preceding areas ML/MF, exhibit mirror-symmetric
head orientation tuning [16]. That is, an AL neuron tuned to one profile view of the head typically responds similarly to the opposite profile, but not to the front view (Fig. 1-B). This phenomenon is not
predicted by simulations of classical and current view-based computational models of the ventral stream
2

i

η(wi . x)

η(wi . x)
x

HMAX C1

HMAX S1

Tuning
Pooling

Figure 2: Illustration of the model. Inputs are encoded in HMAX C1 [43], then projected onto wi . In the
view-based model the wi represent faces at specific views. In the Oja-model, the wi are principal components. Units in the output layer pool over all the units in the previous layer corresponding to projections
onto the same template individual’s views (view-based model) or PCs (Oja-model).

[59]. In this paper we ask why the primate brain may compute a mirror symmetric representation as a
necessary intermediate step towards invariant face-representation and what this tells us about the brain’s
mechanisms of learning.

Results
Assumptions underlying the model
We consider a feedforward face-processing hierarchy as a model for how the ventral stream rapidly computes invariant representations. Invariant information can be decoded from inferotemporal cortex, and the
face areas within it, roughly 100ms after stimulus presentation [23, 34]. This is too fast of a timescale for
feedback to play a large role [23, 50, 27]. Thus while the actual face processing system might operate
in other modes as well, all indications are that fundamental properties of shape-selectivity and invariance
need to be explained as a property of feedforward processing.
The population of neurons in ML/MF is highly face selective [53] and incoming information can be thought
of as passing through a face-likeness filter. We thus assume the existence of a functional gate that routes
only images of face-like objects at the input of the face system. The existence of large “face-like” templates
or filters explains many of the so-called holistic effects of face perception, including face inversion and the
composite face [61] effect [49, 14]. This property has one further computational implication: it provides an
automatic face-specific gating mechanism to the face-processing system.
We make the standard assumption that a neuron’s basic operation is a pooled dot product between inputs
x and synaptic weight vectors {wi }, yielding complex-like cells as

|G|
E
1 X D
µ (x) =
η( x, gi wk )
|G|
k

i=1

3

(1)

where η : R → R is a nonlinear function e.g., squaring as in [1]. We suppose that gi ∈ G are image plane
transformations corresponding to rotations in depth of the face. Note that G is a set of transformations but
it is not a group (see Appendix section 1.1). We call µ
~ (x) ∈ RK the signature of image x.

Approximate view invariance
The model of Eq. (1) encodes a novel face by its similarity to a set of stored template faces. For example, the gi wk could correspond to views i of each of a set of well-known individuals k from an early
developmental period e.g., parents, caretakers, etc. One could regard the acquisition of this set of familiar
faces as the algorithm’s (unsupervised) training phase. To see why the algorithm works, consider that
whenever wik encodes a non-matching orientation to I , the value of x, gi wk will be very low. Among the
wk tuned to the correct orientation, there will be a range of response values since different template faces
will have different levels of similarity to I . When the novel face appears at a different orientation, the only
effect is to change which specific view-tuned units carry its signature. Since the pooled neural response
is computed by summing over these, the large responses carrying the signature will dominate. Thus the
pooled neural response will be approximatively unchanged by rotation (see the Appendix section 1). Since
these models are based on stored associations of frames, they can be interpreted as taking advantage of
temporal continuity to learn the simple-to-complex wiring from their view-specific to view-tolerant layers.
They associate temporally adjacent frames from the video of visual experience as in, e.g., [26].
The computational insight enabling depth-rotation tolerant representations to be learned from experience
is that, due to properties of how objects move in the world, temporally adjacent frames (the gi wk ) almost
always depict the same object [22, 48, 15, 58, 10, 26]. Short videos containing a face almost always contain multiple views of the same face. There is considerable evidence from physiology and psychophysics
that the brain employs a temporal-association strategy of this sort [35, 56, 12, 30, 57, 31]. Thus, our
assumption here is that in order to get invariance to non-affine transformations (like rotation in depth), it is
necessary to have a learning rule that takes advantage of the temporal coherence of object identity.
More formally, this procedure achieves depth-rotation tolerance because the set of rotations in depth
approximates the group structure of affine transformations in the plane (see Appendix section 1). For the
latter case, there are theorems guaranteeing invariance without loss of selectivity by operations resembling
the convolution in space performed by simple cells and the pooling done by complex cells [5].
Furthermore, [29] showed that Eq. 1 is approximately invariant to rotations in depth for x a face, provided
the templates wk also correspond to images of faces. For each template wk , the rotated views {gi wk , i =
1, · · · , |G|} must have been observed and stored. The η( x, gi wk ) can be interpreted as the output
of “simple” cells each with tuning gi wk when stimulated with image x. In a similar way µk (x) can be
interpreted as the activity of the “complex” cell indexed by k .

Biologically plausible learning
The simple-complex algorithm described above can provide an invariant representation but relies on a biologically implausible learning step: storing a set of discrete views observed during development. Instead
we propose a more biologically plausible mechanism: Hebb-like learning [21] at the level of simple cells
(see Equation (2)). Instead of storing separate frames, cortical neurons exposed to the rotation in depth
of a face update their synaptic weights according to a Hebb-like rule, effectively becoming each tuned
to one of a set of basis functions corresponding to different combinations of the set of views. Different
Hebb-like rules lead to different sets of basis functions such as Independent Components (IC) or Principal
Components (PC). Since each of the neurons become tuned to one of these basis functions instead of
one of the views, a set of basis functions replaces the gi wk (for a given k ) in the pooling Equation (1). The
question is whether invariance is still present under this new tuning.
4

The surprising answer is that most unsupervised learning rules will learn approximate invariance to viewpoint when provided with the appropriate training set (see Appendix section 2 for a proof). In fact, unsupervised Hebb-like plasticity rules such as Oja’s, Foldiak’s trace rule, and ICA provide a basis that when
used in the pooling equation provide invariance. Supervised learning rules such as backpropagation also
satisfy the requirement as long as the training set is appropriate.
In the following we consider as an example a simple Hebbian learning scheme called Oja’s rule [37, 38].
At this point we are concerned only with establishing the model and why it computes a view-tolerant face
representation. For this purpose we could use any of the other learning rules—like Foldiak’s trace rule or
ICA—but we focus on the Oja rule because it will turn out to be of singular relevance for mirror symmetry.
The Oja rule can be derived as the first order expansion of a normalized Hebb rule. The assumption of
this normalization is plausible, because normalization mechanisms are widespread in cortex [55].
For learning rate α, Oja’s rule is

∆w = α(xy − y 2 w) = α(xx| w − (w| xx| w)w) .

(2)

The original paper of Oja showed that the weights of a neuron updated according to this rule will converge
to the top principal component (PC) of the neuron’s past inputs, that is to an eigenvector of the input’s
covariance C . Thus the synaptic weights correspond to the solution of the eigenvector-eigenvalue equation Cw = λw. Plausible modifications of the rule—involving added noise or inhibitory connections with
similar neurons—yield additional eigenvectors [45, 38]. This generalized Oja rule can be regarded as an
online algorithm to compute the principal components of incoming stream of vectors, in our case, images.
What is learned and how it is stored depends on the choice of a timescale over which learning takes
place since learning is dictated by the underlying covariance C of the inputs (see Appendix, section
3). In order for familiar faces to be stored so that the neural response modeled by Eq. (1) tolerates
rotations in depth of novel faces, we propose that Oja-type plasticity leads to representations for which
the wik are given by principal components (PCs) of an image sequence depicting depth-rotation of face
k . Consider an immature functional unit exposed, while in a plastic state, to all depth-rotations of a
face. Oja’s rule will converge to the eigenvectors corresponding to the top r eigenvalues and thus to the
subspace spanned by them. The Appendix, section 2 shows that for each template face k the signature
P
µk (x) = ri=1 η( x, wik ) obtained by pooling over all PCs represented by different wik is an invariant. This
is analogous to Eq. (1) with gi wk replaced by the i-th PC. The appendix also shows that other learning
rules for which the solutions are not PCs but a different set of basis functions, generate invariance as
well—for instance, independent components (see Appendix section 2).

Empirical evaluation of view-invariant face recognition performance
View-invariance of the two models was assessed by simulating a sequence of same-different pair-matching
tasks, each demanding more invariance than the last. In each test, 600 pairs of face images were sampled
from the set of faces with orientations in the current testing interval. 300 pairs depicted the same individual
and 300 pairs depicted different individuals. Testing intervals were ordered by inclusion and were always
symmetric about 0◦ , the set of frontal faces; i.e., they were [−r, r] for r = 5◦ , . . . , 95◦ . The radius of the
testing interval r, dubbed the invariance range, is the abscissa in Fig. 3.
To classify an image pair (a, b) as depicting the same or a different individual, the cosine similarity
(a · b)/(kakkbk) of the two representations was compared to a threshold. The threshold was varied
systematically in order to compute the area under the ROC curve (AUC), reported on the ordinate of Fig.
3. AUC declines as the range of testing orientations is widened. As long as enough PCs are used, the proposed model performs on par with the view-based model. It even exceeds its performance if the complete
set of PCs is used. Both models outperform the baseline HMAX C1 representation (Fig. 3).
5

Example Test Pairs

1

C1-based
View-based
Top 4 PCs
Top 15 PCs
All 39 PCs

Same ID?

Same ID?

Area Under ROC Curve

0.9

0.8

0.7

0.6

0.5

0

20

40

60

Invariance radius (deg.)

80

Figure 3: Model performance at the task of same-different pair matching as a function of the extent of
depth rotations appearing in the test set (the invariance range of the task). All models were based on
HMAX C1 features [46].

Mirror symmetry
Consider the the case where, for each of the templates wk , the developing organism has been exposed
to a sequence of images showing a single face rotating from a left profile to a right profile. Faces are
approximately bilaterally symmetric. Thus, for each face view gi wk , its reflection over the vertical midline
g−i wk will also be in the training set. It turns out that this property—along with the assumption of Oja
plasticity, but not other kinds of plasticity—is sufficient to explain mirror symmetric tuning curves. The
argument is as follows.
Consider a face, x and its orbit in 3D w.r.t. the rotation group:

Ox = (r0 x, · · · , rN x).
where r is a rotation matrix in 3D, w.r.t., e.g., the z axis.

Projecting onto 2D we have

P (Ox ) = (P (r0 x), · · · , P (rN x)).
Note now that, due to the bilateral symmetry, the above set can be written as:

P (Ox ) = (x0 , · · · , x N , Rx1 , · · · , Rx N ).
2

2

where xn = P rn x, n = 1, · · · , N/2 and R is the reflection operator. Thus the set consists of a collection
of orbits w.r.t. the group G = {e, R} of the templates {x1 , · · · , xN/2 }.
This property of the training set is used in the appendix in two ways. First, it is needed in order to show
that the signature µ(x) computed by pooling over the solutions to any equivariant learning rule, e.g., Hebb,
Oja, Foldiak, ICA, or supervised backpropagation learning, is approximately invariant to depth-rotation
(sections 1 – 2).
Second, in the specific case of the Oja learning rule, it is this same property of the training set that is used
to prove that the solutions for the weights (i.e., the PCs) are either even or odd (section 3). This in turn
implies that the penultimate stage of the signature computation: the stage where η(hw, x, )i is computed,
will have orientation tuning curves that are either even or odd functions of the view angle.
6

A

PC 1

1.5

Response

1

1.5

2

1.5

1
0.5

1

−0.5

0

0.5

−1

−0.5

−1.5

−1
0

50

100

−1.5
−100

Orientation (deg.)

1.5

1
0.5
0

0

−0.5

−0.5

−1

−1
−50

0

50

100

−1.5
−100

PC 4

2.5
2

0

−50

PC 3

3
2.5

0.5

−2
−100

B

PC 2

2

−50

0

50

100

−1.5
−100

−50

0

50

100

Figure 4: Mirror symmetric orientation tuning of the raw pixels-based model (A) (wi · xθ )2 as a function of
the orientation of xθ . Here each curve represents a different PC. (B) Solutions to the Oja equation (wi )
visualized as images. They are either symmetric or antisymmetric about the vertical midline.

Finally, to get mirror symmetric tuning curves like those in AL, we need one final assumption: the nonlinearity before pooling at the level of the “simple” cells in AL must be an even nonlinearity such as η(z) = z 2 .
This is the same assumption as in the "energy model” of [1]. This assumption is needed in order to predict
mirror symmetric tuning curves for the neurons corresponding to odd solutions to the Oja equation. The
neurons corresponding to even solutions have mirror symmetric tuning curves regardless of whether η is
even or odd.
An orientation tuning curve is obtained by varying the orientation of the test image θ. Fig. 4-A shows
example orientation tuning curves for the model based on a raw pixel representation. It plots (hxθ , wi i)2 as
a function of the test face’s orientation for five example units tuned to features with different corresponding
eigenvalues. All of these tuning curves are symmetric about 0◦ —i.e., the frontal face orientation. Fig. 5-A
shows how the three populations in the C1-based model represent face view and identity and Fig. 5-B
shows the same for populations of neurons recorded in ML/MF, AL, and AM. The model is the same one
as in Fig. 3.
In contrast to the Oja/PCA case, we show through a simulation analogous to Fig. 5 that ICA does not yield
mirror symmetric tuning curves (appendix section 4). Though this is an empirical finding for a specific
form of ICA, we do not expect, based on our proof technique for the Oja case, that a generic learning rule
would predict mirror symmetric tuning curves.
These results imply that if neurons in AL learn according to a broad class of Hebb-like rules, then there
will be invariance to viewpoint. Different AM cells would come to represent components of a view-invariant
signature—one per neuron. Each component can correspond to a single face or to a set of faces, different
for each component of the signature. Additionally, if the learning rule is of the Oja-type and the output
nonlinearity is, at least roughly, squaring, then the model predicts that on the way to view invariance,
mirror-symmetric tuning emerges, as a necessary consequence of the intrinsic bilateral symmetry of faces.

7

A

B

ML/MF

AL

AM

Figure 5: Population representations of face view and identity (A) Model population similarity matrices (B)
Neural population similarity matrices from [16].

Discussion
The model discussed here provides a computational account of how experience and evolution may wire up
the ventral stream circuitry to achieve the computational goal of view-invariant face recognition. Neurons
in top-level face patch AM maintain an explicit representation selective for face identity and tolerant to
position, scale, and viewing angle [16] (along with other units tolerant to identity but selective for other
variables such as viewing angle). The approach in this paper explains how this property may arise in a
feed-forward hierarchy. To the best of our knowledge, it is the first account that provides a computational
explanation of why cells in the face network’s penultimate processing stage, AL, are tuned symmetrically
to head orientation.
Our assumptions about the architecture for invariance conform to i-theory [4, 5] which is a theory of
invariant recognition that characterizes and generalizes the convolutional and pooling layers in deep networks. i-theory has recently been shown to predict domain-specific regions in cortex [29] with the function of achieving invariance to class-specific transformations (e.g. for faces) and the specific form of
eccentricity-dependent cortical magnification [42]. Our assumption of Hebbian-like plasticity for learning
template views is, however, outside the mathematics of i-theory: it links it to biological properties of cortical
synapses.
This argument of this paper has been made, as nearly as possible, from first principles. It begins with a
claim about the computational problem faced by a part of the brain: the need to compute view-tolerant
representations for faces. Yet it seeks to explain properties of single neurons in a specific brain region, AL,
far from the sensory periphery. The argument proceeds by considering which of the various biologicallyplausible learning rules satisfy requirements coming from the theory while also yielding non-trivial predictions for AL neurons in qualitative accord with the available data. It seems significant then that the
argument only works in the case of Oja-like plasticity; it may suggest the hypothesis that such plasticity
may indeed be driving learning in AL.
The class of learning rules yielding invariance includes those that emerge from principles such as sparsity
and the efficient coding hypothesis [6, 7, 39]. However, explaining the mirror symmetric tuning of AL

8

neurons apparently requires the Oja rule. An interesting direction for future work in this area could be to
investigate the role of sparsity in the face processing system. Perhaps a learning algorithm derived from
the efficient coding perspective that also explains AL’s mirror symmetry could be found.
Our model is designed to account only for the feed-forward processing in the ventral stream. Backprojections between visual areas—and of course within each area—are well known to exist in the ventral stream and probably also exist in the face patch network. They are likely to play a major role in
visual recognition after ∼ 80 ms from image onset. Representations computed in the first feeforward
sweep are likely used to provide information about a few basic questions such as the identity or pose
of a face. Additional processing is likely to require iterations and even top-down computations involving
shifts of fixation and generative models. An example for face recognition is recent work [60] which combines a feedforward network like ours—also showing mirror-symmetric tuning of cell populations—with
a probabilistic generative model. Thus our feedforward model, which succeeds in explaining the main
tuning and invariance properties of the macaque face-processing system, may serve as a building block
for future object-recognition models addressing brain areas such as prefrontal cortex, hippocampus and
superior colliculus, integrating feed-forward processing with subsequent computational steps that involve
eye-movements and their planning, together with task dependency and interactions with memory.

Materials
Stimuli
40 face models were rendered with perspective projection. Each face was rendered (using Blender [11])
at each orientation in 5◦ increments from −95◦ to 95◦ . The untextured face models were generated using
Facegen [47]. All faces appeared on a uniform gray background.

View-invariant Same-different Pair Matching Task
For each of the 5 repetitions of the same-different pair matching task, 20 template and 20 test individuals
were randomly selected from the full set of 40 individuals. The template and test sets were chosen
independently and were always disjoint. 50% of the 600 test pairs sampled from each testing interval
depicted the same two individuals. Each testing interval was symmetric about 0◦ (frontal) and testing
intervals were ordered by inclusion. The smallest was [−10◦ , 10◦ ] and the largest was [−95◦ , 95◦ ] (left
and right profile views). The classifier compared the Cosine similarity of the two zero-mean, and unitstandard deviation representations to a threshold. The threshold was integrated over to compute the area
under the ROC curve (AUC). The abscissa of Fig. 3 is the radius of the testing interval from which test
pairs were sampled. The ordinate of Fig. 3 is the mean AUC ± the standard deviation computed over the
5 repetitions of the experiment.
A similarity matrix in Figure 5 was obtained by computing Pearson’s linear correlation coefficient between
each test sample pair. The same matrix was computed 10 times with different training/test splits and the
average was reported. Same procedures were repeated for features from area MLMF, AL and AM to get
corresponding matrices.

9

Acknowledgments
This material is based upon work supported by the Center for Brains, Minds, and Machines (CBMM),
funded by NSF STC award CCF-1231216. This research was also sponsored by grants from the National
Science Foundation (NSF-0640097, NSF-0827427), and AFOSR-THRL (FA8650-05-C-7262). Additional
support was provided by the Eugene McDermott Foundation.

References
[1] E.H. Adelson and J.R. Bergen. Spatiotemporal energy models for the perception of motion. Journal
of the Optical Society of America A, 2(2):284–299, 1985. ISSN 1084-7529. URL http://www.
opticsinfobase.org/abstract.cfm?URI=josaa-2-2-284.
[2] Arash Afraz, Edward S Boyden, and James J DiCarlo. Optogenetic and pharmacological suppression
of spatial clusters of face neurons reveal their causal role in face gender discrimination. Proceedings
of the National Academy of Sciences, 112(21):6730–6735, 2015.
[3] Fabio Anselmi, Joel Z Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio. Unsupervised learning of invariant representations in hierarchical architectures. arXiv preprint
arXiv:1311.4158, 2013.
[4] Fabio Anselmi, Joel Z Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio.
Unsupervised learning of invariant representations. Theoretical Computer Science, 2015. doi: http:
//dx.doi.org/10.1016/j.tcs.2015.06.048.
[5] Fabio Anselmi, Lorenzo Rosasco, and Tomaso Poggio. On invariance and selectivity in representation
learning. arXiv preprint arXiv:1503.05938, 2015.
[6] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183,
1954.
[7] Horace B Barlow. Possible principles underlying the transformations of sensory messages. Sensory
Communication, pages 217–234, 1961.
[8] E. Bart and S. Ullman. Class-based feature matching across unrestricted transformations. Pattern
Analysis and Machine Intelligence, IEEE Transactions on, 30(9):1618–1631, 2008. ISSN 0162-8828.
URL http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4378342.
[9] E Bart, E Byvatov, and S Ullman. View-invariant recognition using corresponding object fragments.
In European Conference on Computer Vision (ECCV), volume 3024, pages 152–165, Prague, Czech
Republic, 2004. Springer. URL http://www.springerlink.com/index/GGBDRQ3WQFGB9LDN.pdf.
[10] Pietro Berkes, Richard E. Turner, and Maneesh Sahani. A structured model of video reproduces
primary visual cortical organisation. PLoS Computational Biology, 5(9):e1000495, 10 2009. doi:
10.1371/journal.pcbi.1000495.
[11] Blender.org. Blender 2.6, 2013.
[12] DD Cox, P Meier, N Oertelt, and James J. DiCarlo. ’Breaking’ position-invariant object recognition.
Nature Neuroscience, 8(9):1145–1147, 2005. URL http://www.nature.com/neuro/journal/v8/
n9/abs/nn1519.html.
[13] James J. DiCarlo, D Zoccolan, and Nicole C. Rust. How does the brain solve visual object recognition? Neuron, 73(3):415–434, 2012. URL http://www.sciencedirect.com/science/article/
pii/S089662731200092X.

10

[14] Amirhossein Farzmahdi, Karim Rajaei, Masoud Ghodrati, Reza Ebrahimpour, and Seyed-Mahdi
Khaligh-Razavi. A specialized face-processing model inspired by the organization of monkey face
patches explains several face-specific phenomena observed in humans. Scientific Reports, 6, 2016.
[15] P. Földiák. Learning invariance from transformation sequences. Neural Computation, 3(2):194–200,
1991. URL http://www.mitpressjournals.org/doi/abs/10.1162/neco.1991.3.2.194.
[16] Winrich A. Freiwald and D.Y. Tsao. Functional Compartmentalization and Viewpoint Generalization
Within the Macaque Face-Processing System. Science, 330(6005):845, 2010. ISSN 0036-8075.
URL http://www.sciencemag.org/cgi/content/abstract/330/6005/845.
[17] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202, April 1980.
ISSN 0340-1200. doi: 10.1007/BF00344251. URL http://www.springerlink.com/content/
r6g5w3tt54528137.
[18] M. Golubitsky and I. Stewart. The symmetry perspective; from equilibrium to chaos in phase space
and physical space. 2002.
[19] Singer Hardt, Recht. Train faster, generalize better: Stability of stochastic gradient descent.
arXiv:1509.01240, 2016.
[20] M. H. Hassoun. Fundamentals of artificial neural networks. MIT Press, 1995.
[21] D. O. Hebb.

The organization of behavior: A neuropsychological theory.

Wiley, 1949.

URL

http://books.google.com/books?hl=en&amp;lr=&amp;id=gUtwMochAI8C&amp;oi=fnd&amp;
pg=PP1&amp;dq=Hebb&amp;ots=w1kQ2jqppz&amp;sig=QmaxGp399apRC1HQccm_nu9WnU8.
[22] Geoffrey E Hinton and Suzanna Becker. An unsupervised learning procedure that discovers surfaces in random-dot stereograms. In Proceedings of the International Joint Conference on Neural
Networks, Washington DC, pages 218–222, 1990.
[23] Chou P. Hung, Gabriel Kreiman, Tomaso Poggio, and James J. DiCarlo. Fast Readout of Object
Identity from Macaque Inferior Temporal Cortex. Science, 310(5749):863–866, November 2005. doi:
10.1126/science.1117593.
[24] Aapo Hyvärinen. Fast and robust fixed-point algorithms for independent component analysis. Neural
Networks, IEEE Transactions on, 10(3):626–634, 1999.
[25] Aapo Hyvärinen and Erkki Oja. Independent component analysis by general nonlinear hebbian-like
learning rules. Signal Processing, 64(3):301–313, 1998.
[26] Leyla Isik, Joel Z. Leibo, and Tomaso Poggio. Learning and disrupting invariance in visual
recognition with a temporal association rule. Frontiers in Computational Neuroscience, 6(37),
2012. doi: 10.3389/fncom.2012.00037. URL http://www.frontiersin.org/Computational_
Neuroscience/10.3389/fncom.2012.00037/abstract.
[27] Leyla Isik, Ethan M Meyers, Joel Z Leibo, and Tomaso Poggio. The dynamics of invariant object
recognition in the human visual system. Journal of Neurophysiology, 111(1):91–102, 2014.
[28] S.P. Ku, A.S. Tolias, N.K. Logothetis, and J. Goense. fMRI of the Face-Processing Network in the
Ventral Temporal Lobe of Awake and Anesthetized Macaques. Neuron, 70(2):352–362, 2011. URL
http://linkinghub.elsevier.com/retrieve/pii/S0896627311002054.
[29] Joel Z. Leibo, Qianli Liao, Fabio Anselmi, and Tomaso Poggio. The invariance hypothesis implies
domain-specific regions in visual cortex. PLoS Computational Biology, 11(10):e1004390, 10 2015.
doi: 10.1371/journal.pcbi.1004390.
[30] Nuo Li and James J DiCarlo. Unsupervised natural experience rapidly alters invariant object representation in visual cortex. Science, 321(5895):1502–7, September 2008. ISSN 1095-9203.
doi: 10.1126/science.1160028. URL http://www.sciencemag.org/cgi/content/abstract/321/
5895/1502.
11

[31] Nuo Li and James J DiCarlo. Unsupervised Natural Visual Experience Rapidly Reshapes SizeInvariant Object Representation in Inferior Temporal Cortex. Neuron, 67(6):1062–1075, 2010. URL
http://www.cell.com/neuron/fulltext/S0896-6273(10)00639-2.
[32] NK Logothetis and DL Sheinberg. Visual object recognition. Annual Review of Neuroscience, 19(1):
577–621, 1996. URL http://arjournals.annualreviews.org/doi/abs/10.1146/annurev.ne.
19.030196.003045.
[33] NK Logothetis, J Pauls, and T Poggio. Shape representation in the inferior temporal cortex of monkeys. Current Biology, 5(5):552–563, 1995. URL http://linkinghub.elsevier.com/retrieve/
pii/S0960982295001084.
[34] Ethan M Meyers, Mia Borzello, Winrich A Freiwald, and Doris Tsao. Intelligent information loss: The
coding of facial identity, head pose, and non-face information in the macaque face patch system. The
Journal of Neuroscience, 35(18):7069–7081, 2015.
[35] Y. Miyashita. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature, 335(6193):817–820, 1988. URL http://hebb.mit.edu/courses/9.641/readings/
Miyashita88.pdf.
[36] S. Moeller, Winrich A. Freiwald, and D.Y. Tsao. Patches with links: a unified system for processing faces in the macaque temporal lobe. Science, 320(5881):1355, 2008. URL http://www.
sciencemag.org/content/320/5881/1355.short.
[37] E. Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology,
15(3):267–273, 1982. URL http://www.springerlink.com/index/u9u6120r003825u1.pdf.
[38] E. Oja. Principal components, minor components, and linear neural networks. Neural Networks, 5(6):927–935, 1992.
URL http://www.sciencedirect.com/science/article/pii/
S0893608005800899.
[39] Bruno A Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. URL http://redwood.
psych.cornell.edu/papers/olshausen_field_nature_1996.pdf.
[40] T Poggio and S Edelman. A network that learns to recognize three-dimensional objects. Nature, 343(6255):263–266, 1990. URL http://cbcl.mit.edu/people/poggio-new/journals/
poggio-edelman-nature-1990.pdf.
[41] T. Poggio, T. Vetter, and H. Bulthoff. 3D Object Recognition: Symmetry and Virtual Views, 1992. URL
http://www.stormingmedia.us/83/8379/A837952.pdf.
[42] Tomaso Poggio, Jim Mutch, and Leyla Isik. Computational role of eccentricity dependent cortical
magnification. CBMM Memo No. 017. arXiv preprint arXiv:1406.1770, 2014.
[43] M Riesenhuber and T Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11):1019–1025, November 1999. ISSN 1097-6256. doi: 10.1038/14819.
[44] ET Rolls. Invariant visual object and face recognition: neural and computational bases, and a model,
VisNet. Frontiers in Computational Neuroscience, 6, 2012.
[45] T.D. Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural networks, 2(6):459–473, 1989. URL http://www.sciencedirect.com/science/article/pii/
0893608089900440.
[46] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio. Robust Object Recognition with CortexLike Mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3):411–426,
2007. URL http://portal.acm.org/citation.cfm?id=1263421&dl=.
[47] Singular Inversions. FaceGen Modeller 3, 2003.

12

[48] Michael P. Stryker. Temporal associations. Nature, (6349):108–109, 1991. URL http://www.
nature.com/nature/journal/v354/n6349/abs/354108d0.html.
[49] C. Tan and T. Poggio. Neural tuning size in a model of primate visual processing accounts for three
key markers of holistic face processing. Public Library of Science | PLoS ONE, 1(3): e0150980,
2016.
[50] S Thorpe, D Fize, and C Marlot. Speed of processing in the human visual system. Nature, 381
(6582):520–522, 1996.
[51] Doris Y Tsao, Sebastian Moeller, and Winrich A Freiwald. Comparing face patch systems in
macaques and humans. Proceedings of the National Academy of Sciences, 105(49):19514–19519,
2008.
[52] D.Y. Tsao, Winrich A. Freiwald, T.A. Knutsen, J.B. Mandeville, and R.B.H. Tootell. Faces and objects
in macaque cerebral cortex. Nature Neuroscience, 6(9):989–995, 2003. URL http://www.nature.
com/neuro/journal/v6/n9/abs/nn1111.html.
[53] D.Y. Tsao, Winrich A. Freiwald, R.B.H. Tootell, and M.S. Livingstone. A cortical region consisting
entirely of face-selective cells. Science, 311(5761):670, 2006. URL http://www.sciencemag.org/
content/311/5761/670.short.
[54] D.Y. Tsao, S. Moeller, and Winrich A. Freiwald. Comparing face patch systems in macaques and
humans. Proceedings of the National Academy of Sciences, 105(49):19514, 2008. URL http:
//www.pnas.org/content/105/49/19514.short.
[55] Gina G. Turrigiano and Sacha B. Nelson. Homeostatic plasticity in the developing nervous system.
Nature Reviews Neuroscience, 5(2):97–107, 2004. URL http://www.nature.com/nrn/journal/
v5/n2/abs/nrn1327.html.
[56] G Wallis and H H Bülthoff. Effects of temporal association on recognition memory. Proceedings of
the National Academy of Sciences of the United States of America, 98(8):4800–4, April 2001. ISSN
0027-8424. doi: 10.1073/pnas.071028598. URL http://www.pnas.org/cgi/content/abstract/
98/8/4800.
[57] G. Wallis, B.T. Backus, M. Langer, G. Huebner, and H. Bülthoff. Learning illumination-and orientationinvariant representations of objects through temporal association. Journal of vision, 9(7), 2009.
[58] L. Wiskott and T.J. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural
computation, 14(4):715–770, 2002. URL http://www.mitpressjournals.org/doi/abs/10.1162/
089976602317318938.
[59] D.L.K. Yamins and J.D. Dicarlo. Using goal-driven deep learning models to understand sensory
cortex. Nature Neuroscience, 19,3:356–365, 2016.
[60] I. Yildirim, T. Kulkarni, and J. B. Freiwald, W.and Tenenbaum. Efficient and robust analysis-bysynthesis in vision: A computational framework, behavioral tests, and modeling neuronal representations. Annual Conference of the Cognitive Science Society, 2015.
[61] Andrew W Young, Deborah Hellawell, and Dennis C Hay. Configurational information in face perception. Perception, 16(6):747–759, 1987.

13

Appendix
The key results in this appendix can be informally stated as follows:

• We prove than a number of learning rules, supervised and unsupervised, are equivariant with respect to the symmetries of the training data. We use this result in the case of training data consisting
of images of faces for all view angles obtaining equivariance of the solutions of the learning rules
with respect to the reflection group and the group of rotations. The implications that we use in the
paper are
– the solutions of all learning rules can be used as templates in the computation of an invariant
signature. The algorithm consists of performing dot products of the input image with each
template, transforming nonlinearly (for instance using a rectifier nonlinearity or a square) the
result and then pooling over all templates, i.e., the solutions of the learning rule. The result is
approximately invariant to rotation in depth.
– in the case of the Oja rule we prove that the solutions are even or odd functions of the view
angle; a square nonlinearity provides even functions, which are mirror-symmetric. We were
not able to prove such a property for any of the other learning rules.

• in the case of the ICA rule we show empirical evidence that the solutions are neither odd nor even.
This suggests that most learning rules do not lead to even or odd solutions.

The appendix is divided into four sections:

1. In section A we show how recent theorems on invariance under group transformations could be
extended to nongroups and under which conditions. We show how an approximately invariant signature can be computed in this setting. In particular we analyze the case of rotation in depth and
mirror symmetry transformations of bilateral symmetric objects such as faces.
2. In section B we describe how the group symmetry properties of the set of images to which neurons
are exposed (the “unsupervised” training set) determine the symmetries of the learned weights.
In particular we show how the weight symmetries gives a simple way of computing an invariant
signature.
3. In section C we prove that the solutions of the Oja equation, given that the input vectors that are
reflections of each other (like a face’s view at θ degrees and its view at −θ degrees), must be odd
or even.
4. In section D we provide empirical evidence that there are solutions of ICA algorithms—on the same
data as above—that do not show any symmetry.

In the following we indicate with x ∈ Rd an image, with w ∈ Rd a filter or neural weight and with G a
locally compact group.

A

Approximate Invariance for non-group transformations

In this section we analyze the problem of getting an approximately invariant signature for image transformations that do not have a group structure. In fact, clearly, not all image transformations have a group
structure. However assuming that the object transformation defines a smooth manifold we have (by the
14

theory of Lie manifolds) that locally a Lie group is defined by the generators on the tangent space. We
illustrate this in a simple example. Let x ∈ Rd . Let s : Rd × RQ → Rd a C ∞ transformation depending
on Θ = (θ1 , · · · , θQ ) parameters. For any fixed x ∈ Rd the set M = (s(x, Θ), Θ ∈ RQ ) describe a
differentiable manifold. If we expand the transformation around e.g. ~0 we have:

s(x, Θ) = s(x, ~0) +

Q
X
∂s(x, Θ)
i=1

∂θi

2

θi + o(kΘk ) = x +

Q
X

θi Lθi (x) + o(kΘk2 )

(3)

i=1

where Lθi are the infinitesimal generators of the transformation in the ith direction.
Therefore locally (when the term o(kΘk2 ) can be neglected) the associated group transformation can be
expressed by exponentiation as:

g(Θ) = exp(θ1 Lθ1 + θ2 Lθ2 + · · · + θQ LθQ ).
Note that the above expansion is valid only locally. In other words instead of a global group structure
of the transformation we will have a collection of local transformations that obey a group structure. The
results derived in section B will then say that the local learned weights will be orbits w.r.t. the local group
approximating the non-group global transformation.

A.1

Invariance under rotations in depth

The 3D “views” of an object undergoing a 3D rotation are group transformations but the 2D projections
of an object undergoing a 3D rotation are not group transformations. However for any fixed angle θ0
and for small rotations the projected images approximately follow a group structure. This can be easily
seen making the substitution in eq. (3) s(x, Θ) = P (rθ x) where P is the 2D projection. Let η : R → R
be a nonlinear function, e.g., squaring or rectification. For small values of θ we have therefore that the
signature:

Z

θ0

µw (x) =

dθ η(hP x, P rθ wi)
−θ0

or its discrete version

µw (x) =

X

η(hP x, P rθi wi) =

i

X

η(hP x, g(θi )P wi)

i

is invariant under 3D rotation of x of an angle θ̄ up to a factor proportional to O( θ̄ ). Alternatively if the
following property holds:
hP x, P rθ wi = 0 θ > θ̄
(4)
the invariance will be exact (see [3, 29]); this is the case e.g. when both w and x are faces.
The locality of the group structure (eq. (4)) means that we have invariance of the signature only within
each local neighborhood but not over all viewpoints. A reasonable scenario could be that each local neighborhood may consist of, say, ±30 degrees (depending on the universe of distractors). Almost complete
view invariance can be obtained from a single view at +30 degrees. In fact the view, together with the
associated virtual view at −30 degrees because of mirror symmetry, provides invariance over −60, +60
degrees [41].

A.2

Rotation in depth and mirror symmetry.

As explained on the previous paragraph, projected rotations in depth are not group transformations. However in the case of a bilateral symmetric objects, as we will see below, projected rotations in depth are
a collection of orbits of the mirror symmetry group. Section B will clarify why this property is important
15

proving that it forces the set of solutions of a variety of learning rules to be a collection of orbits w.r.t. the
mirror symmetry group.
Consider e.g. a face, x, which is a bilateral symmetric object and its orbit in 3D w.r.t. the rotation group:

Ox = (r0 x, · · · , r2π x).
where r is a rotation matrix in 3D, e.g. w.r.t. the z axis.
Projecting onto 2D we have

P (Ox ) = (P (r0 x), · · · , P (r2π x)).
Note now that, due to the bilateral symmetry, the above set can be written as:

P (Ox ) = (x0 , · · · , x N , Rx1 , · · · , Rx N ).
2

2

where xn = P rθn x, n = 1, · · · , N/2 and R is the reflection operator. The set consists of a collection of
orbits w.r.t. the group G = {e, R}. This is due to the relation

xn = P (rθn x) = Rx N +n = R(P r−θ x).
2

i.e. a face rotated by an angle θ and then projected is equal to the reflection of the same face rotated by
an angle −θ and projected.
The reasoning generalizes to multiple faces. In summary in the specific case of bilateral symmetric objects
rotating in depth, a projection onto a plane parallel to the rotation axis creates images which are transformations w.r.t. the group of reflection, thus falling in the group case described in the above paragraphs.

B

Unsupervised and supervised learning and data symmetries

In the following we show how symmetry properties on the neuronal inputs affect the learned weights.
We model different unsupervised (Hebbian, Oja, Foldiak, ICA) or supervised learning (SGD) rules as
dynamical systems coming from the requirement of minimization of some target function. We see how
these dynamical systems are equivariant (in the sense specified below) and how equivariance determines
the symmetry properties of their solutions.
This gives a simple way to generate an invariant signature by averaging over all solutions.

B.1

Equivariant dynamical systems and their solutions.

We make the general assumption that the dynamical system can be described in terms of trying to minimize a non-linear functional of the form:

arg min L(w, x),

L(w, x) = h(w, x),

x, w ∈ Rd

(5)

w∈X

The associated dynamical system reads as:

ẇ = f (w) = ḣ(w, x).

(6)

A general result holds for equivariant dynamical systems. A dynamical system is called equivariant w.r.t.
a group G if f in eq. (6) commutes with any transformation g ∈ G i.e.

f (gw) = gf (w), ∀g ∈ G.
In this case we have:

16

(7)

Theorem 1. If an equivariant dynamical system has a solution w, then the whole group orbit of w will also
be a set of solutions (see [18]).
In the following we are going to analyze different cases of updating rules for neuronal weights showing,
under the hypothesis that the training set is a (scrambled) collection of the orbits i.e. we specialize the set
X to be of the form:
X = GT , T ∈ Rd×N , X = {x1 , · · · , xN },
(8)
that the dynamical system is equivariant.
We will see that the following variant of the equivariance holds for many dynamical systems:

f (gw, x) = gf (w, πg (x)), ∀g ∈ G, x ∈ X.

(9)

where πg (x) is permutation of the set X that depends on g . The derivation stands on the simple observation:

hx, gwi = g −1 x, w
and the hypothesis that the training set is a collection of orbits. In fact in this case

gX = πg (X).
In general if the training set X is large enough the dynamical system will be equivalent to the unpermuted
one due to the stability of the stochastic gradient descent method [19]. Since the dynamical systems
associated with the Oja and the ICA rules minimize statistical moments they are clearly independent of
S
training data permutations. The fact that the set of solutions is a collections of orbits, S = i Oi implies
that any average operator over them is invariant. In our case the operator is the signature:

µ(x) =

X

η(hx, Oij i)

ij

where Oij is the element j of the orbit i and η : R → R is a non-linear function.
In the following we prove equivariance of a few learning rules.

1. Unsupervised learning rules[20]:
In the following x ∈ X and α > 0 and with the notation πg (x) we indicate the permutation of the
element x in the training set X due to the transformation g .

• Hebbian learning. Choosing
L(w, x) =

α 2
y
2

(10)

where y = hx, wi is the neuron’s response, we have the associated dynamical system is:

ẇ = f (x, w) = α hx, wi x.

(11)

The system is equivariant. In fact:

f (x, gw) = α hx, gwi x = gα g −1 x, w g −1 x = gα hπg (x), wi πg (x) = gf (πg (x), w).
• Oja learning. Choosing
L(w, x) =

α
hx, wi2
2 kwk2

(12)

we obtain by differentiation:

ẇ = f (w, x) = α

y
w
(x − y
).
kwk2
kwk2

(13)

The obtained dynamical system is that of Oja’s for the choice kwk2 = 1. The system is
equivariant (note that kgwk2 = kwk2 ). In fact:

f (gw, x) = α hx, gwi (x − hx, gwi gw) = α g −1 x, w g(g −1 x − g −1 x, w w)
= α hπg (x), wi g(πg (x) − hπg (x), wi w) = gf (w, πg (x))
17

• ICA. Choosing
L(w, x) = α

hx, wi4 kwk22
+
4
2

(14)

we obtain the dynamical system:

ẇ = α(hx, wi3 x − w)

(15)

which can be shown to extract one ICA component [25]. The system is equivariant. In fact:

f (x, gw) = α(hx, gwi3 x − gw) = gα( g −1 x, w

3 −1

g

x − w) = gf (πg (x), w).

• Foldiak. Choosing:
α 2
ȳ ,
2

L(x, w) =

Z

t

dτ hw, xi (τ )

ȳ =

(16)

t0

the associated dynamical system is:

ẇ(t) = α

Z

t


dτ hw, xi (τ ) x(t) = αȳx(t)

(17)

t0

which is the so called Foldiak updating rule. The system is equivariant. In fact:

f (x, gw) = α

Z

t



dτ hgw, xi (τ ) x = gα

Z

t0

t

dτ


w, g −1 x (τ ) g −1 x

t0

= αg ȳ(w, πg (x))πg (x) = gf (w, πg (x))
2. Supervised learning in deep convolutional networks. The reasoning above can be extended to
supervised problems of the form:

arg min L(X, `, W ), X = (x1 , . . . , xN )

(18)

W

where L(X, `, W ) = Loss(X, `, W ). The term Loss(X, `, W ) is a function defined using the loss of
representing a set of observations X , their labels `, and a the set of the network weights W . The
updating rule for each weight wl is given by the backpropagation algorithm:

ẇl =

∂L
.
∂wl

(19)

If the equation above is equivariant the same results of the previous section will hold, i.e., if there
exists a solution the whole orbit will be a set of solutions. In the following we analyze the case of
deep networks showing that equivariance holds if the output at each layer l, ol is covariant w.r.t. the
transformation, i.e.:
ol (gx) = gol (x), ∀ g ∈ G
(20)
We analyze the case of deep convolutional networks with pooling layers between each convolutional layer. In this case the response at each layer is covariant w.r.t. to the input transformation: the
output at layer l is of the form:

Z

Z
dĝ η(hol−1 (X, Wl−2 ), ĝwl i) =

ol (X, Wl−1 )(g) =
gGl

dĝ η(ol−1 (X, Wl−2 ) ∗ wl )(ĝ)

(21)

gGl

i.e. it is an average of a group convolution where ol−1 is the output of layer l − 1 and Wl−1 is the
collection of weights up to layer l − 1. Using the property that the group convolution commutes with
group shift i.e. [(Tḡ f ) ∗ h](g) = Tḡ [f ∗ h](g) we have:

Z

Z
dĝ η(ḡol−1 (X, Wl−2 ) ∗ wl )(ĝ) =

ol (ḡX, Wl−1 )(g) =
gGl

dĝ η(ol−1 (X, Wl−2 ) ∗ wl )(ḡĝ)
gGl

Z
dĝ η(ol−1 (X, Wl−2 ) ∗ wl )(ĝ) = ol (X, Wl−1 )(ḡg) = ḡol (X, Wl−1 )(g).

=
ḡgGl

18

where we used the property ol−1 (ḡX, W ) = ḡol−1 (X, W ). This can be seen to hold using an
inductive reasoning up to the first layer where:

Z

Z

dĝ η(x ∗ w1 )(ĝ) = ḡo1 (x, W1 )(g).

dĝ η((ḡx) ∗ w1 )(ĝ) =

o2 (ḡx, W1 )(g) =

ḡgG1

gG1

In the following we prove that the dynamical systems (updating rules for the weights) associated to
a deep convolutional network are equivariant. We consider e.g. the square loss function (the same
reasoning can be extended to many commonly used loss functions):

L(φL (X, W ), `) =

X

(1 − y` φ(X, W ))2 .

`

where

φL (X, W ) = φL (· · · , φ3 (φ2 (X, w1 ), w3 ), · · · , wl · · · , wL )
being L the layers number and ` is a set of labels. The associated dynamical system reads as:

X
∂L(φL (X, W ), `)
∂φL (X, W )
∂φL (X, W )
= L̇(φL (X, W ), `)
=2
(1 − y` φL (X, W ))
∂wl
∂wl
∂wl
`

Substituting wl with ḡwl we have, by the covariance property, that the first factor of the r.h.s. of the
P
equation above becomes ` (1 − y` φL (πḡ (X), W )). We are then left to prove the equivariance of
the second factor.
Using the chain rule, we have:

∂φL (· · · , φ3 (φ2 (x, w1 ), w2 ) · · · , wL )
∂wl
= φ̇L [oL (WL−1 , x)] φ̇L−1 [oL−1 (WL−2 , x)] · · · φ̇l (ol−1 (x, Wl−2 ), wl ), · · · , wL )

ẇl =

where oj (Wl−1 , x) = φj (· · · φl (ol+1 (x, Wl−1 ), wl ), · · · , wL ), l < j < L, being the output at layer j .
Notice that, in the case of covariant layer outputs, we have:

φj (· · · φl+1 (ol (X, Wl−1 ), ḡwl ), · · · , wL ) = φj (· · · φl+1 (ḡ −1 ol (X, Wl−1 ), wl ), · · · , wL )
= φj (· · · φl+1 (ol (ḡ −1 X, Wl−1 )), wl ), · · · , wL )
= φj (· · · φl+1 (ol (πḡ (X), Wl−1 )), wl ), · · · , wL )
where we used the covariance property in eq. (20) and the fact that the training set is a collection of
orbits w.r.t. the group G.
Finally we have:

∂L(πg (X), {w1 , · · · , wl , · · · , wL }, `)
∂L(X, {w1 , · · · , ḡwl , · · · , wL }, `)
= ḡ
∂wl
∂wl
where the ḡ comes from the derivative of ḡwl w.r.t. wl .
Summarizing we have the following result
Theorem 2. For i = 1, . . . , L, let φi : Rdi → Rdi+1 depend on a set of weights wi . Consider a deep
convolutional network with output of the form

φL (X, W ) = φL (· · · , φ2 (φ1 (X, w1 ), w2 ), · · · , wl ) · · · , wL ).

(22)

and a differentiable square loss L(φL (X, W ), `), being ` a set of labels.
If X is a collection of orbits and and each φi is covariant, then the associated dynamical systems
for each layer’s weights’ evolution in time

ẇl =

∂L(φL (X, W ), `)
∂wl

are equivariant w.r.t. the group G.

19

4

8

3
2

IC 3

6
4

IC 4

4

4

3

3

response

10

response

IC 2
12

5

response

response

IC 1
6

2

1
1

2

0
-100

0
-100

0

50

100

-50

angle

0

50

100

IC 5

IC 6

3
2

2

0

50

100

0
-100

-50

angle

0

50

100

100

50

100

6

2

4

2

-50

angle

(A)

50

IC 8

3

0
-100

0

8

1

1
-50

-50

angle

4

1

0
-100

0
-100

100

5

response

response

response

4

50

IC 7

4

3

AM

0

angle

5

AL

-50

angle

6

MLMF

1

response

-50

0
-100

2

0

angle

50

100

0
-100

-50

0

angle

(B)

Figure 6: Experiments with ICA: we adopt the same pipeline as shown in the main text but replaced PCA
with ICA [24] (which includes a ZCA-whitening preprocessing step). Similar to the original pipeline, for
each training identity, ICA is performed to get 39 independent component directions. A testing image is
projected to these directions. A square nonlinearity and a pooling are performed on the results. We show
(A) the model population similarity matrices of different stages (similar to Fig 5A) and (B) some single
cell responses in stage AL (similar to Fig 4A). Unlike PCA, the order of the independent components are
arbitrary.

C

Proof that the Oja equation’s solutions are odd or even.

So far we have shown how biologically plausible learning dynamics in conjunction with appropriate training
sets lead to solutions capable of supporting the computation of a view-invariant face signature (Sections
A – B). We showed that several different learning rules satisfied these requirements: Hebb, Oja, Foldiak,
ICA, and supervised backpropagation (Section B.1). Now we use properties specific to the Oja rule to
address the question of why mirror symmetric responses arise in an intermediate step along the brain’s
circuit for computing view-invariant face representations.
We now use the following well-known property of Oja’s learning rule: that it implements an online algorithm
for principal component extraction [38]. More specifically, we use that the Oja dynamics converge to an
eigenfunction of the training set’s covariance C(X).
Recall from section A.2 that in order to guarantee approximate view-invariance for bilaterally symmetric
objects like faces, the training set X must consist of a collection of orbits of faces w.r.t. the reflection
group G = (e, R). We now show that this implies the eigenfunctions of C(X) (equivalently, the principal
components (PCs) of X ) must be odd or even.
Under this hypothesis the covariance matrix C(X) can be written as

C(X) = XX | = T T | + RT T | R|
where T is the set of the orbit representatives (untransformed vectors).
It is immediate to see that the above implies [C(X), R] = 0 (they commute). Thus C(X) and R must
share the same eigenfunctions. Finally, since the eigenfunctions of the reflection operator R are odd or
even, this implies the eigenfunctions of C(X) must also be odd or even.
Finally, we note that in the specific case of a frontal view, even basis functions (w.r.t. the zero view) are
mirror symmetric.

20

D

Empirical ICA solutions do not show any symmetry

Fig. 6 shows results from the analogous experiment to main Fig. 4. but with ICA instead of PCA. Note
that the ICA result is not mirror symmetric.

21

