Non-Parametric Transformation Networks
Dipan K. Pal 1 Marios Savvides 1

arXiv:1801.04520v3 [cs.CV] 14 Feb 2018

Abstract
ConvNets, through their architecture, only enforce invariance to translation. In this paper, we
introduce a new class of deep convolutional architectures called Non-Parametric Transformation Networks (NPTNs) which can learn general
invariances and symmetries directly from data.
NPTNs are a natural generalization of ConvNets
and can be optimized directly using gradient descent. Unlike almost all previous works in deep
architectures, they make no assumption regarding
the structure of the invariances present in the data
and in that aspect are flexible and powerful. We
also model ConvNets and NPTNs under a unified framework called Transformation Networks
(TN), which yields a better understanding of the
connection between the two. We demonstrate the
efficacy of NPTNs on data such as MNIST and
CIFAR10 where they outperform ConvNet baselines with the same number of parameters. We
show it is more effective than ConvNets in modelling symmetries from data, without the explicit
knowledge of the added arbitrary nuisance transformations. Finally, we replace ConvNets with
NPTNs within Capsule Networks and show that
this enables Capsule Nets to perform even better.

1. Introduction
The Fundamental Problem. One of the central problems
of deep learning, and machine learning in general, has been
supervised classification. An instantiation of which, in vision, is object classification. A core challenge towards these
problems is the encoding or learning of invariances and
symmetries that exist in the training data. Any general classification problem would require invariance to the withinclass transformations and symmetries (nuisance transformations) while being selective to between-class transformations. There also has been evidence that the sample complexity of a model is inversely proportional to the amount of
invariance it can invoke towards nuisance transformations
(Anselmi et al., 2013). Indeed, methods which incorporate
some known invariances or promote learning of more powerful invariances for a learning problem perform better in

Figure 1. Operation performed by a single Non-Parametric Transformation Network (NPTN) node (single channel input and single
channel output). NPTNs are a generalization of ConvNets towards
learning general invariances and symmetries. The node has two
main components (a) Convolution and (b) Transformation Pooling. The dot-product between the input patch (x) and a set of |G|
number of filters gw (green) is computed (this results in convolution when implemented with spatially replicated nodes). Here
|G| = 6 (different shades of green indicate transformed templates).
g indicates the transformation applied to the template or filter w
. The resultant six output scalars (red) are then max-pooled over
to produce the final output s (black). The pooling operation here
is not spatially (as in vanilla ConvNets) but rather across the |G|
channels which encode non-parametric transformations. The output s is now invariant to the transformation encoded by the set of
filters G. Each plane indicates a single feature map/filter.

the target task given a certain amount of data. A number
of ways exist to achieve this. One can present transformed
versions of the training data (Burges & Schölkopf, 1997;
Niyogi et al., 1998), minimize auxiliary objectives promoting invariance during training (Schroff et al., 2015; Hadsell
et al., 2006) or pool over transformed versions of the representation itself (Liao et al., 2013; Pal et al., 2016; 2017).
Convolutional Networks and Beyond. Towards this goal,
ideas proposed in (LeCun et al., 1998) with the introduction of convolutional neural networks have proved to be
very useful. Weight sharing implemented as convolutions,
helped to regularize the network and vastly reduce the number of parameters learned. Additionally, it resulted in the
hard encoding of translation invariances (and symmetries)
in the network, making it one of the first applications of
modelling invariance through a network’s architecture itself.
Such a mechanism resulted in greater sample efficiency and
in regularization in the form of a structural or inductive
bias in the network. With this motivation in mind, it is almost natural to ask whether networks which model more
complicated invariances and symmetries perform better?

Non-Parametric Transformation Networks

Investigating architectures which invoke invariances not implicitly through the model’s functional map but explicitly
through an architectural property seems important.
New Dimensions in Network Architecture. Over the
years, deep convolutional networks (ConvNets) have enjoyed a wide array of improvements in architecture. It was
observed early on that a larger number of filters (width) in
ConvNets led to improved performance, though with diminishing returns. Another significant milestone was the development and maturity of residual connections and dense skip
connections (He et al., 2016; Huang et al., 2016). Though
there have been more advances in network architecture,
many of the improvements have been derivatives of these
two ideas (Zagoruyko & Komodakis, 2016; Chen et al.,
2017; Hu et al., 2017). Recently however, Capsule Nets
were introduced (Sabour et al., 2017) which presented another potentially fundamental idea of encoding properties of
an entity or an object in an activity vector rather than a scalar.
With the goal of designing more powerful networks, ideas
for modelling general invariances in the same framework
as ConvNets, open up a new and potentially key dimension
for architecture development.
Primary Contribution. In this work, we explore one such
architecture class, called Transformation Networks (TN).
We introduce a new layer which can form a class of networks
called Non-Parametric Transformation Networks (NPTNs).
These networks have the ability to learn invariances to general transformations present in the data which are (potentially) non-parametric in nature. NPTNs are named so for
their explicit handling of transformation invariances and
symmetries in the data. They can be easily implemented
using standard off-the-shelf deep learning frameworks and
libraries. Further, they can be optimized using vanilla gradient descent methods such as SGD. Unlike other methods
that enforce additional invariances in convolutional architectures, NPTNs do not need to transform the input or the
filters at any stage of the learning/testing process. They enjoy benefits of a standard convolutional architecture such as
speed and memory efficiency while being more powerful in
modelling invariances and being elegant in their operation.
When forced to ignore any learnable transformation invariances in data, they gracefully reduce to vanilla ConvNets in
theory and practice. However, when allowed to do so, they
outperform ConvNets by capture more general invariances.
Some properties of NPTNs. The architecture itself of an
NPTN allows it to be able to learn powerful invariances from
data (a single node is illustrated in Fig. 1). This offer better sample complexity and shortens the generalization gap1 .
Learning invariances from data is different and more pow1
Indeed, empirically we find that three or more layered NPTNs
have a smaller generalization gap, i.e. have higher training losses
than vanilla ConvNets, but lower testing losses.

erful than enforcing known and specific invariances such
as rotation symmetry in networks. Such networks which
enforce predefined symmetries (including vanilla ConvNets)
force the same invariances at all layers which is a strong
prior. More complex invariances are left for the network
to learn using the implicit functional map as opposed to
the explicit architecture. The proposed NPTNs have the
ability to learn different and independent invariances for
different layers and in fact for different channels themselves.
Standard convolution architectures enforce translation invariance through the convolution operation followed by a
aggregation operation (either pooling or a second convolution layer). In this aspect, each node has a predefined
invariance (translation) and only needs to learn the filter
instantiation. However, an NPTN node needs to learn two
independent entities. First, the instantiation of the filter and
second, the transformation that the particular node is invariant towards. Each node learns these entities independently
of each other which allows for a more flexible invariance
model as opposed to architectures which replicate invariances across the network.

2. Prior Art
There has been considerable interest in the past, in developing methods which incorporate prior knowledge of invariances or symmetries in data. Although the applications
previously tackled were specific and relatively narrow, development of such methods offers a better understanding of
the importance of modelling symmetries in data. In these
cases, the architectures explicitly enforce such structure
through different approaches. Though in this work we focus
on deep architectures, it is interesting to note a number of
works on modifications of Markov Random Fields and Restricted Boltzman Machines to achieve rotational invariance
(Schmidt & Roth, 2012; Kivinen & Williams, 2011; Sohn
& Lee, 2012).
Incorporating known invariances using deep networks.
Convolutional architectures have also seen efforts to produce
rotation invariant representations. (Fasel & Gatica-Perez,
2006) and (Dieleman et al., 2015) rotate the input itself before feeding it into stacks of CNNs and generating rotation
invariant representations through gradual pooling or parameter sharing across orientations. (Teney & Hebert, 2016;
Wu et al., 2015; Li et al., 2017) rotate the convolution filters
instead of the input. This is followed by a pooling operation
to invoke invariance. Having the filters transform instead
alleviates the need to transform the inputs, which is more
expensive. Nonetheless, this requirement also remains considerably expensive during training. A similar approach was
explored for scale by (Xu et al., 2014). (Clark & Storkey,
2015) propose a network to play Go incorporating reflective
symmetry within the filters through weight tying. An inter-

Non-Parametric Transformation Networks

esting direction of research was explored by (Sifre & Mallat,
2013) where the filters were fixed and non-trainable. The
properties of the filter structure allowed them to be rotation,
scale and translation invariant. All these methods however
make an critical assumptions about the invariances present
in each of the applications. Although this is useful, it is very
restrictive when dealing with general applications or cases
where the transformations present are unknown and complex in nature such as general vision. (Cohen & Welling,
2016a) presented a method to incorporate invariances towards parametric groups. Recently, (Henriques & Vedaldi,
2017) proposed an interesting warped convolution based
layer which can implement more general (but parameterized) equivariance. The transformations are known apriori
and the sample grids are generated offline. Also, (Cohen &
Welling, 2016b) introduced steerable filters into the convolutional framework. They require generating the filters offline
and apriori, and hence have limited capability in learning
arbitrary and adaptive transformations. NPTNs need no
such apriori knowledge, can learn arbitrary non-parametric
transformations and finally are simpler and more elegant in
theory and in implementation.

variance through complex weight sharing. Optimization
was carried out through Topographic ICA and only carried
out layer wise for deep networks. A separate approach
towards modelling invariances was also developed where
a normalizing transformation is applied to every input independently. This approach was applied to transforming
auto encoders (Hinton et al., 2011) and Spatial Transformer
Networks (Jaderberg et al., 2015).

Learning unknown invariances from data. To address
the previous shortcoming, a few studies with deep networks
have been conducted towards learning more general transformations. Indeed, in most real world problems, nuisance
transformations present in data are unknown or too complicated to be parameterized by some function. (Anselmi
et al., 2013) proposed a theory of group invariances called
I-theory and explored its connection to general classification
problems and deep networks. Based off the core idea of
measuring moments of a group invariant distribution, multiple works had demonstrated efficacy of the ideas in more
challenging real-world problems such as face recognition,
though not in a neural network setting (Liao et al., 2013; Pal
et al., 2016; 2017).

Each TN node (single input channel and single output channel) internally consists of two operations 1) (convolution)
the convolution operation with a bank of filters and 2) (transformation pooling) a max pooling operation across the set
of the resultant convolution feature maps from the single
input channel. The pooling here is across only transformed
versions of the single input channel. This is in contrast to the
single convolution operation of the vanilla convolution node.
Note that the pooling operation is not the same superficially
to the standard pooling in a ConvNet, where the pooling is
spatial in nature. Here the pooling is not spatial but rather
across channels originating from the same input channel.
Fig. 1 illustrates the operation of single TN node with a
single input channel for a single patch. The single channel illustrated in the figure takes in a single input feature
map and convolves it with a bank of |G| filters. Here |G|
is the cardinality (or size) of the set of transformations that
the TN node is invariant towards, with G being the actual
set itself. Next, the transformation max pooling operation
simply max pools across the |G| feature values to obtain a
single TN activation value. When this node is replicated
spatially, standard convolution layers can be utilized. A TN
layer with M input channels and N output channels will
simply have M N of these TN nodes with each of the N
output channel having a TN node connected to each of the
M input channels. Fig. 2 illustrates a multi-channel TN.

Learning unknown invariances from data using deep
networks. (Gens & Domingos, 2014) introduced Symnets
which was one of the first to model general invariances in
deep networks with back propagation. They utilize kernel
based interpolation to tie weights model general symmetries.
Consistent with the connection between sample complexity
and invariance modelling, they find that Symnets perform
better with fewer samples compared to vanilla ConvNets.
Nonetheless, the approach is complicated and difficult to
scale. To the best of our knowledge, this was the only approach which learned invariances from data in a neural
network setting, albeit with a more complicated approach.
(Anselmi et al., 2017) provide sufficient conditions to enforce the learned representation to have symmetries learned
from data. (Kavukcuoglu et al., 2009) modelled local invariances using pooling over sparse coefficients of a dictionary
of basis functions. (Ngiam et al., 2010) achieved local in-

3. Transformation Networks
The Transformation Network (TN) is a feed forward network with its architecture designed to enforce invariance to
some class of transformations. At the core of the framework
is the TN node, whose structure itself enforces desirable
invariance properties. A TN network consists of multiple
such nodes stacked in layers. A TN node is analogous to
a single channel output with a single channel input in a
ConvNet. More specifically, if a vanilla ConvNet layer has
M input channels and N output channels, the TN version of
the layer would have M N TN nodes arranged in the same
fashion.

Formally, a TN node denoted by Υ acting on a 2D image
patch vectorized as x ∈ Rd can be defined as follows.
Υ(x) = max(hx, gwi)
g∈G

(1)

Here, G is formally defined as a unitary group, i.e. a finite

Non-Parametric Transformation Networks

(a) Convolution layer

(b) NPTN layer

Figure 2. Comparison between (a) a standard Convolution layer
and (b) a TN layer with |G| = 3. Each layer depicted has 2
input (shades of grey) and 2 output channels (shades of blue). The
convolution layer has therefore, 2 × 2 = 4 filters, whereas the TN
layer has 2 × 2 × 3 = 12 filters. NPTN (later introduced) has the
same structure as the TN layer. The different shades of filters in
the TN layer denote transformed versions of the same filter which
are max pooled over (support denoted by inverted curly bracket).
The + operation denotes channel addition. In our experiments, we
adjust the input/output channels of the NPTN layer to have the
same number of filters as the ConvNet baselines.

set obeying group axioms with each element being unitary.
w ∈ Rd is the weight or template instantiation of the set
of transformed weights {gw | g ∈ G}2 . Therefore, the
convolution kernel weights of a TN node are simply the
transformed versions of w as transformed by the unitary
group G. In a TN node, both the entities (w, G) are learned,
i.e. a TN node is tasked with learning both the template
instantiation w, and the set of transformations G to which
the node is to be invariant towards. This is in sharp contrast
with the vanilla convolutional node in which only the template instantiation w is learned, where G is hard coded to
be the translation group. Theoretically, the TN node has to
transform weight template w according to G to generate the
rest of the filters to be pooled over during the trasformation
pooling stage. In practice however, these are simply stored
as a set of templates or filters which only implicitly encodes
G. Gradient descent updates each filter differently over time
facilitating this encoding. Thus, during any forward pass, no
generation of transformed filters is necessary which significantly reduces computational complexity comapred to some
previous works (Teney & Hebert, 2016; Wu et al., 2015).
Invariances in a TN node. Invariance in the TN node arises
in theory due to the symmetry of the unitary group structure of the filters. The max operation simply measures the
infinite moment of an invariant distribution which leads to
invariance. We demonstrate this in the form of the following
result.
Theorem 3.1. (Invariance Property) Given vectors x, w ∈
Rd , a unitary group G and Υ(x) = maxg∈G (hx, gwi), for
any fixed g 0 ∈ G, the following is true
Υ(x) = Υ(g 0 x)
2
We denote the action of a group element g on w with the
notation gw to reduce clutter.

Proof. Consider the distribution of elements of the set
Sg0 = {hg 0 x, gwi} over all g ∈ G and for any particular g 0 ∈ G. This 1-D distribution characterizes the vector
g 0 x through the projections onto gw. Due to unitarity of G,
and that g 0 ∈ G, we have hg 0 x, gwi = hx, g 0−1 gwi. Now,
since G is a group, we have for any g 0 ∈ G, g 0−1 g ∈ G
due to the closure property. The set of elements in Sg0 contains all elements of G and hence must also contain g 0−1 g.
This implies that the action of g 0−1 on the group G results
in just a reordering of the group, leaving the distribution
unchanged. Thus, the set Sg0 is unchanged. More specifically, Sg0 = {hg 0 x, gwi} = {hx, gwi} = Se , where e is the
identity element of G. Thus, the two sets invoke the exact
same distribution, which results in their moments being the
same. This includes the infinite moment, which implies
Υ(g 0 x) = maxg∈G Sg0 = maxg∈G Se = Υ(x).
Theorem 3 shows that for any input x, the node output is
invariant to the transformation group G. This is interesting,
since one does not need to observe any transformed version of x during training which reduces sample complexity.
Invariance is invoked for any arbitrary input x during test
thereby demonstrating good generalization properties. It is
worthwhile to note that vanilla Convnets with their pooling
layers pool over perfect unitary groups since translation is a
unitary operation and pooling structure is enforced over a
finite group.
General Non-group Structure in a TN node. In practice,
a TN node does not explicitly enforce any group structure
to the set of templates or convolution filters. Although such
a structure is required for theoretically generating invariance, sufficient approximate invariance has been observed
in empirical studies on real-world data utilizing non-group
structures as found in (Pal et al., 2016; Liao et al., 2013).
In our experiments, we observe that the TN architecture
networks are able to perform better by learning invariance
towards both group structured transformations such as translation and rotation, and also towards general non-parametric
transformations.
ConvNets are a kind of Parametric Transformation Networks (PTNs). The vanilla convolution layer simply performs the convolution operation of the filters onto the image.
This by itself does not produce any invariance to any transformation. However, when followed by a spatial pooling
operation (or even a second convolution layer), the resultant
feature is explicitly invariant to only translation. This convolution and pooling operation can be modelled by the TN
node when the group G is defined to be a finite translation
group. Following the TN node operation, the translation
group G acts on the filter template w and transformed versions of it are computed on-the-fly for a ConvNet, which (for
a single patch) are then dot-producted with the input patch
to generates features. These features are then max-pooled

Non-Parametric Transformation Networks

0.5

0.8
0.78

0.45

0.76

Conv (48)
NPTN (9 5)
NPTN (12 4)
NPTN ch (16 3)
NPTN (24 2)
NPTN (48 1)

0.74

0.4

0.35

Loss

0.72

Loss

over by the second operation of the TN node resulting in
translation invariance. It is straight-forward to observe that
a generalization of the vanilla convolution node would be
where the group G is parameterized to model more complicated transformations. Networks with this more general
type of parametric TN node are called Parametric Transformation Networks (PTN). Thereby, a vanilla convolution
network (ConvNet) with its pooling operation is a kind of
PTN.

0.7
0.68

0.25

0.66

0.2
0.64

0.15

0.62

4. Non-Parametric Transformation Networks
We now introduce our main practical contribution, which
are Transformation Networks that can learn general nonparametric symmetries from data. A Non-Parametric Transformation Network (NPTN) is a network of TN nodes that
lack any parametric model for the transformation which
describes the set of templates/filters in the nodes. They are
able to explicitly learn arbitrary invariances and symmetries
from data itself better than a vanilla CNN. Compared to
other approaches to model general invariances, their architecture is a natural generalization of CNNs and is elegantly
simple. Setting |G| = 1 reduces a NPTN to a standard CNN
in theory and in practice.
NPTN layer structure. NPTNs can be implemented using
standard deep learning libraries and convolutional routines.
They can be optimized using standard gradient descent methods such as SGD. They can replace any convolution layer
in any architecture making them versatile. We describe one
way to implement a NPTN layer with M input channels, N
output channels and that models a set of transformations
with cardinality |G| (as illustrated in Fig. 2). Note that
each of the M × N channels (analogous to the input ×
output number of channels in ConvNets) models |G| filters
independently. For every input channel, |G| filters must
be learned which would encode the pre-transformed filter
set. This continuously maintained set of filters bypasses
the need to transform the filters on-the-fly during a forward
pass or whenever the weights change due to an update. The
gradients itself encode the invariance updates. Once the
input is convolved with these M × |G| filters, the M sets
each with |G| feature maps each are max pooled across.
More specifically, each |G| number of feature maps from
a single input channel results in one intermediate feature
map after max pooling (across the |G| channels). After this
operation there are M intermediate feature maps which are
transformation invariant. Now, the sum (alternatively the
mean) of these M feature maps results in one output feature
map or channel. Each of the N output channels performs
the same operation with independent of the other channels.
The total number of filters learned for a NPTN layer with
M input channels, N output channels and that models |G|
number of transformations is M N |G|. Importantly, every

0.3

0.6
150

200

250

300

Epoch

0.1
150

Conv (48)
Conv (72)
Conv (89)
Conv (104)
Conv (118)
Conv (130)
Conv (141)
NPTN (48 1)
NPTN (48 2)
NPTN (48 3)
NPTN (48 4)
NPTN (48 5)
NPTN (48 6)
NPTN (48 7)

200

250

300

Epoch

(a) Training loss for a 2 layer net-(b) Training loss for a 3 layer network
work
Figure 3. Training loss on CIFAR10. (a) 2 layered network. Each
network has the same number of parameters in the second layer
and lower number of parameters for the first layer for the NPTN
variants. (b) 3 layered network. Networks with the same color have
the same number of parameters. All non-trivial NPTN variants
(|G| > 1) have higher training error but lower test error, thereby
shortening the generalization gap.

connection from input to output of a layer independently
models invariance using separate |G| number of filters.
Non-unitary structure in a NPTN node. NPTNs do not
explicitly enforce a unitary structure. Preserving a unitary
structure would restrict the complexity of transformations
that can be modelled. Nonetheless, such a restriction might
also bring about regularization benefits which is a question
that is left for future exploration. NPTNs in this aspect are
an approximation whereas ConvNets exactly maintain the
unitary and group structure explicitly through hard coded
operations. Despite this fact, we find in our experiments
that NPTNs perform better, leading to the hypothesis that
modelling more complex transformations is more important
than exactly preserving the unitary group structure.
Relation to Maxout Networks. NPTNs deviate significantly in motivation and architecture from Maxout Networks (Goodfellow et al., 2013). Maxout Networks were
introduced as a more general activation function which also
applied max pooling across channels. However, each of
the set of channels pooled over has a support over all input channels. There is no relation to invariance modelling
of a single input feature map. NPTNs on the other hand
only max pool across channels which take in only a single
channel as input. The filters of the pooled over channels get
applied to a single input. Pooling across these responses
results in an invariant description of the input.

5. Empirical Evaluation
5.1. Implementation.
We implement NPTNs using PyTorch with its standard convolution routines. Our NPTN implementation consisted of

Non-Parametric Transformation Networks

5.2. Benchmarking against ConvNets
In our first set of experiments, we benchmark and characterize the behavior of NPTNs against the standard ConvNets
augmented with Batch Normalization (Ioffe & Szegedy,
2015). The goal of this set of experiments is to observe
whether learning non-parametric transformation invariance
from complex visual data itself helps with object classification. For this experiment, we utilize the CIFAR10 dataset3 .
The networks we experiment with are not designed to compete on this data but rather throw light into the behavior of
NPTNs. We therefore utilize shallow networks, namely a
two and three layered network for these experiments. Each
layer block of the baseline ConvNets consist of the convolution layer, followed by batch normalization and the
non-linearity (PReLU) and finally by a 2 by 2 spatial max
pooling layer. Each corresponding NPTN network replaces
only the convolution layer with the NPTN layer. Thus,
NPTN is allowed to model non-parametric invariance in
addition to the typically enforced translation invariance due
to spatial max pooling.
Two layered NPTN. Our first pilot experiment works with
a two layered network with the baseline ConvNet having
channels [48, 16] with a total of 3 × 48 + 48 × 16 = 912 filters. The NPTN variants in this experiment keep the number
of filters in the second layer constant with 48 channels with
|G| = 1 denoted by (48 1), 24 channels with |G| = 2 denoted by (24 2), and so on up until 9 channels with |G| = 5
3
With standard data augmentation of random cropping after a
4 pixel pad, and random horizontal flipping. Training was for 300
epochs with the learning rate being 0.1 and decreased at epoch
150, and 225 by a factor of 10.

ConvNet
NPTN

0.64

Test loss

four operations performed in sequence, namely convolution,
volumettric max pooling, channel reordering and finally volumetric mean pooling. For M inputs, N outputs and |G|
transformations, the first function is the standard convolution function with M inputs and M N |G| outputs with the
groups option set to M . Hence a total of M N |G| filters are
learned. The second function is the volumetric max pooling
to pool only across channels with a kernel size (|G|, 1, 1)
(where the kernel size is (channels, pool h, pool w)). The
kernel size along height and width can be increased from
1 if spatial max pooling is desired as well. Note that the
output of the convolution function will have N channels in
sequence from the same input channel replicated M times.
However, we want the alternate configuration before the
volumetric mean pooling layer. We instead require one feature map from every input channel to be together in order
(i.e. from each of the M input channels) replicated N times.
This is solved by a channel reordering operation which is
the third operation. The final operation is the volumetric
mean pooling with a kernel of size (M, 1, 1) resulting N
output channels as required.

0.62

0.6
(48, 1)

(24, 2)

(16, 3)

(12, 4)

(9, 5)

(Channel, |G|)
Figure 4. Test losses on CIFAR10 for the two layered network.
Each network listed has the same number of filters. ConvNet
denoted had 48 channels.

Method
ConvNet (48)
NPTN (48, 1)
ConvNet (72)
NPTN (48, 2)
ConvNet (89)
NPTN (48, 3)
ConvNet (104)
NPTN (48, 4)
ConvNet (118)
NPTN (48, 5)
ConvNet (130)
NPTN (48, 6)
ConvNet (141)
NPTN (48, 7)

(k = 3)
0.555
0.548
0.506
0.498
0.485
0.476
0.474
0.479
0.464
0.478
0.453
0.483
0.456
0.477

(k = 5)
0.467
0.472
0.440
0.431
0.440
0.407
0.441
0.418
0.431
0.412
0.438
0.422
0.435
0.421

(k = 7)
0.465
0.456
0.481
0.427
0.467
0.438
0.485
0.448
0.482
0.461
0.478
0.457
0.473
0.455

Table 1. Test loss on CIFAR10 for the three layered network. Each
NPTN is paired with its corresponding ConvNet baseline which
has approximately the same number of parameters.

(9 5). Fig. 3(a) and Fig. 4 show the training and testing
losses. Each network experimented with has the same number of parameters. For the two layered network we find that
the (48 1) performs slightly worse than the ConvNet baseline in training. However, all NPTN variants which learn
a non-trivial set of transformations (|G| > 1) have lower
training loss except (9 5). NPTNs (24 2), (16 3), (12 4)
perform significantly better. Fig. 3(a) shows that all NPTN
variants have much lower test losses with (16 3) performing
the best.
Three layered NPTN. For the second comparison experiment, we explore the behavior of a three layered network
with different kernel sizes. To maintain a fair comparison,
we have an approximately equal number of filters between
the corresponding baseline ConvNet (by slightly increasing
the number of channels) and the NPTN variant while keeping the number of channels constant for the NPTNs at 48
but instead increasing cardinality of the transformation set
(|G|). Thus, the transformation modelling capacity of the

Non-Parametric Transformation Networks

Rotations
ConvNet (36)
NPTN (36, 1)
NPTN (18, 2)
NPTN (12, 3)
NPTN (9, 4)
Translations
ConvNet (36)
NPTN (36, 1)
NPTN (18, 2)
NPTN (12, 3)
NPTN (9, 4)

0◦
0.022
0.021
0.020
0.019
0.018
0 pixels
0.019
0.018
0.019
0.019
0.019

30◦
0.037
0.041
0.034
0.033
0.036
4 pixels
0.030
0.029
0.023
0.022
0.023

60◦
0.066
0.064
0.053
0.055
0.057
8 pixels
0.063
0.061
0.054
0.051
0.051

90◦
0.106
0.108
0.092
0.087
0.095
12 pixels
0.218
0.224
0.190
0.189
0.195

Table 2. Test loss on progressively transformed MNIST with (a)
random rotations and (b) random pixel shifts. NPTNs can learn
invariances to arbitrary transformations from the data itself without
any apriori knowledge.

network is slowly increased. We train/test all networks on
CIFAR10 and report results in Fig. 3(b) and Table 1. We
find that a three layered NPTN has higher training loss as
compared to its ConvNet baseline but demonstrates lower
test loss. This indicates that NPTN have a structural bias
which further shortens the generalization gap. This behavior
in training is in contrast to the two layered network but in
line with the behavior during test time with decreased loss.
We find that NPTNs with roughly the same computational
complexity and parameters generalize better than ConvNets.
Note that however, we do not observe such benefits for a
kernel size of 3, in fact in many cases NPTNs perform worse.
We hypothesize that, this is because a 3 × 3 filter size is
too small an area of an activation map to exhibit meaningful structured spatial transformations that can be learned.
Better generalization is consistently observed for the larger
kernel sizes of 5 and 7 which do contain more structured
transformations due to larger receptive field sizes.
Effect of training with larger |G|. We find that performance peaks at |G| being around 3 and going higher offers
less performance gains. We believe that this is because the
forward pass of the NPTN selects a single transformation
channel (per input channel) to update due to the max operation. Every back-propagation therefore updates only a
single filter out of the |G| filters (per input channel) leading
to each filter being updated on an average by the total num1
ber of iterations times a factor of |G|
. This results in the
filters being less optimized than standard ConvNet filters
resulting in a trade off against modelling invariance. The
effect seems more pronounced for higher |G|. A standard
ConvNet does not face this problem as every filter is updated
for any given backprop iteration (|G| = 1).

5.3. Learning Transformation Invariances
Efficient learning of unknown invariances from data.
We demonstrate the ability of NPTN networks to learn invariances directly from data without any apriori knowledge.
For this experiment, we augment MNIST with a) random
rotations b) random translations, both in training and testing data thereby increasing the complexity of the learning
problem itself. For each sample, a random instantiation
of the transformation was applied. For rotation, the angular range was increased, whereas for translations it was
the pixel shift range. Table 2 presents these results. All
networks in the table are two layered and have the exact
same number of parameters. As expected, NPTNs match
the performance of vanilla ConvNets when there were no
additional transformations added (0◦ and 0 pixels)4 . However, as the transformation intensity (range) is increased,
NPTNs perform significantly better than ConvNets. Trends
consistent with previous experiments were observed with
the highest performance observed with NPTN (|G| = 3).
This highlights the main feature of NPTNs, i.e. their ability
to model arbitrary transformations observed in data without
any apriori information and without changes in architecture
whatsoever. They exhibit better performance in settings
where both rotation invariance and stronger translation invariance is required (even though ConvNets are designed
specifically to handle translations). This ability is something that previous deep architectures did not posses nor
demonstrate.
Efficacy with depth. As an exploratory study, we examine
how the relative depth of the NPTN layer assists in learning
transformation invariance on MNIST. For this, we train
three networks with number of layers being 5, 6 and 8
with 48 channels per layer expect the last layer with 16
channels. We replace layers at different depth with a single
NPTN layer (|G| = 3) starting at depth 2 through to the
last but one layer and observe the impact on performance.
We do this while keeping the number of parameters and the
computation complexity exactly the same for all networks of
a particular depth. To enforce this, the number of channels
in the ConvNet baselines were slightly increased to maintain
the same number of parameters as the NPTN versions. Fig. 5
shows the performance of these networks on CIFAR10. The
term layer ratio signifies the relative depth of the NPTN
layer replacement (layer ratio is the layer of replacement
divided by the total number of layers). We find that, as a
general trend, NPTNs offer most performance gains towards
the lower layers where it can learn more complicated low
level transformations than translations. Performance gains
decrease towards a layer ratio of 0.7. It is not clear at this
NPTNs perform slightly better than ConvNets for 0◦ rotations
because for all rotation experiments, small translations up to 2
pixels were applied only in training.
4

Non-Parametric Transformation Networks

2

Conv 5 Layers
Conv 6 Layers
Conv 8 Layers
NPTN 5 layers
NPTN 6 layers
NPTN 8 layers

0.7

0.6

0.5

Test Error %

Test Loss

0.8

1.5

1

(128)

(64, 2)

(42, 3)

(32, 4)

(Channel, |G|)
0.2

0.4

0.6

0.8

1

1.2

Layer Ratio

Figure 5. Performance of NPTN layer replacement (|G| = 3)
w.r.t the relative depth of the replacement on MNIST. Layer ratio
signifies how close to the top layer is the NPTN replacement.

point why this effect takes place. Nonetheless, the trend of
better performance at lower layer replacements is clear.
5.4. NPTNs with Capsule Networks
Capsule Networks with dynamic routing were recently introduced as an extension of standard neural networks (Sabour
et al., 2017). The main motivation behind the use of capsules (group of neurons) to represent entities is to allow
capsules to encode different properties of the object. However, since the architecture is implemented using vanilla
convolution layers, invariance properties of the networks
are limited. Our goal for this final experiment is to augment Capsule Nets with NPTNs. We do this by replacing
the convolution layers in the Primary Capsule layer of the
published architecture with NPTN layers while maintaining
the same number of parameters (by reducing number of
channels and increasing |G|). In effect, the convolutional
capsules were replaced by NPTN capsules. Our baseline is
the proposed CapsuleNet with 3 layers using a third party
implementation PyTorch5 . The number of output channels
in the first convolution layer was kept at 48. The baseline
convolution capsule layer had 128 output channels. The
NPTN variants progressively decreased the number of channels as |G| was increased. All other hyperparameters were
preserved. The networks were trained on the 2-pixel shifted
MNIST for 50 epochs with a learning rate of 10−3 . They
were tested on the test set with no shift. The performance
statistics of 5 runs are reported in Fig. 6. We find that for
roughly the same number of kernel filters (and parameters),
Capsule Nets have much to gain from the use of NPTN
layers (a significant loss decrease from 1.90 to 0.78 for 13 of
the baseline number of channels and |G| = 3). The learning
of invariances within each capsule significantly increases
efficacy and performance of the overall architecture.
5

https://github.com/dragen1860/CapsNet-Pytorch.git

Figure 6. Test errors on MNIST for Capsule Nets augmented with
NPTNs. (128) denotes a Capsule Network with a vanilla ConvNet.
Other labels are NPTNs with (channels, |G|). The number of
filters from left to right is {4224, 4160, 4074, 4128}. NPTNs
significantly outperform ConvNets in Capsule Nets with fewer
filters.

6. Discussion
It is clear that the success of ConvNets is not the whole story
towards solving perception. Studies into different aspects of
network design will prove to be paramount in addressing the
complex problem of not just visual but general perception.
The development of NPTNs offer one such design aspect,
i.e. modelling non-parametric invariances and symmetries
simultaneously from data. Through our experiments, we
found that NPTNs can indeed effectively learn general invariances without any apriori information. Further, they are
effective and improve upon vanilla ConvNets even when
applied to general vision data as presented in CIFAR10 with
complex unknown symmetries. This seems to be a critical
requirement for any system that is aimed at taking a step
towards general perception. Assuming knowledge of symmetries in real-world data (not just visual) is impractical and
succesful models would need to adapt accordingly.
In all of our experiments, NPTNs were compared to vanilla
ConvNet baselines with the same number of filters (and
thereby more channels). Interestingly, the superior performance of NPTNs with fewer channels than ConvNet
baselines indicates that better modelling of invariances is
a useful goal to pursue during design. Explicit and efficient modelling of invariances has the potential to improve
many existing architectures. In our experiments, we also
find that Capsule Networks which utilized NPTNs instead
of vanilla ConvNets performed much better. This motivates
and will justify more attention towards architectures and
other solutions that efficiently model general invariances
in deep networks. Such an endeavour might not only produce networks performing better in practice, it promises to
deepen our understanding of deep networks and perception
in general.

Non-Parametric Transformation Networks

References
Anselmi, Fabio, Leibo, Joel Z, Rosasco, Lorenzo, Mutch,
Jim, Tacchetti, Andrea, and Poggio, Tomaso. Unsupervised learning of invariant representations in hierarchical
architectures. arXiv preprint arXiv:1311.4158, 2013.
Anselmi, Fabio, Evangelopoulos, Georgios, Rosasco,
Lorenzo, and Poggio, Tomaso. Symmetry regularization.
Technical report, Center for Brains, Minds and Machines
(CBMM), 2017.
Burges, Christopher JC and Schölkopf, Bernhard. Improving the accuracy and speed of support vector machines.
In Advances in neural information processing systems,
pp. 375–381, 1997.
Chen, Yunpeng, Li, Jianan, Xiao, Huaxin, Jin, Xiaojie, Yan,
Shuicheng, and Feng, Jiashi. Dual path networks. In
Advances in Neural Information Processing Systems, pp.
4470–4478, 2017.
Clark, Christopher and Storkey, Amos. Training deep convolutional neural networks to play go. In International
Conference on Machine Learning, pp. 1766–1774, 2015.
Cohen, Taco and Welling, Max. Group equivariant convolutional networks. In International Conference on Machine
Learning, pp. 2990–2999, 2016a.
Cohen, Taco S and Welling, Max. Steerable cnns. arXiv
preprint arXiv:1612.08498, 2016b.
Dieleman, Sander, Willett, Kyle W, and Dambre, Joni.
Rotation-invariant convolutional neural networks for
galaxy morphology prediction. Monthly notices of the
royal astronomical society, 450(2):1441–1459, 2015.
Fasel, Beat and Gatica-Perez, Daniel. Rotation-invariant
neoperceptron. In Pattern Recognition, ICPR. 18th International Conference on, volume 3, pp. 336–339. IEEE,
2006.
Gens, Robert and Domingos, Pedro M. Deep symmetry
networks. In Advances in neural information processing
systems, pp. 2537–2545, 2014.
Goodfellow, Ian, Warde-Farley, David, Mirza, Mehdi,
Courville, Aaron, and Bengio, Yoshua. Maxout networks.
In International Conference on Machine Learning, pp.
1319–1327, 2013.
Hadsell, Raia, Chopra, Sumit, and LeCun, Yann. Dimensionality reduction by learning an invariant mapping. In
Computer vision and pattern recognition, 2006 IEEE computer society conference on, volume 2, pp. 1735–1742.
IEEE, 2006.

He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision
and pattern recognition, pp. 770–778, 2016.
Henriques, João F and Vedaldi, Andrea. Warped convolutions: Efficient invariance to spatial transformations. In
International Conference on Machine Learning, 2017.
Hinton, Geoffrey E, Krizhevsky, Alex, and Wang, Sida D.
Transforming auto-encoders. In International Conference
on Artificial Neural Networks, pp. 44–51. Springer, 2011.
Hu, Jie, Shen, Li, and Sun, Gang. Squeeze-and-excitation
networks. arXiv preprint arXiv:1709.01507, 2017.
Huang, Gao, Liu, Zhuang, Weinberger, Kilian Q, and
van der Maaten, Laurens. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
Ioffe, Sergey and Szegedy, Christian. Batch normalization:
Accelerating deep network training by reducing internal
covariate shift. In International conference on machine
learning, pp. 448–456, 2015.
Jaderberg, Max, Simonyan, Karen, Zisserman, Andrew, et al.
Spatial transformer networks. In Advances in Neural
Information Processing Systems, pp. 2017–2025, 2015.
Kavukcuoglu, Koray, Fergus, Rob, LeCun, Yann, et al.
Learning invariant features through topographic filter
maps. In Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on, pp. 1605–1612. IEEE,
2009.
Kivinen, Jyri J and Williams, Christopher KI. Transformation equivariant boltzmann machines. In International Conference on Artificial Neural Networks, pp. 1–9.
Springer, 2011.
LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 86(11):2278–2324,
1998.
Li, Junying, Yang, Zichen, Liu, Haifeng, and Cai, Deng.
Deep rotation equivariant network. arXiv preprint
arXiv:1705.08623, 2017.
Liao, Qianli, Leibo, Joel Z, and Poggio, Tomaso. Learning
invariant representations and applications to face verification. In Advances in Neural Information Processing
Systems, pp. 3057–3065, 2013.
Ngiam, Jiquan, Chen, Zhenghao, Chia, Daniel, Koh,
Pang W, Le, Quoc V, and Ng, Andrew Y. Tiled convolutional neural networks. In Advances in neural information
processing systems, pp. 1279–1287, 2010.

Non-Parametric Transformation Networks

Niyogi, Partha, Girosi, Federico, and Poggio, Tomaso. Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE, 86(11):
2196–2209, 1998.
Pal, Dipan, Kannan, Ashwin, Arakalgud, Gautam, and Savvides, Marios. Max-margin invariant features from transformed unlabelled data. In Advances in Neural Information Processing Systems 30, pp. 1438–1446, 2017.
Pal, Dipan K, Juefei-Xu, Felix, and Savvides, Marios. Discriminative invariant kernel features: a bells-and-whistlesfree approach to unsupervised face recognition and pose
estimation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pp. 5590–
5599, 2016.
Sabour, Sara, Frosst, Nicholas, and Hinton, Geoffrey E. Dynamic routing between capsules. In Advances in Neural
Information Processing Systems, pp. 3859–3869, 2017.
Schmidt, Uwe and Roth, Stefan. Learning rotation-aware
features: From invariant priors to equivariant descriptors.
In Computer Vision and Pattern Recognition (CVPR),
2012 IEEE Conference on, pp. 2050–2057. IEEE, 2012.
Schroff, Florian, Kalenichenko, Dmitry, and Philbin, James.
Facenet: A unified embedding for face recognition and
clustering. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pp. 815–823,
2015.
Sifre, Laurent and Mallat, Stéphane. Rotation, scaling and
deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference on computer
vision and pattern recognition, pp. 1233–1240, 2013.
Sohn, Kihyuk and Lee, Honglak. Learning invariant representations with local transformations. arXiv preprint
arXiv:1206.6418, 2012.
Teney, Damien and Hebert, Martial. Learning to extract
motion from videos in convolutional neural networks.
In Asian Conference on Computer Vision, pp. 412–428.
Springer, 2016.
Wu, Fa, Hu, Peijun, and Kong, Dexing. Flip-rotatepooling convolution and split dropout on convolution
neural networks for image classification. arXiv preprint
arXiv:1507.08754, 2015.
Xu, Yichong, Xiao, Tianjun, Zhang, Jiaxing, Yang, Kuiyuan,
and Zhang, Zheng. Scale-invariant convolutional neural
networks. arXiv preprint arXiv:1411.6369, 2014.
Zagoruyko, Sergey and Komodakis, Nikos. Wide residual
networks. arXiv preprint arXiv:1605.07146, 2016.

