Improving Efficiency in Convolutional Neural
Network with Multilinear Filters
Dat Thanh Tran∗ , Alexandros Iosifidis† and Moncef Gabbouj∗
∗ Laboratory

arXiv:1709.09902v3 [] 23 Oct 2017

† Department

of Signal Processing, Tampere University of Technology, Tampere, Finland
of Engineering, Electrical & Computer Engineering, Aarhus University, Aarhus, Denmark
Email:{dat.tranthanh,moncef.gabbouj}@tut.fi, alexandros.iosifidis@eng.au.dk

Abstract—The excellent performance of deep neural networks
has enabled us to solve several automatization problems, opening an era of autonomous devices. However, current deep net
architectures are heavy with millions of parameters and require
billions of floating point operations. Several works have been
developed to compress a pre-trained deep network to reduce
memory footprint and, possibly, computation. Instead of compressing a pre-trained network, in this work, we propose a
generic neural network layer structure employing multilinear
projection as the primary feature extractor. The proposed architecture requires several times less memory as compared to the
traditional Convolutional Neural Networks (CNN), while inherits
the similar design principles of a CNN. In addition, the proposed
architecture is equipped with two computation schemes that
enable computation reduction or scalability. Experimental results
show the effectiveness of our compact projection that outperforms
traditional CNN, while requiring far fewer parameters.

I. I NTRODUCTION
In recent years, deep neural network architectures have
excelled in several application domains, ranging from machine
vision [1], [2], [3], natural language processing [4], [5] to
biomedical [6], [7] and financial data analysis [8], [9]. Of
those important developments, Convolutional Neural Network
(CNN) has evolved as a main workhorse in solving computer
vision tasks nowadays. The architecture was originally developed in the 1990s for handwritten character recognition
using only two convolutional layers [10]. Over the years,
with the development of Graphical Processing Units (GPUs)
and efficient implementation of convolution operation, the
depth of CNNs has been increased to tackle more complicated problems. Nowadays, prominent architectures such as
Residual Network (ResNet) [11] or Google Inception [12]
with hundreds of layers have become saturated. Researchers
started to wonder whether millions of parameters are essential
to achieve such performance. In order to extend the benefit of
such deep nets to embedded devices with limited computation
power and memory, recent works have focused on reducing the
memory footprint and computation of a pre-trained network,
i.e. they apply network compression in the post-training stage.
In fact, recent works have shown that traditional network
architectures such as Alexnet, VGG or Inception are highly
redundant structures [13], [14], [15], [16], [17], [18], [19],
[20], [21], [22]. For example, in [13] a simple heuristic based
on magnitude of the weights was employed to eliminate the

connections in a pre-trained network, which achieved considerable amount of compression without hurting the performance
much. Additionally, representing network parameters with low
bitwidth numbers, like in [23], [24], [25], has shown that
the performance of a 32-bit network can be closely retained
with only 4-bit representations. It should be noted that the
two approaches are complementary to each other. In fact, a
compression pipeline called “Deep Compression” [13] which
consists of three compression procedures, i.e. weight pruning, weight quantization and Huffman-based weight encoding,
achieved excellent compression performance on AlexNet and
VGG-16 architectures.
Along pruning and quantization, low-rank approximation
of both convolutional layers and fully connected layers was
also employed to achieve computational speed up [26], [27],
[28]. Viewed as high-order tensors, convolutional layers were
decomposed using traditional tensor decomposition methods,
such as CP decomposition [21], [20], [29] or Tucker decomposition [30], and the convolution operation is approximated
by applying consecutive 1D convolutions.
Overall, efforts to remove redundancy in already trained
neural networks have shown promising results by determining
networks with a much simpler structure. The results naturally
pose the following question: why should we compress an already trained network and not seek for a compact network representation that can be trained from scratch?. Subsequently,
one could of course exploit the above mentioned compression
techniques to further decrease the cost. Under this perspective,
the works in [19], [22] utilizing a low-rank approximation
approach were among the first to report simplified network
structures.
The success of Convolutional Neural Networks can be
attributed to four important design principles: sparse connectivity, parameter sharing, pooling and multilayer structure.
Sparse connectivity (in convolutional layers) only allows local
interaction between input neurons and output neurons. This
design principle comes from the fact that in many natural
data modalities such as images and videos local/neighboring
values are often highly correlated. These groups of local
values usually contain certain distinctive patterns, e.g. edges
and color blobs, in images. Parameter sharing mechanism in
CNNs enables the underlying model to learn location invariant

cues. In other words, by sliding the filters over the input, the
patterns can be detected regardless of the location. Pooling and
multilayer structure design of deep neural networks in general
and CNN in particular, captures the compositional hierarchies
embedded within many natural signals. For example, in facial
images, lower level cues such as edges, color and texture
patterns form discriminative higher level cues of facial parts,
like nose, eyes or lips. Similar compositional structure can be
seen in speech or text, which are composed of phonemes, syllables, words and sentences. Although the particular structure
of a deep network has evolved over time, the above important
design principles remain unchanged. At the core of any convolution layer, each filter with d×d×C elements operates as a
micro feature extractor that performs linear projection of each
data patch/volume from a feature space of d2 C dimensions to a
real value. In order to enhance the discrimination power of this
micro feature extractor, the authors of [31] proposed to replace
the GLM model by a general nonlinear function approximator,
particularly the multilayer perceptron (MLP). The resulting
architecture was dubbed Network in Network (NiN) since it
consists of micro networks that perform the feature extractor
functionality instead of simple linear projection.
In this paper, instead of seeking a more complex feature
extractor, we propose to replace the linear projection of the
traditional CNN by multilinear projection in the pursuit of
simplicity. There has been a great effort to extend traditional
linear methods to multilinear ones in an attempt to directly
learn from the natural representation of the data as high
order tensors [32], [33], [34], [35], [36], [37]. The beauty of
multilinear techniques lies in the property that the input tensor
is projected simultaneously in each tensor mode, allowing only
certain connections between the input dimensions and output
dimensions, hence greatly reducing the number of parameters. Previous works on multilinear discriminant learning and
multilinear regression [38], [32], [33], [34], [35], [36] have
shown competitive results of multilinear-based techniques. The
proposed architecture still inherits the four fundamental design
properties of a traditional deep network while utilizing multilinear projection as a generic feature extractor. The complexity
of each feature extractor can be easily controlled through the
“rank” hyper-parameter. Besides a fast computation scheme
when the network is compact, we also propose an alternative
computation method that allows efficient computation when
complexity increases.
The contribution of our paper can be summarized as follows:
• We propose a generic feature extractor that performs
multilinear mapping to replace the conventional linear
filters in CNNs. The complexity of each individual feature
extractor can be easily controlled via its rank, which is
a hyperparameter of the method. By having the ability
to adjust individual filter’s complexity, the complexity of
the entire network can be adjusted without the need of
increasing the number of filters in a layer, i.e. the width of
the layer. Since the proposed mapping is differentiable,
the entire network can be easily trained end-to-end by
using any gradient descent-based training process.

We provide the analysis of computation and memory
requirements of the proposed structure. In addition, based
on the properties of the proposed mapping, we propose
two efficient computation strategies leading to two different complexity settings.
• The theoretical analysis of the proposed approach is supported by experimental results in real-world classification
problems, in comparison with CNN and the low-rank
scheme in [19].
The remainder of the paper is organized as follows: In
section 2, we provide an overview of the related works
focusing on designing compact network structures. Section 3
gives the necessary notations and definitions before presenting
the proposed structure and its analysis. In section 4, we provide
details of our experiment procedures, results and quantitative
analysis. Section 5 concludes our work and discusses possible
future extensions.
•

II. R ELATED W ORK
Research focusing on the design of a less redundant network
architecture has gained much more attention recently. One
of the prominent design pattern is the bottleneck unit which
was first introduced in the ResNet architecture [11]. The
bottleneck pattern is formed by two 1 × 1 convolution layers
with some 3 × 3 convolution layers in between. The first
1 × 1 convolution layer is used to reduce the number of input
feature maps while the latter is used to restore the number
of output feature maps. Several works such as [39], [40],
[41] have incorporated the bottleneck units into their network structure to reduce computation and memory consumed.
Recently MobileNet architecture [42] was proposed which
replaced normal convolution operation by depthwise separable
convolution layers. Constituted by depthwise convolution and
pointwise convolution, the depthwise separable convolution
layer performs the filtering and combining steps independently.
The resulting structure is many times more efficient in terms of
memory and computation. It should be noted that bottleneck
design or depthwise separable convolution layer is a design
on a macro level of the network structure in which the
arrangements of layers are investigated to reduce computation.
On a micro level, the works in [19] and [22] assumed a
low rank structure of convolutional kernels in order to derive
a compact network structure. In fact, low rank assumption has
been incorporated into several designs prior to deep neural
networks, such as dictionary learning, wavelet transform of
high dimensional data. The first incorporation of low rank
assumption in neural network compression was proposed in
[21], [29], [20]. In [29], CP decomposition was proposed to
decompose the entire 4D convolutional layer into four 1D
convolutions. Although the effective depth of the network
remains the same, replacing one convolution operation by four
can potentially lead to difficulty in training the network from
scratch. With a carefully designed initialization scheme, the
work of [22] was able to train a mixture of low-rank filters
from scratch with competitive performances. Improving on
the idea of [29], a different low-rank structure that allows

both approximating an already trained network and training
from scratch was proposed in [19]. Specifically, let us denote
a convolution layer of N kernels by W ∈ Rd×d×C×N , where
C and d are the number of input feature maps and spatial size
of the kernel, respectively. [19] proposed to approximate W
using a vertical kernel V ∈ Rd×1×C×K and a horizontal kernel
H ∈ R1×d×K×N . The approximation is in the following form:
W̃nc ≃

K
X

Vkc (Hnk )T ,

(1)

k=1

where the superscript and subscript denote the index of the
channel and the kernel respectively. K is a hyper-parameter
controlling the rank of the matrix approximation. Here Wnc
is just the 2D kernel weight of the n-th filter applied to the
c-th channel of the input feature map; Vkc and Hnk are just
d-dimensional vectors.
As can be seen from (1), the authors simplify a convolutional layer by two types of parameter sharing. The first is
the sharing of right singular vectors (Hnk ) across all C input
channels within the n-th filter while the second enforces the
sharing of left singular vectors (Vkc ) across all N filters. The
work in [19] is closely related to ours since we avoid designing a particular initialization scheme by including a Batch
Normalization step [43]. The resulting structure was easily
trained from scratch with different network configurations.

One of the nice properties of mode-k product is that the
result of the projection does not depend on the order of
projection, i.e.


(2)
X ×k1 A ×k2 B = X ×k2 B ×k1 A.
The above property allows efficient computation of the
projection by selecting the order of computation.
B. Multilinear filter as generic feature extractor
Let Xi,j ∈ Rd×d×C and W ∈ Rd×d×C denote the input
patch centered at spatial location (i, j) and the convolution
kernel respectively. At the core of a classic CNN, each convolution kernel operates as a feature extractor sliding through
the input tensor to generate a higher level representation.
Specifically, the kernel performs the following linear mapping:

Yi,j = f Xi,j ; W; b = hXi,j , Wi + b,
(3)
where Yi,j and b denotes the response at (i, j) and the intercept
term respectively. h·, ·i denotes the dot-product between two
tensors. After the above linear projection, a nonlinearity is
applied to Yi,j using the layer’s activation function.
We propose to replace the above linear projection by the
following multilinear mapping:
R
3
Y
 X
Yi,j = f˜ Xi,j ; W̃ ; b =
Xi,j
×k wk (r)T + b,
r=1

III. P ROPOSED M ETHOD
We start this section by introducing some notations and
definitions related to our work. We denote scalar values
by either low-case or upper-case characters (x, y, X, Y . . . ),
vectors by low-case bold-face characters (x, y, . . . ), matrices
by upper-case bold-face characters (A, B, . . . ) and tensors
by calligraphic capital characters (X , Y, . . . ). A tensor is a
multilinear matrix with K modes, and is defined as X ∈
RI1 ×I2 ×···×IK , where Ik denotes the dimension in mode-k.
The entry in the ik th index in mode-k for k = 1, . . . , N is
denoted as Xi1 ,i2 ,...,iK .
A. Multilinear Algebra Concepts
Definition 1 (Mode-k Fiber and Mode-k Unfolding): The
mode-k fiber of a tensor X ∈ RI1 ×I2 ×···×IK is a vector
of Ik -dimensional, given by fixing every index but ik . The
mode-k unfolding of X , also known as mode-k matricization,
transforms the tensor X to matrix X(k) , which is formed by
arranging the mode-k fibers as columns. The shape of X(k) is
QK
RIk ×Ik̄ with Ik̄ = i=1,i6=k Ii .
Definition 2 (Mode-k Product): The mode-k product between a tensor X = [xi1 , . . . , xiK ] ∈ RI1 ×...IK and a
matrix W ∈ RJk ×Ik is another tensor of size I1 × · · · ×
Jk × · · · × IK and denoted by X ×k W. The element of
X ×k W is defined as [X ×k W]i1 ,...,ik−1 ,jk ,ik+1 ,...,iK =
PIK
ik =1 [X ]i1 ,...,ik−1 ,ik ,...,iK [W]jk ,ik .
For
Q convenience, we denote X ×1 W1 × · · · ×K WK by
X K
k=1 ×k Wk .

(4)

k=1

where R is the rank hyper-parameter of the projection and
wk (r) denotes the projection along mode-k. In our case,
w1 (r) ∈ Rd , w2 (r) ∈ Rd , w3 (r) ∈ RC , ∀r = 1, . . . , R.
Since the mapping in Eq. (4) operates on similar input patch
and yields a scalar response as a linear mapping does in CNNs,
the proposed multilinear mapping acts as a generic feature
extractor and can be incorporated into any design of the CNN
topology, such as AlexNet [44], VGG [45], Inception [41]
or ResNet [11]. In addition, since the mapping in Eq. (4) is
differentiable with respect to each individual weight vector
wk (r), the resulting network architecture can be trained in an
end-to-end fashion by back propagation algorithm. We hereby
denote the layer employing our proposed multilinear mapping
as MLconv.
Recently, mode-k multiplication has been introduced as a
tensor contraction layer in [46] to project the entire input layer
as a high-order tensor to another tensor. This is fundamentally
different from our approach since the tensor contraction layer
is a global mapping which does not incorporate sparse connectivity and parameter sharing principles. In general, modek multiplication can be applied to an input patch/volume to
output another tensor instead of a scalar as in our proposal.
We restrict the multilinear projection in the form of Eq. (4) to
avoid the increase in the output dimension which leads to computation overhead in the next layer. Moreover, tensor unfolding
operation required to perform the multilinear projection that
transforms a tensor to another tensor will potentially increase
the computation. On the contrary, our proposed mapping is a
special case of the general multilinear mapping using mode-k

Fig. 1. Illustration of the proposed multilinear mapping according to equation (6) in sequence: mode-3, mode-1 and mode-2

product in which the output tensor degenerates to a scalar. This
special case allows efficient computation of the projection, as
shown in the next section.
C. Memory and Computation Complexity
One the most obvious advantages of the mapping in Eq. (4)
is that it requires far fewer parameters to estimate the model,
compared to the linear mapping in a CNN. In a CNN utilizing
the mapping in Eq. (3), a layer with N kernels requires the
storage of d2 CN parameters. On the other hand, a similar
layer configuration with N mappings utilizing the projection
in Eq. (4) requires only R(2d + C)N parameters. The gain
ratio is:
d2 C
.
(5)
R(2d + C)
As compared to a similar CNN topology, the memory
reduction utilizing the mapping in Eq. (4) varies for different
layers. The case where C >> d (which is the usual case)
leads to a gain ratio approximately equal to d2 /R. In our
experiments, we have seen that with d = 3 and R = 2 in all
layers, memory reduction is approximately 4×, while having
competitive performance compared to a CNN with similar
network topology.
Let us denote by Xl ∈ RX×Y ×C and Wl ∈ Rd×d×C×N
the input and N kernels of the l-th convolutional layer
having C input feature maps and N output feature maps. In
addition, we assume zero-padding and sliding window with
stride of 1. By using linear projection as in case of CNN,
the computational complexity of this layer is O(d2 XY CN ).
Before evaluating the computational cost of a layer using the
proposed method, it should be noted that the projection in Eq.
(4) can be efficiently computed by applying three consecutive
convolution operations. Details of the convolution operations
depend on the order of three modes. Therefore, although the
result of the mapping in Eq. (4) is independent of the order

of mode-k projection, the computational cost actually depends
on the order of projections. For C >> d, it is computationally
more efficient to first perform the projection in mode-3 in order
to reduce the number of input feature maps for subsequent
mode-1 and mode-2 projection:
Yi,j =

R
X

Xi,j ×3 w3 (r)T ×1 w1 (r)T ×2 w2 (r)T + b. (6)

r=1

The response Yi,j in Eq. (6) is the summation of R
independent projections with each projection corresponding to
the following three consecutive steps, as illustrated in Figure
III-B:
• Projection of Xi,j along the third mode which is the linear
combination of C input feature maps. The result is a
(3)
tensor Xi,j of size d × d × 1.
(3)
• Projection of Xi,j along the first mode which is the linear
(1)
combination of d rows. The result is a tensor Xi,j of size
1 × d × 1.
(1)
• Projection of Xi,j along the second mode which is the
linear combination of d elements.
With the aforementioned configuration of the l-th layer, the
computational complexity of the l-th MLconv layer utilizing
our multilinear mapping is as follows:
• Mode-3 projection that corresponds to applying N R
convolutions to the input Xl with kernels of size 1 ×
1 × C elements, having computational complexity of
O(XY CN R). The output of the projection along the
third mode is a tensor of size X × Y × N R.
• Mode-1 projection is equivalent to applying convolution
with one d×1×N R separable convolution kernel, having
complexity of O(dXY N R). This results in a tensor of
size X × Y × N R.
• Mode-2, similar to mode-1 projection, can be computed
by applying convolution with one 1 × d × N R separable

convolution kernel, requiring O(dXY N R) computation.
This results in a tensor of size X ×Y ×N R. By summing
over R ranks, we arrive at the output of layer l of size
X × Y × N.
The total complexity of layer l using our proposed mapping
is thus O(XY N R(C +2d)). Compared to linear mapping, our
method achieves computational gain of:

the scalability of our proposed mapping when R is large
while previous subsection proposes an efficient computation
scheme that allows computation savings when R is small.
Overall, we can conclude that the computation of the proposed
layer structure is efficient while, as will be shown in the
experimental evaluation, changing the rank of the adopted
tensor definitions can increase performance.

d2 C
.
R(C + 2d)

IV. E XPERIMENTS

(7)

From Eqs. (5) and (7), we can conclude that the proposed
feature extractor achieves approximately d2 /R savings in both
computation and memory when C >> d.

In this section, we provide experimental results to support
the theoretical analysis in section III. The experimental protocol and datasets are described first, followed by the discussion
of the experimental results.

D. Initialization with pre-trained CNN

A. Network Topology

The proposed mapping in Eq. (4) can be viewed as a
constrained form of convolution kernel as follows:

Traditional CNN topology consists of two modules: feature
extractor module and classifier module. Several convolution
and pooling layers stacked on top of each other act as feature
extractor while one or two fully-connected layers act as the
classifier. In order to evaluate the effectiveness of the proposed
multilinear filter, we constructed the network architecture
with only feature extractor layers, i.e. convolution layer or
MLconv layer together with pooling layer while skipping
fully-connected layer. As the name suggests, fully-connected
layer has dense connections, accounting for large number of
parameters in the network while being prone to overfitting.
Moreover, a powerful and effective feature extractor module
is expected to produce a highly discriminative latent space
in which the classification task is made simple. Such fullyconvolutional networks have attracted much attention lately
due to their compactness and excellent performance in imagerelated problems like semantic segmentation, object localization and classification [31], [2], [48]
The configuration of the baseline network adopted in our
experiment benchmark is shown in Table I where 3 × 3 × N
denotes N kernels with 3 × 3 spatial dimension, BN denotes
Batch Normalization [43] and LReLU denotes Leaky Rectified
Linear Unit [49] with α = 0.2. Our baseline architecture is
similar to the one proposed in [50] with four key differences.
Firstly, we choose to retain a proper pooling layer instead
of performing convolution with a stride of 2 as proposed in
[50]. Secondly, Batch Normalization was applied after every
convolution layer except the last one where the output goes
through softmax to produce the class probability. In addition,
LReLU activation unit was applied to the output of batch
normalization. It has been shown that the adoption of BN
and LReLU speeds up the learning process of the network
by being more tolerant to the learning rate with the possibility
of arriving at better minimas [43], [51].
Based on the configuration of the network topology, we
compare the performance between standard linear convolution
kernel (CNN), our proposed multilinear kernel (MLconv) and
the low-rank (LR) structure proposed in [19]. The last two 1×1
convolution layers were not replaced by LR or MLconv layer.
It should be noted that BN and LReLU are applied to all three
competing structures in our experiments while in [19], BN was

Yi,j = hXi,j , W̃i + b,

(8)
PR

where W̃ is expressed in Kruskal form W̃ = r=1 w1 (r) ◦
w2 (r) ◦ w3 (r) as the outer-product of the corresponding
projection in three modes. By calculating Yi,j using modek product definition as in Eq. (4) and using dot-product as in
Eq. (8), the equivalance of Eq. (4) and Eq. (8) can be found
[47].
Consequently, a convolution layer can be converted to an
MLconv layer by decomposing each 3D convolution filter
into Kruskal form using any CP decomposition method [47].
It should be noted here that, since there is no closed-form
solution of the CP decomposition, such a conversion corresponds to an approximation step. Under this perspective, a
pre-trained CNN can be used to initialize our network structure
to speed up the training process. However, as we will show in
the experimental section, random initialization of multilinear
filters can lead to better performance.
In addition to an initialization scheme, Eq. (8) also complements our proposed mapping with an efficient computation
strategy when R is large. The computation cost discussed
in the previous subsection depends linearly with parameter
R. When R is large, it is more efficient to compute the
mapping according to Eq. (8) by first calculating W̃ and then
convolving the input with W̃. The computational complexity
of the first step is O(d2 CRN ) while for the convolution
step is O(d2 XY CN ), resulting to an overall complexity
of O(d2 CRN + d2 XY CN ) for the entire layer. The ratio
between normal convolution layer and MLconv layer using
this computation strategy is:
XY
.
(9)
R + XY
It is clear that XY is usually much larger than R, therefore, the
increase in computation as compared to normal convolution is
marginal. Following this calculation strategy, a rank 6 network
is marginally slower than a rank 1 network or a CNN. This
will be demonstrated in our experiment section. In conclusion,
the computation method discussed in this subsection allows

C. Experimental settings

TABLE I
BASELINE N ETWORK A RCHITECTURE

Input layer
3 × 3 × 96 - BN - LReLU
3 × 3 × 96 - BN - LReLU
3 × 3 × 96 - BN - LReLU
2 × 2 MaxPooling
3 × 3 × 192 - BN - LReLU
3 × 3 × 192 - BN - LReLU
3 × 3 × 192 - BN - LReLU
2 × 2 MaxPooling
3 × 3 × 192 - BN - LReLU
1 × 1 × 192 - BN - LReLU
1 × 1 × #class LReLU
Global Average over spatial dimension
softmax activation

not applied to the baseline CNN which could potentially lead
to biased result.
B. Datasets
1) CIFAR-10 and CIFAR-100: CIFAR dataset [52] is an
object classification dataset which consists of 50000 color
images for training and 10000 for testing with the resolution
32 × 32 pixels. CIFAR-10 refers to the 10-class classification
problem of the dataset in which each class has 5000 images for
training and 1000 images for testing while CIFAR-100 refers
to a more fine-grained classification of the images into 100
classes.
2) SVHN: SVHN [53] is a well-known dataset for handwritten digit recognition problem which consists of more
than 600k images of house numbers extracted from natural
scenes with varying number of samples from each class. This
dataset poses a much harder character recognition problem as
compared to the MNIST dataset [10]. We used 32×32 cropped
images provided by the database from which each individual
image might contain some distracting digits on the sides.

All networks were trained using both SGD optimizer [54]
as well as Adam [55]. While the proposed structure tends to
arrive at better minimas with Adam, this is not the case for
the other two methods. For SGD optimizer, the momentum
was fixed to 0.9. We adopted two sets of learning rate schedule SC1 = {0.01, 0.005, 0.001, 0.0005, 0.0001} and SC2 =
{0.01, 0.001, 0.0001}. Each schedule has initial learning rate
γ = 0.01 and decreases to the next value after E epochs where
E was cross-validated from the set {40, 50, 60, 100, 120}. We
trained each network with maximum of 300 and 100 epochs
for CIFAR and SVHN respectively. The batch size was fixed
to 200 samples for all competing networks.
Regarding data augmentation, for CIFAR dataset, random
horizontally flipped samples were added as well as random
translation of the images by maximum 5 pixels were performed during the training process; for SVHN dataset, only
random translation of maximum 5 pixels was performed. For
both dataset, no further preprocessing step was applied.
Regarding regularization, both weight decay and max-norm
[56] are individually and together exploited in our experiments.
Max-norm regularizer was introduced in [56] where it was
used together with Dropout. During the training process, the
l2 norm of each individual filter is constrained to lie inside the
ball of a given radius r which was cross-validated from the
set {1.0, 2.0, 4.0, 6.0, 8.0}. The weight decay hyper-parameter
λ was searched from the set {0.001, 0.0005}. In addition,
Dropout with pi = 0.2 was applied to the input and Dropout
with p = po was applied to the output of all pooling layers with
the optimal po obtained from the set {0.1, 0.2, 0.3, 0.4, 0.5}.
Due to the differences between the three competing structures,
we observed that while the baseline CNN and LR networks
work well with weight decay, applying weight decay to the
proposed network structure tends to drive all the weight values
close to zeros when λ is large, or the regularization effect
is marginal when using a small value for λ, leading to the

TABLE II
CIFAR-10 C LASSIFICATION ERROR (%)

Scratch
Pretrained
# Parameters

CNN
7.47
−
≈ 1.38M

MLconv1
8.54
8.17
≈ 0.20M

LR26
9.14
8.64
≈ 0.20M

MLconv2
7.68
7.76
≈ 0.35M

LR53
8.31
7.49
≈ 0.35M

MLconv4
7.34
7.38
≈ 0.65M

LR106
8.00
7.10
≈ 0.64M

MLconv6
7.30
7.06
≈ 0.97M

MLconv4
28.27
28.45
≈ 0.67M

LR106
30.11
28.40
≈ 0.67M

MLconv6
28.08
28.51
≈ 0.98M

TABLE III
CIFAR-100 C LASSIFICATION ERROR (%)

Scratch
Pretrained
# Parameters

CNN
29.60
−
≈ 1.39M

MLconv1
31.32
31.88
≈ 0.21M

LR26
35.79
33.98
≈ 0.21M

MLconv2
29.10
29.86
≈ 0.37M

LR53
31.45
30.00
≈ 0.37M

CNN
MLconv
LR
MLconv-i
LR-i

Classification Error (%) on CIFAR-10

9.0
LR26i
MLconv1
LR53

8.5

MLconv1i

8.0

LR106

MLconv2i
MLconv2

7.5

LR53i

MLconv4
LR106i

7.0

0.2

0.4

CNN

MLconv4i
MLconv6
MLconv6i

0.6
0.8
1.0
Number of parameters (10e6)

1.2

36
Classification Error (%) on CIFAR-100

LR26

CNN
MLconv
LR
MLconv-i
LR-i

LR26

35
LR26i

34
33
32
31

MLconv1i
LR53
MLconv1
MLconv2i
LR53i
MLconv2

30
29
28

1.4

Fig. 2. Model size versus Classification Error on CIFAR-10 for different
structures. MLconv and LR network initialized with CNN marked with ”i” at
last

0.2

0.4

LR106
MLconv4i
LR106i
MLconv4

CNN
MLconv6i
MLconv6

0.6
0.8
1.0
Number of parameters (10e6)

1.2

1.4

Fig. 3. Model size versus Classification Error on CIFAR-100 for different
structures. MLconv and LR network initialized with CNN marked with ”i” at
last

D. Experimental results
exhaustive search of suitable hyper-parameter λ. On the other
hand, max-norm regularization works well with our method
without being too sensitive to the performance.
For MLconv and LR structures, we experimented with
several values for the rank parameter, namely R for the
proposed mapping and K in Eq. (1) from [19]. In all of our
experiments, we made no attempt to optimize R and K for
each individual filter and layer in order to get the maximal
compact structure, since such an approach is impractical in
real cases. We instead used the same rank value throughout all
layers. The experiments are, hence, different from [19] where
the authors reported performance for different values of K
at each layer without discussing the rank selection method.
The experiments were conducted with R = 1, 2, 4, 6 and the
corresponding structures are denoted as MLconv1, MLconv2,
MLconv4, MLconv6. The values of K are selected so that
the number of parameters in an LR network is similar to the
number of parameters of its MLconv counterpart with given R.
The corresponding LR structures are denoted as LR26, LR53
and LR106, where the number denotes the value of K. We did
not perform experiments with K = 159, which corresponds
to R = 6, since training the network is computationally much
slower and falls out of the objective of this paper.
All of three competing structures training from scratch
were initialized with random initialization scheme proposed
in [57]. We additionally trained MLconv and LR structure
with weights initialized from an optimal pre-trained CNN
on CIFAR dataset. The aforementioned protocols were also
applied for this configuration. The weights of MLconv were
initialized with CP decomposition using canonical alternating
least square method [47], while for the LR structure we
followed the calculation proposed in [19].

After obtaining the optimal hyper-parameter values, each
network was trained for five times and the median value is
reported. The second row of Tables II and III shows the
classification errors of all competing methods trained from
scratch on CIFAR-10 and CIFAR-100, respectively, while
the third row shows the performance when initialized with
a pre-trained CNN. The last row reports the model size of
each network. As can be seen from both Tables II and III,
using the proposed multi-linear filters leads to a 2× reduction
in memory, while outperforming the standard convolution
filters in both coarse and fine-grained classification in CIFAR
datasets. More interestingly, in CIFAR-100, a rank 4 multilinear filter network attains an improvement over 1%. As
we increase the number of projections in each mode to 6,
i.e. when using R = 6, the performance of the network
increases by a small margin. In both CIFAR-10 and CIFAR100, constraining R = 2 gains 4× memory reduction while
keeping the performance relatively closed to the baseline CNN
with less than 0.5% increment in classification error. Further
limiting R to 1 maximizes the parameter reduction to nearly
7× with the cost of 1.07% and 1.72% increase in error
rate for CIFAR-10 and CIFAR-100, respectively. A graphical
illustration of the compromise between number of network’s
parameters and classification error on CIFAR-10 and CIFAR100 is illustrated in Figures 2 and 3, respectively.
The classification error of each competing network trained
from scratch on SVHN dataset is shown in Table IV. Using our
proposed MLconv layers, we achieved 4× reduction in model
size while slightly outperforming CNN. At the most compact
configuration of MLconv structure, i.e. MLconv1, we only
observed a small increment of 0.12% in classification error as
compared to CNN baseline. As we increased the complexity
of MLconv layers, little improvement was seen with MLconv4

TABLE IV
SVHN C LASSIFICATION ERROR

CNN
MLconv1
LR26
MLconv2
LR53
MLconv4
LR106
MLconv6

Error (%)
1.80
1.92
1.96
1.76
1.85
1.75
1.78
1.80

#Parameters
≈ 1.38M
≈ 0.20M
≈ 0.20M
≈ 0.35M
≈ 0.35M
≈ 0.65M
≈ 0.64M
≈ 0.97M

while MLconv6 layers became slightly overfitted.
Comparing the proposed multi-linear filter with the low
rank structure LR, all configurations of MLconv network
significantly outperform their LR counterparts. Specifically,
in the most compact configuration, MLconv1 is better than
LR26 by 0.6% and 4.47% on CIFAR-10 and CIFAR-100,
respectively. The margin shrinks as the complexity increases
but the proposed structure consistently outperforms LR when
training the network from scratch. Similar comparison results
can be observed on SVHN dataset: using MLconv layers
obtained lower classification errors as compared to LR layers
at all complexity configurations. As opposed to the experimental results reported in [19], we observed inferior results
of the LR structure compared to standard CNN when training
from scratch. The difference might be attributed to two main
reasons: we incorporated batch normalization into the baseline
CNN which could potentially improve the performance of
the baseline CNN; our baseline configuration has no fullyconnected layer to solely benchmark the efficiency of different
filter structures as a feature extractor.
One interesting phenomenon was observed when we initialized MLconv and LR with a pre-trained CNN. For the LR
structure, most configurations enjoy substantial improvement
by initializing the network with weights decomposed from
a pre-trained CNN on CIFAR dataset. The contrary happens
for our proposed MLconv structure, since most configurations
observe a degradation in performance. This can be explained
by the fact that LR structure was designed to approximate
each individual 2D convolution filter at every input feature
map and the resulting structure comes with a closed-form
solution for the approximation. With good initialization from
a CNN, the network easily arrived at a good minimum

while training a low-rank setting from scratch might have
difficulty at achieving a good local minimum. Although the
proposed mapping can be viewed as a form of convolution
filter, the mapping in Eq. (4) embeds a multi-linear structure,
hence possessing certain degree of difference. Initializing the
proposed mapping by applying CP decomposition, which has
no closed-form solution, may lead the network to a suboptimal
state.
Table V reports the average forward propagation time of a
single sample measured on CPU for all three network structures on CIFAR-10. The second and third columns report the
theoretical and actual speed-ups, respectively, measured by the
number of multiply-accumulate operations normalized with
respect to their convolution counterparts. For the proposed
MLconv structure, we report the computation cost of both
calculation strategies discussed in Section III. We refer to
the first calculation strategy using the separable convolution
as Scheme1, while the latter one using normal convolution
as Scheme2. Results from Scheme2 are denoted with the
asterisk. All the networks are implemented using Keras library
[58] with Tensorflow [59] backend. It is clear that there
is a gap between theoretical speed-up and actual speed-up,
especially for the proposed structure implemented by an unoptimized separable convolution operation. In fact, at the time
of writing, implementation of separable convolution operation
is still missing in most libraries, not to mention efficient
implementation. On the contrary, results from Scheme2 using
normal convolution show a near perfect match between theory
and implementation. This is due to the fact that normal
convolution operation has been efficiently implemented and
optimized in most popular libraries. This also explains why
the computation gain of LR structure is inferior to MLconv
structure (Scheme1) in theory but similar to ours in practice
since LR structure is realized by normal convolution operation.
The last four columns of Table V additionally prove the
scalability of Scheme2 with respect to the hyper-parameter
rank R as discussed in Section III-D.
V. C ONCLUSIONS
In this paper, we proposed a multilinear mapping to replace
the conventional convolution filter in Convolutional Neural
Networks. The resulting structure’s complexity can be flexibly
controlled by adjusting the number of projections in each
mode through a hyper-parameter R. The proposed mapping
comes with two computation schemes which either allow
memory and computation reduction when R is small, or the
scalability when R is large. Numerical results showed that with
far fewer parameters, architectures employing our mapping

TABLE V
F ORWARD C OMPUTATION T IME ON CIFAR10 ( NORMALIZED WITH RESPECT TO C ONV )

Theory
Implementation

MLconv1
8.14×
1.77×

LR26
6.48×
1.79×

MLconv2
4.11×
1.33×

LR53
3.20×
1.54×

MLconv4
2.06×
1.03×

LR106
1.60×
1.23×

MLconv6
1.38×
0.74×

MLconv1*
0.997×
0.990×

MLconv2*
0.993×
0.980×

MLconv4*
0.987×
0.960×

MLconv6*
0.980×
0.930×

could outperform standard CNNs. This are promising results
and opens future research directions focusing on optimizing
parameter R on individual convolution layers to achieve the
most compact structure and performance.
R EFERENCES
[1] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,”
in Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 580–587, 2014.
[2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look
once: Unified, real-time object detection,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, pp. 779–788,
2016.
[3] M. A. Waris, A. Iosifidis, and M. Gabbouj, “Cnn-based edge filtering
for object proposals,” Neurocomputing, 2017.
[4] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly,
A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al., “Deep neural
networks for acoustic modeling in speech recognition: The shared views
of four research groups,” IEEE Signal Processing Magazine, vol. 29,
no. 6, pp. 82–97, 2012.
[5] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with
deep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pp. 6645–6649,
IEEE, 2013.
[6] M. Zabihi, A. B. Rad, S. Kiranyaz, M. Gabbouj, and A. K. Katsaggelos,
“Heart sound anomaly and quality detection using ensemble of neural
networks without segmentation,” in Computing in Cardiology Conference (CinC), 2016, pp. 613–616, IEEE, 2016.
[7] X. An, D. Kuang, X. Guo, Y. Zhao, and L. He, “A deep learning method
for classification of eeg data based on motor imagery,” in International
Conference on Intelligent Computing, pp. 203–210, Springer, 2014.
[8] A. Tsantekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj, and
A. Iosifidis, “Using deep learning to detect price change indications
in financial markets,” in European Signal Processing Conference (EUSIPCO), Kos, Greece, 2017.
[9] A. Tsantekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj, and
A. Iosifidis, “Forecasting stock prices from the limit order book using
convolutional neural networks,” in Business Informatics (CBI), 2017
IEEE 19th Conference on, vol. 1, pp. 7–12, IEEE, 2017.
[10] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning
applied to document recognition,” Proceedings of the IEEE, vol. 86,
no. 11, pp. 2278–2324, 1998.
[11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, pp. 770–778, 2016.
[12] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
in Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 1–9, 2015.
[13] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and huffman
coding,” arXiv preprint arXiv:1510.00149, 2015.
[14] Y. Guo, A. Yao, and Y. Chen, “Dynamic network surgery for efficient
dnns,” in Advances In Neural Information Processing Systems, pp. 1379–
1387, 2016.
[15] W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and
Y. Chen, “Compressing convolutional neural networks,” arXiv preprint
arXiv:1506.04449, 2015.
[16] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured
sparsity in deep neural networks,” in Advances in Neural Information
Processing Systems, pp. 2074–2082, 2016.
[17] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep
convolutional networks using vector quantization,” arXiv preprint
arXiv:1412.6115, 2014.
[18] D. Lin, S. Talathi, and S. Annapureddy, “Fixed point quantization of
deep convolutional networks,” in International Conference on Machine
Learning, pp. 2849–2858, 2016.
[19] C. Tai, T. Xiao, Y. Zhang, X. Wang, and E. Weinan, “Convolutional neural networks with low-rank regularization,” arXiv preprint
arXiv:1511.06067, 2015.

[20] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus,
“Exploiting linear structure within convolutional networks for efficient
evaluation,” in Advances in Neural Information Processing Systems,
pp. 1269–1277, 2014.
[21] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv preprint
arXiv:1405.3866, 2014.
[22] Y. Ioannou, D. Robertson, J. Shotton, R. Cipolla, and A. Criminisi,
“Training cnns with low-rank filters for efficient image classification,”
arXiv preprint arXiv:1511.06744, 2015.
[23] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio,
“Quantized neural networks: Training neural networks with low precision weights and activations,” arXiv preprint arXiv:1609.07061, 2016.
[24] P. Gysel, M. Motamedi, and S. Ghiasi, “Hardware-oriented approximation of convolutional neural networks,” arXiv preprint
arXiv:1604.03168, 2016.
[25] S.-C. Zhou, Y.-Z. Wang, H. Wen, Q.-Y. He, and Y.-H. Zou, “Balanced
quantization: An effective and efficient approach to quantized neural
networks,” Journal of Computer Science and Technology, vol. 32, no. 4,
pp. 667–682, 2017.
[26] M. Denil, B. Shakibi, L. Dinh, N. de Freitas, et al., “Predicting parameters in deep learning,” in Advances in Neural Information Processing
Systems, pp. 2148–2156, 2013.
[27] A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov, “Tensorizing neural networks,” in Advances in Neural Information Processing
Systems, pp. 442–450, 2015.
[28] S. Lin, R. Ji, X. Guo, X. Li, et al., “Towards convolutional neural networks compression via global error reconstruction.,” in IJCAI, pp. 1753–
1759, 2016.
[29] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, “Speeding-up convolutional neural networks using fine-tuned cpdecomposition,” arXiv preprint arXiv:1412.6553, 2014.
[30] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, “Compression
of deep convolutional neural networks for fast and low power mobile
applications,” arXiv preprint arXiv:1511.06530, 2015.
[31] M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv preprint
arXiv:1312.4400, 2013.
[32] Q. Li and D. Schonfeld, “Multilinear discriminant analysis for higherorder tensor data classification,” IEEE transactions on pattern analysis
and machine intelligence, vol. 36, no. 12, pp. 2524–2537, 2014.
[33] S. Yan, D. Xu, Q. Yang, L. Zhang, X. Tang, and H.-J. Zhang, “Discriminant analysis with tensor representation,” in Computer Vision and Pattern
Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on,
vol. 1, pp. 526–532, IEEE, 2005.
[34] H. Zhou, L. Li, and H. Zhu, “Tensor regression with applications
in neuroimaging data analysis,” Journal of the American Statistical
Association, vol. 108, no. 502, pp. 540–552, 2013.
[35] D. T. Thanh, J. Kanniainen, M. Gabbouj, and A. Iosifidis, “Tensor representation in high-frequency financial data for price change prediction,”
arXiv preprint arXiv:1709.01268, 2017.
[36] D. Tao, X. Li, X. Wu, and S. J. Maybank, “General tensor discriminant
analysis and gabor features for gait recognition,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 29, no. 10, 2007.
[37] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Mpca: Multilinear
principal component analysis of tensor objects,” IEEE Transactions on
Neural Networks, vol. 19, no. 1, pp. 18–39, 2008.
[38] W. Guo, I. Kotsia, and I. Patras, “Tensor learning for regression,” IEEE
Transactions on Image Processing, vol. 21, no. 2, pp. 816–827, 2012.
[39] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” arXiv preprint
arXiv:1707.01083, 2017.
[40] M. Wang, B. Liu, and H. Foroosh, “Design of efficient convolutional
layers using single intra-channel convolution, topological subdivisioning
and spatial bottleneck structure,”
[41] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4,
inception-resnet and the impact of residual connections on learning.,” in
AAAI, pp. 4278–4284, 2017.
[42] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint
arXiv:1704.04861, 2017.
[43] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in International
Conference on Machine Learning, pp. 448–456, 2015.

[44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.
[45] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[46] J. Kossaifi, A. Khanna, Z. C. Lipton, T. Furlanello, and A. Anandkumar,
“Tensor contraction layers for parsimonious deep nets,” arXiv preprint
arXiv:1706.00439, 2017.
[47] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,”
SIAM review, vol. 51, no. 3, pp. 455–500, 2009.
[48] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference
on Medical Image Computing and Computer-Assisted Intervention,
pp. 234–241, Springer, 2015.
[49] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” in Proc. ICML, vol. 30, 2013.
[50] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller,
“Striving for simplicity: The all convolutional net,” arXiv preprint
arXiv:1412.6806, 2014.
[51] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified
activations in convolutional network,” arXiv preprint arXiv:1505.00853,
2015.
[52] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from
tiny images,” 2009.
[53] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng,
“Reading digits in natural images with unsupervised feature learning,”
in NIPS workshop on deep learning and unsupervised feature learning,
vol. 2011, p. 5, 2011.
[54] D. E. Rumelhart, G. E. Hinton, R. J. Williams, et al., “Learning
representations by back-propagating errors,” Cognitive modeling, vol. 5,
no. 3, p. 1, 1988.
[55] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[56] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neural networks
from overfitting.,” Journal of machine learning research, vol. 15, no. 1,
pp. 1929–1958, 2014.
[57] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification,” in
Proceedings of the IEEE international conference on computer vision,
pp. 1026–1034, 2015.
[58] F. Chollet, “keras.” https://github.com/fchollet/keras, 2015.
[59] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow,
A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur,
J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah,
M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale
machine learning on heterogeneous systems,” 2015. Software available
from tensorflow.org.

