Tensor Regression Networks with various Low-Rank Tensor Approximations
Xingwei Cao Guillaume Rabusseau
cs.LG stat.ML


































The future of gravitational theories in the era of the gravitational
wave astronomy
    Christian Corda
    
===================================================================================













Tensor regression networks achieve high compression rate of neural networks while having slight impact on performances.

They do so by imposing low tensor rank structure on the weight matrices of fully connected layers.

In recent years, tensor regression networks have been investigated from the perspective of their compressive power, however, the regularization effect of enforcing low-rank tensor structure has not been investigated enough.



We study tensor regression networks using various low-rank tensor approximations, aiming to compare the compressive and regularization power of different low-rank constraints.



We evaluate the compressive and regularization performances of the proposed model with both deep and shallow convolutional neural networks.

The outcome of our experiment suggests the superiority of Global Average Pooling Layer over Tensor Regression Layer when applied to deep convolutional neural network with CIFAR-10 dataset.

On the contrary, shallow convolutional neural networks with tensor regression layer and dropout achieved lower test error than both Global Average Pooling and fully-connected layer with dropout function when trained with a small number of samples.













§ INTRODUCTION
 
Tensor has been attracting increasing interests from the machine learning community over past decades. One of the reasons for such appreciation towards tensor is the natural representation of multi-modal data using the tensor structure. Such multi-modal dataset are often encountered in scientific fields including image analysis <cit.>, signal processing <cit.> and spatio-temporal analysis <cit.>.
Tensor methods allow statistical models to efficiently learn multilinear relationship between inputs and outputs by leveraging multilinear algebra and efficient low-rank constraints. The low-rank constraints on higher-order multivariate regression can be interpreted as a regularization technique. As shown in <cit.>, efficient low-rank multilinear regression model with tensor response can improve the performance of regression.

Incorporating tensor methods into deep neural network has become a prominent area of studies. In particular, over the past decade, tensor decomposition and approximation algorithms have been introduced to deep neural networks, notably for 1) efficient compression of the model with low-rank constraints <cit.> and 2) leveraging the multi-modal structure of the high-dimensional dataset <cit.>.
For illustration, Kossaifi et al. proposed tensor regression layer (TRL) which replaces the vectorization operation and fully-connected layers of Convolutional Neural Networks (CNNs) with higher-order multivariate regression <cit.>. The advantage of such replacement is the high compression rate of the model while preserving multi-modal information of dataset by enforcing efficient low-rank constraints. Given such high-dimensional dataset, the vectorization operation will lead to the loss of multi-modal information. The higher-level dependencies among various modes are lost when the data is mapped to a linear space. For instance, applying flattening operation to a colored image (3rd-order tensor) will remove the relationship between the red-channel and the blue-channel. Tensor regression layer is able to capture such multi-modal information by performing multilinear regression tasks between the output of the last convolutional layer and the softmax.

Following <cit.>, we investigate the property and performance of tensor regression layers from the perspectives of regularization and compression. We interpret low-rank constraints as a regularization technique for higher-order multivariate regression and enforce low-rank constraints on the weight tensor between output tensors of CNN and output vectors. Furthermore, we compare tensor regression layer with various tensor decomposition approximations. We aim to provide a comparative insight on different low-rank constraints that can be enforced on higher-order multivariate regression. We compare the performances of TRL using Tucker, CP and Tensor Train decompositions in a small standard CNN on MNIST and Fashion-MNIST. We also investigate such comparison in Residual Networks (ResNet) <cit.> on CIFAR-10. To investigate the regularization effect, we employed shallow CNNs and trained them with different numbers of training samples and compare the performances.


We show that a compression rate of 54× can be achieved using TT decomposition with a sacrifice of accuracy less than 0.3% with respect to the weight matrix of a 32-Layer Residual Network with fully-connected layer on CIFAR-10 dataset. Surprisingly, we also show that an even better compression rate with a smaller loss in accuracy on CIFAR-10 can be achieved by simply using global average pooling (GAP) followed by a small fully connected layer. However, using the same trick on the smaller CNN on MNIST led to very poor results.


The remaining of this paper is organized as follows. We start by reviewing background knowledge of multilinear algebra and tensor decomposition formats in Section <ref>. In Section <ref>, we present and investigate tensor regression layer with different tensor decomposition formats. We show that global average pooling GAP) layer is a special case of TRL with Tucker decomposition in Section <ref>. In Section <ref> we present a simple analysis of low-rank constraints showing how particular choices of the tensor rank parameters can drastically affect the expressiveness of the network.  We demonstrate empirical performance of low-rank TRL in Section <ref> followed by discussion and conclusion of our work in Section <ref>. 















§ BACKGROUND
 

















 §.§ Tensor Algebra


We begin with a concise review of notations and basics of tensor algebra. For a more comprehensive review, we refer the reader to <cit.>. Throughout this paper, a vector is denoted by boldface lowercase letter, 𝐯∈ℝ^I_1. Matrices and higher-order tensors are denoted by boldface uppercase and calligraphic letters respectively, 𝐌∈ℝ^I_1 × I_2 and T∈ℝ^I_1 × I_2 ×⋯× I_N. Given an Nth-order tensor T∈ℝ^I_1× I_2×⋯× I_N, its (i_1, i_2, …, i_N)th entry is denoted by T_i_1,i_2,… , i_N or (T)_i_1, i_2, ⋯ , i_N, where i_n=1, 2,…, I_n, ∀ n∈[1,N]. The notation [N,M] denotes the range of integers from N to M inclusive.


Given a 3rd order tensor T∈ℝ^I× J× K, its slices are the matrices obtained by fixing all but two indices; the horizontal, lateral and frontal slices of  T are denoted by 𝐓_i,:,:∈ℝ^J× K,  𝐓_:,j,:∈ℝ^I× K and  𝐓_:,:,k∈ℝ^I× J respectively. 


Similarly, the mode-n fibers of  T are the vectors obtained by fixing every index but the n-th one. The mode-n matricization or mode-n unfolding of a tensor T is the matrix having its mode-n fibers as columns and is denoted by 𝐓_(n). 



Given n vectors 𝐯_1∈ℝ^I_1,𝐯_2∈ℝ^I_2,…,𝐯_n ∈ℝ^I_N, the outer product of these vectors is denoted by 𝐯_1∘𝐯_2∘⋯∘𝐯_n∈ℝ^I_1× I_2×⋯× I_N and is defined by ( 𝐯_1∘𝐯_2∘⋯∘𝐯_n )_i_1, i_2, ⋯ ,i_n = (𝐯_1)_i_1(𝐯_2)_i_2⋯ (𝐯_n)_i_n for all i_k ∈ [1,I_k] where k ∈ [1,N]. An N-th order tensor X∈ℝ^I_1 ×⋯× I_N is called rank-one if it can be written as the outer product of N vectors (i.e. X=𝐯_1∘𝐯_2∘⋯∘𝐯_N).


The n-mode product of a tensor X∈ℝ^I_1× I_2×⋯× I_N with a matrix 𝐔∈ℝ^J× I_n is denoted by X×_n 𝐔∈ℝ^I_1 ×⋯× I_n-1× J × I_n+1×⋯× I_N and is defined by 

    (X×_n𝐔)_i_1, ⋯ , i_n-1 , j , i_n+1,⋯ ,i_N = ∑_i_n =1^I_nX_i_1, i_2, ⋯, i_N  u_j , i_n

for all i_k∈ [1,I_k], j∈[1,J] where k ∈ [1,N]. Similarly, we denote an n-mode product of a tensor X∈ℝ^I_1× I_2×⋯× I_N and a vector 𝐯∈ℝ^I_n by X[.75]_n 𝐯∈ℝ^I_1 ×⋯× I_n-1× I_n+1×⋯× I_N for all n∈[1,N] and it is defined by X[.75]_n 𝐯 = X×_n 𝐯^T.


The Kronecker product of matrices 𝐀∈ℝ^I× J and 𝐁∈ℝ^K× L is the block matrix (𝐀_i,j𝐁)_i,j of size IK× JL and is denoted by 𝐀⊗𝐁. Given matrices 𝐀 and 𝐁, both of  size I× J, their Hadamard product (or component-wise product) is denoted by 𝐀*𝐁 and defined by (𝐀*𝐁)_i,j =𝐀_i,j𝐁_i,j. The Khatri-Rao product 𝐀⊙𝐁 of matrices 𝐀∈ℝ^I× K and 𝐁∈ℝ^J× K is the IJ× K matrix  defined by

    𝐀⊙𝐁 = [
        [ 𝐚_1 ⊗𝐛_1 𝐚_2 ⊗𝐛_2        ⋯ 𝐚_K ⊗𝐛_K ]]

where 𝐚_i (resp. 𝐛_i) denotes the ith column of  𝐀 (resp. B).










 §.§ Various Tensor Decompositions

In this section we present three of the commonly used tensor decomposition formats: Candecomp/Parafac, Tucker and Tensor-Train.


CP decomposition.  The CP decomposition <cit.>  approximates a tensor with a summation of rank-one tensors <cit.>. The rank of the decomposition is simply the number of rank-one tensors used to approximate the input tensor: given an input tensor X∈ℝ^I_1× I_2×⋯× I_N, its approximation with a CP decomposition of rank R  is defined by


    X≈∑_j=1^R𝐚^(j)_1 ∘⋯∘𝐚^(j)_N = 𝐀^(1), 𝐀^(2), ⋯ , 𝐀^(N).


In Eq. (<ref>), 𝐀^(1), 𝐀^(2), ⋯ , 𝐀^(1) denotes the CP approximation of X where each matrix 𝐀^(i)∈ℝ^I_i× R consists of the R column vectors 𝐚_i^(j) for j∈[1,R].

We have the following useful expression of Eq. (<ref>) in terms of the matricization of X:


    X_(n)≈𝐀^(n)( 𝐀^(N)⊙⋯⊙𝐀^(n+1)⊙𝐀^(n-1)⊙⋯⊙𝐀^(1))^T



Tucker decomposition.  The Tucker decomposition approximates a tensor X∈ℝ^I_1× I_2×⋯× I_N by the product of a core tensor G∈ℝ^R_1× R_2×⋯× R_N and N factor matrices 𝐔^(i)∈ℝ^I_i × R_i for i∈[1,N]:

    X≈G×_1 𝐔^(1)×_2 ⋯×_N 𝐔^(N) = G; 𝐔^(1), ⋯ , 𝐔^(N)


The matricization of X from Eq. (<ref>) can be written as


    X_(n)≈𝐔^(n)G_(n)( 𝐔^(N)⊗⋯𝐔^(n+1)⊗𝐔^(n-1)⊗⋯⊗𝐔^(1))^T


The tuple (R_1 , ⋯ , R_N ) is the rank of the Tucker decomposition and determines the size of the core tensor G. An example of a Tucker approximation of a fourth order tensor is given in Figure <ref>.

Tensor train decomposition.  The tensor train (TT) decomposition <cit.> provides a space-efficient representation for higher-order tensors. It approximates a tensor X∈ℝ^I_1× I_2×⋯× I_N with the product of third order tensors G^(i)∈ℝ^R_i-1×  I_i × R_i called core tensors or simply cores. The rank of the TT decomposition is the tuple (R_0, R_1,
⋯,R_N) where R_0 = R_N = 1. 

Given a tensor X∈ℝ^I_1× I_2×⋯× I_N, the approximation by TT decomposition is defined as


    X_i_1, i_2, … , i_N≈𝐆^(1)_i_1 , :×𝐆^(2)_:, i_2, :×⋯×𝐆^(N)_:, i_N
    = ≪G^(1), ⋯ , G^(N)≫

where × denotes the matrix product.


In order to express Eq. (<ref>) in terms of matricizations of X, we first define the following contraction operation on core tensors.


 
Given a set of core tensors G^(i) in Eq. (<ref>) for i∈[1,N], we define G^<n as the product of core tensors G^(j) for j∈[1,n-1]:

    G^<n_i_1 , ⋯ , i_n-1 , i_n = 𝐆^(1)_i_1 , :𝐆^(2)_:, i_2 , :⋯𝐆^(n-1)_:,  i_n-1 , i_n.


Similarly to G^<n, we define G^>n as the product of core tensors G^(j) for j∈[n+1,N] where G^<n∈ℝ^I_1 ×⋯× I_n-1× R_n-1 and G^>n∈ℝ^R_n × I_n+1×⋯× I_N. A tensor network representation of core separation is provided in Figure <ref>.












Using Definition <ref>, the mode-n unfolding of a tensor X≈ ≪G^(1), ⋯ , G^(N)≫ in Eq. (<ref>) where n∈ [1,N] can be written as

    X_(n)≈G^(n)_(2)( G^>n_(1)⊗G^<n_(n))
















§ TENSOR REGRESSION LAYER
 






In this section, we introduce tensor regression layer via various low-rank tensor approximations. As stated in Section <ref>, the last fully-connected layer of traditional CNN represents a large proportion of the model parameters. In addition to such large consumption of computational resources, the flattening operation leads to the loss of rich multi-modal information in the last convolutional layer. Tensor regression layer <cit.> replaces such last flattening and fully connected layers of CNN by a multilinear map with low Tucker rank. In this work, we explore imposing other low-rank constraints on the weight tensor and we compare the compression and regularization effects of using either CP, Tucker or TT decompositions.

Given an input tensor X∈ℝ^I_1×⋯× I_N and a weight tensor W∈ℝ^I_1×⋯× I_N × I_N+1, we investigate the function f: ℝ^ I_1×⋯× I_N→ℝ^I_N+1 where I_N+1 is the number of classes. Given such two tensors, the function f is defined as


    f(X) = W_(N+1) vec(X) + 𝐛


where 𝐛∈ℝ^I_N+1 is a bias vector added to the product of W and X. The tensor network representation of an example of Eq. (<ref>) is given in Figure <ref>. 

The main idea behind tensor tensor regression layers is to enforce a low tensor rank structure on W in order to both reduce memory usage and to 
leverage the multilinear structure of the input X.


Throughout the paper, we denote a TRL with TT decomposition by TT-TRL. Similarly we use CP-TRL and Tucker-TRL for a TRL with CP or Tucker decomposition.

CP decomposition.  First we investigate applying CP decomposition to approximate the weight tensor W. Using Eq. (<ref>) and Eq.(<ref>), Eq. (<ref>) can be rewritten as


    f(X) 	   ≈( 𝐀^(1), 𝐀^(2), ⋯ , 𝐀^(N), 𝐀^(N+1))_(N+1) vec(X)  + 𝐛
       = 𝐀^(N+1)( 𝐀^(N)⊙⋯⊙𝐀^(1))^T  vec(X) + 𝐛


We can use this formulation to obtain the partial derivatives needed to implement gradient based optimization methods (backpropagation), indeed

    ∂ f(X)_i/∂ (𝐀^(n))_jk = (∂𝐀^(N+1)( 𝐀^(N)⊙⋯⊙𝐀^(1))^T vec(X) )_i/∂ (𝐀^(n))_jk
 
for all of the matrices 𝐀^(n) for n ∈ [1,N+1]. Furthermore, for a given mode n, we can naturally arrange these partial derivatives into a third order tensor 
∂ f(X) / ∂𝐀^(n)∈ℝ^I_N+1× I_n× R and obtain their expression using unfolding: 


    (∂ f/∂𝐀^(n))_(2) = (X)_(n) (𝐀^(N)⊙⋯⊙𝐀^(n+1)⊙𝐀^(n-1)⊙⋯⊙𝐀^(1))  (𝐀^(N+1)⊙𝐈_R)^T

for n∈[1,N], and

    (∂ f/∂𝐀^(N+1))_(1) = 𝐈_I_N+1⊗ (vec(X)^T (𝐀^(N)⊙⋯⊙𝐀^(1))).





Tucker decomposition.  As described in Section <ref>, the Tucker decomposition approximates an input tensor by a core tensor and a set of factor matrices. We can rewrite Eq. (<ref>) using approximation of the tensor W by Tucker decomposition as 


    f(X)  ≈𝐔^(N+1)G_(N+1)( 𝐔^(N)⊗⋯⊗𝐔^(1))^T  vec(X) + 𝐛


where the tensor W is approximated with

    W≈G×_1 𝐔^(1)×_2 ⋯×_N 𝐔^(N)×_N+1𝐔^(N+1) = G; 𝐔^(1), ⋯ , 𝐔^(N+1)


The tensor network representations of Eq. (<ref>) is shown in Figure <ref>. Given a tensor of size ℝ^I_1 ×⋯× I_N, the function f maps such tensor to the space ℝ^I_N+1 with low-rank constraints.



We can again  obtain concise expressions for the partial derivatives using unfoldings, for example:

    (∂ f(X)/∂𝐔^(1))_(1) = (G; 𝐈_R_1, 𝐔^(2), ⋯, 𝐔^(N+1))_(N+1)(X_(1)⊗𝐈_R_1)^T,


    (∂ f(X)/∂𝐔^(N+1))_(1) = ((G;  𝐔^(1), ⋯, 𝐔^(N), 𝐈_R_N+1)_(N+1) vec(X))^T ⊗𝐈_I_N+1

and

    (∂ f(X)/∂G)_(1) = 𝐔^(N+1)⊗  vec( X;  (𝐔^(1))^T, ⋯, (𝐔^(N))^T )^T.























Tensor Train decomposition.  The tensor network visualization is given in Figure <ref>, where the weight tensor W  is replaced with its TT representation.
Using Eq. (<ref>) and (<ref>), in the case of TT decomposition Eq. (<ref>) can be rewritten as


    f(X)    ≈(G^(N+1))_(2)( (G^>N+1)_(1)⊗(G^<N+1)_(N+1))  vec(X) + 𝐛
       = (G^(N+1))_(2)(G^<N+1)_(N+1)  vec(X) + 𝐛

where the second equality follows from the fact that G^>N+1=1. Similarly to the case of CP and Tucker decomposition, the 
partial derivatives can be summarized with

    (∂ f(X)_i/∂G^(n))_(2)
     = 
     X_(n)((G^>n_:,⋯,:,i)_(1)⊗(G^<n)_(n))^T

for all i∈[1,I_N+1] and n∈[1,N], and

    (∂ f(X)/∂G^(N+1))_(1)^T
     = 
    ((G^<N+1)_(N+1) vec(X))⊗𝐈_I_N+1.


















































§ TENSOR PERSPECTIVE ON GLOBAL AVERAGE POOLING LAYER
 




In this section, we provide an insight on Global Average Pooling layer from the perspective of tensor algebra. In particular, we show that GAP layer is a special case of Tucker-TRL.

It is a traditional practice to apply flattening operation to the output tensor (i.e. the last convolutional layer) before extracting its features. The problem of such approach lies in the generalization ability to the test dataset. Some work on deep neural networks show that fully-connected layers are prone to overfitting, thus leading to poor performance on test dataset <cit.>.

In order to tackle such generalizability problem and to provide regularization, Global Average Pooling (GAP) layer was presented by Lin et al. <cit.>. It replaces the combination of vectorization operation and fully-connected layer with averaging operation over all slices along the output channel axis. The output of a GAP layer is thus a single vector of size the number of output channels. GAP layer was empirically shown to significantly reduce the number of model parameters in CNNs <cit.>.

The authors of <cit.> claims not only that GAP layer reduces the trainable model parameters but also that it can prevent the model from overfitting during the training stage. Over the last decade, GAP layer has been adopted to some of the most successful image classification models such as Residual Networks and VGG-16 <cit.>.

More general interpretation of the convolutional output is that it is a high-order tensor X in a space ℝ^I_1 ×⋯× I_N. Given such tensor, the GAP layer will output a vector 𝐲 defined by


    ( 𝐲)_i_N = ∏_n=1^N-1 I^-1_n ∑_i_1=1^I_1∑_i_2 = 1^I_2⋯∑_i_N-1 = 1^I_N-1X_i_1 i_2 … i_N   for all  i_N∈[1,I_N].


We here assume that the axis for the output channel corresponds to the last mode of the tensor X.
We now show that a GAP layer mapping  X to 𝐲∈ℝ^I_N is equivalent to a specific Tucker-TRL with rank (1,⋯,1,I_N,I_N). Indeed,
let 

    W = G; 𝐮^(1),⋯, 𝐮^(N-1), 𝐔^(N), 𝐔^(N+1)∈ℝ^I_1×⋯× I_N× I_N

 be the
regression tensor of a Tucker-TRL, with (𝐮^(n))_i_n = 1/I_n where i_n ∈ [1,I_n] for each n∈[1,N-1], and 𝐔^(N) = 𝐔^(N+1) = G = 𝐈_I_N.
We have

    f(X)_i_N    = 
     (𝐔^(N+1)G_(N+1)(𝐔^(N)⊗𝐮^(N-1)⊗⋯⊗𝐮^(1))^T  vec(X))_i_N
       =
     (X[.75]_1 𝐮^(1)[.75]_2⋯[.75]_N-1𝐮^(N-1))_i_N
        = ∑_i_N-1=1^I_N-1⋯∑_i_2=1^I_2∑_i_1=1^I_1X_i_1 , i_2 , … , i_N (𝐮^(1))_i_1 (𝐮^(2))_i_2⋯ (𝐮^(N-1))_i_N-1
        =  ∏_n=1^N-1 I_n^-1∑_i_1=1^I_1∑_i_2 = 1^I_2⋯∑_i_N-1 = 1^I_N-1X_i_1 , i_2 , …  , i_N
       =
    (𝐲)_i_N.





Observe that the composition of a GAP layer with a fully connected layer mapping 𝐲∈ℝ^I_N to 𝐳∈ℝ^O can also be achieved using a unique Tucker-TRL
by setting  𝐔^(N+1)∈ℝ^I_N× O to be the weight matrix of the fully connected layer instead of the identity. A graphical representation of this equivalence is shown in Figure <ref>. 




