Tensor Regression Networks
Jean Kossaifi Zachary C. Lipton Arinbjorn Kolbeinsson Aran Khanna Tommaso Furlanello Anima Anandkumar
cs.LG







Aapo Hyvarinen
 
Tensor Regression Networks
    Jean Kossaifi jean.kossaifi@imperial.ac.uk 

       NVIDIA & Imperial College London
       Zachary C. Lipton zlipton@cmu.edu 

       Carnegie Mellon University
       Arinbjörn Kolbeinsson ak711@imperial.ac.uk

       Imperial College London
       Aran Khanna arankhan@amazon.com 

       Amazon AI
       Tommaso Furlanello furlanel@usc.edu

       University of Southern California
       Anima Anandkumar anima@caltech.edu

       NVIDIA &
       California Institute of Technology

    December 30, 2023
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




Convolutional neural networks 
typically consist of many convolutional layers
followed by one or more fully connected layers. 
While convolutional layers map between 
high-order activation tensors, 
the fully connected layers 
operate on flattened activation vectors.
Despite empirical success,
this approach has notable drawbacks.
Flattening followed by fully connected layers 
discards multilinear structure in the activations 
and requires many parameters. 
We address these problems
by incorporating tensor algebraic operations 
that preserve multilinear structure at every layer.
First, we introduce Tensor Contraction Layers (TCLs)
that reduce the dimensionality of their input 
while preserving their multilinear structure using tensor contraction. 
Next, we introduce Tensor Regression Layers (TRLs),
which express outputs through a low-rank multilinear mapping 
from a high-order activation tensor 
to an output tensor of arbitrary order. 
We learn the contraction and regression factors end-to-end, 
and produce accurate nets with fewer parameters.
Additionally, our layers regularize networks
by imposing low-rank constraints 
on the activations (TCL) and regression weights (TRL).
Experiments on ImageNet show that, 
applied to VGG and ResNet architectures,
TCLs and TRLs reduce the number of parameters
compared to fully connected layers by more than 65% 
while maintaining or increasing accuracy. 
In addition to the space savings, 
our approach's ability 
to leverage topological structure 
can be crucial for structured data such as MRI. 
In particular, we demonstrate significant performance improvements 
over comparable architectures 
on three tasks associated with the UK Biobank dataset. 



    Machine Learning, Tensor Methods, Tensor Regression Networks, Low-Rank Regression, Tensor Regression Layers, Deep Learning, Tensor Contraction




§ INTRODUCTION

Many natural datasets exhibit multi-modal structure. 
We represent audio spectrograms as 2-order tensors (matrices)
with modes corresponding 
to frequency and time. 
We represent images as 3-order tensors 
with modes corresponding to  
width, height and the color channels.
Videos are expressed 
as 4-order tensors,
and the signal processed 
by an array of video sensors 
can be described as a 5-order tensor.
Multilinear structure arises naturally in many medical applications:
MRI images are 3-order tensors
and functional MRI images are 4-order tensors.
Generally, a broad array of multimodal data
can be naturally encoded as tensors. 
Tensor methods extend linear algebra 
to higher order tensors and
are promising tools for manipulating and analyzing such data.
 
The mathematical properties of tensors
have long been the subject of theoretical study.
Previously, in machine learning,  
data points were typically assumed to be vectors 
and datasets to be matrices. 
Hence, spectral methods, 
such as matrix decompositions, 
have been popular in machine learning. 
Recently, tensor methods,
which generalize these techniques to higher-order tensors, have gained prominence.
One class of broadly useful techniques 
within tensor methods are tensor decompositions, 
which have been studied for a variety of applications
in signal processing and machine learning <cit.>, 
data mining and fusion <cit.>, 
blind source separation <cit.>,
computer vision <cit.>
and learning latent variable models <cit.>.

 
Deep Neural Networks (DNNs) frequently 
manipulate high-order tensors:
in a standard deep Convolutional Neural Network (CNN) for image recognition,
the inputs and the activations 
of convolutional layers
are 3-order tensors. 
And yet, to wit, most architectures 
output predictions
by first flattening the activation tensors
and then connecting to the output neurons 
via one or more fully connected layers. 
This approach presents several issues: 
(i) we lose multimodal information
during the flattening process; 
and (ii) the fully connected layers
require a large number of parameters. 




  
In this paper,
we propose Tensor Contraction Layers (TCLs) 
and Tensor Regression Layers (TRLs)
as end-to-end trainable 
components of neural networks. 
In doing so, we exploit multilinear structure 
without giving up the power and flexibility 
offered by modern deep learning methods. 
By replacing fully connected layers 
with tensor contractions, 
we aggregate long-range spatial information 
while preserving multi-modal structure.
Moreover, by enforcing low rank, 
we reduce the number of parameters needed significantly with minimal impact on accuracy.

Our proposed TRL
expresses the regression weights
through the factors of a low-rank tensor decomposition. 
The TRL obviates the need for flattening,
instead leveraging the structure
when generating output.
By combining tensor regression 
with tensor contraction,
we further increase efficiency. 
Augmenting the VGG and ResNet architectures, 
we demonstrate improved performance on the ImageNet dataset 
despite significantly reducing the number of parameters (almost by 65%). 
The ability to preserve the topological structure in the data 
is particularly crucial for prediction from MRI data. 
In this context, we conduct experiments for 3 different tasks 
(gender classification, body mass index prediction and age regression) on the UK biobank dataset, 
the largest available MRI dataset. 
There, we demonstrate superior performance with our approach
and show large performance improvements for all 3 tasks.
This is the first paper to present 
an end-to-end trainable architecture 
that retains the multi-modal tensor structure throughout the network.



  
 Related work: 
Several recent papers apply tensor decomposition to deep learning. 
One notable line of application is to re-parametrize existing layers
using tensor decomposition either to speed these up or reduce the number of parameters.
<cit.> propose using CP decomposition 
to speed up convolutional layers. 
Similarly, <cit.> propose to use tensor decomposition 
to remove redundancy in convolutional layers and express these 
as the composition of two convolutional layers with less parameters. 

<cit.> take a pre-trained network 
and apply tensor (Tucker) decomposition on the convolutional kernel tensors
and then fine-tune the resulting network. 
<cit.> propose weight sharing 
in multitask learning and <cit.> 
propose sharing residual units.
<cit.> use 
the Tensor-Train (TT) format  
to impose low-rank tensor structure on weights 
of the fully connected layers in order to compress them. 
However, they still retain 
the fully connected layers for the output.
In addition, the reshaping to arbitrary higher orders and dimensions
does not guarantee that the multilinear structure is preserved. 
By contrast, we present an end-to-end tensorized network architecture 
that focuses on leveraging that structure.
Many of these contributions are orthogonal to ours and can be applied together.


Despite the success of DNNs, 
many open questions remain
as to why they work so well
and whether they really need so many parameters.
Tensor methods have emerged as promising tools of analysis 
to address these questions
and to better understand the success of deep neural networks.
<cit.>, for example,
use tensor methods as tools of analysis 
to study the expressive power of CNNs, 
while the follow up work <cit.> 
focuses on the expressive power of overlapping architectures of deep learning. 
<cit.> derive sufficient conditions
for global optimality and optimization of non-convex factorization problems,
including tensor factorization and deep neural network training.
<cit.> explore 
contraction as a composition operator for NLP.
Other papers investigate tensor methods 
as tools for devising neural network learning algorithms
with theoretical guarantees of convergence <cit.>. 


Several prior papers address the 
power of tensor regression to
preserve natural multi-modal structure 
and learn compact predictive models <cit.>.
However, these works typically rely on analytical solutions 
and require manipulating large tensors containing the data.
They are usually used for small datasets or require to downsample the data 
or extract compact features prior to fitting the model,
and do not scale to large datasets such as ImageNet.


To our knowledge, no prior work combines tensor contraction or tensor regression 
with deep learning in an end-to-end trainable fashion. 




 


§ MATHEMATICAL BACKGROUND



  
Notation
Throughout the paper, we define tensors as multidimensional arrays, 
with indexing starting at 0. 
First order tensors are vectors, denoted v. 
Second order tensors are matrices, denoted M 
and  is the identity matrix. 
By X, we denote tensors of order 3 or greater. 
For a third order tensor X,
we denote its element (i, j, k) as X_i_1, i_2, i_3. 
A colon is used to denote all elements of a mode, e.g.,
the mode-1 fibers of X 
are denoted as X_:, i_2, i_3. 
The transpose of M is denoted M. 
Finally, for any i, j ∈,  i < j, ij denotes
the set of integers { i, i+1, ⋯ , j-1, j}. 



  
Tensor unfolding
Given a tensor,
X∈^I_0 × I_1 ×⋯× I_N,
its mode-n unfolding is a matrix X_[n]∈^I_n × I_M, 
with I_M = ∏_k=0,
k ≠ n^N I_k
and is defined by the mapping from element
(i_0, i_1, ⋯, i_N) to (i_n, j), with 
j = ∑_k=0,
k ≠ n^N i_k ×∏_l=k+1,
 l ≠ n^N I_l. 
We use the definition introduced in <cit.>,
which corresponds to an underlying row-wise ordering of the elements. 
This differs from the definition used by <cit.>,
which correponds to an underlying column-wise ordering of the elements. Throughout the paper, we assume the elements are arranged in a row-wise manner, which is reflected in the definitions of unfolding, 
vectorization, and the resulting formulas. 
This row-ordering elements matches the actual ordering on GPUs
and allows for more efficient implementation.



  
Tensor vectorization
Given a tensor,
X∈^I_0 × I_1 ×⋯× I_N, 
we can flatten it into a vector vec(X) 
of size (I_0 ×⋯× I_N) 
defined by the mapping from element
(i_0, i_1, ⋯, i_N) of X to element j of vec(X), with 
j = ∑_k=0^N i_k ×∏_m=k+1^N I_m. As for unfolding, we assume a row-ordering of the elements, following <cit.>.



  
n-mode product
For a tensor X∈^I_0 × I_1 ×⋯× I_N and a matrix M∈^R × I_n, the n-mode product of a tensor 
is a tensor of size 
(I_0 ×⋯× I_n-1× R × I_n+1×·× I_N) 
and can be expressed using unfolding of X 
and the classical dot product as

    X×_n M = MX_[n]∈^I_0 ×⋯× I_n-1× R × I_n+1×⋯× I_N.




  
Generalized inner-product
For two tensors X, Y∈^I_0 × I_1 ×⋯× I_N of same size, their inner product is defined as 
XY= ∑_i_0=0^I_0-1∑_i_1=0^I_1 - 1⋯∑_i_n=0^I_N - 1X_i_0, i_1, ⋯, i_nY_i_0, i_1, ⋯, i_n 
For two tensors X∈^I_x × I_1 × I_2 ×⋯× I_N and Y∈^I_1 × I_2 ×⋯× I_N × I_y sharing N modes of same size,
we similarly define a “generalized inner product” 
along the N last (respectively first) modes 
of X (respectively Y) as 

    XY_N = ∑_i_1=0^I_1-1∑_i_2=0^I_1 - 1⋯∑_i_n=0^I_N - 1X_:, i_1, i_2, ⋯, i_nY_i_1, i_2, ⋯, i_n, :,

with XY_N ∈^I_x × I_y.



  
Tucker decomposition
Given a tensor X∈^I_0 × I_1 ×⋯× I_N, 
we can decompose it into a low rank core 
G∈^R_0 × R_1 ×⋯× R_N 
by projecting along each of its modes
with projection factors 
( U^(0), ⋯,U^(N)), with U^(k)∈^R_k × I_k, k ∈0N.
In other words, we can write

    X   = 
    G×_0 U^(0)×_1  U^(2)×⋯×_N U^(N)
       = GU^(0),
    		  ⋯,
              U^(N).

Typically, the factors and core of the decomposition 
are obtained by solving a least squares problem. 
In particular, closed form solutions 
can be obtained for the factor by considering the n-mode unfolding of X 
that can be expressed as

    X_[n] = U^(n)G_[n](U^(-k)),

where U^(-k) is defined as follows:

    U^(-k)  =  U^(0)⊗⋯U^(n-1)⊗U^(n+1)⊗⋯⊗U^(N).

Notice the natural ordering of the factors, from 0 to N, 
that follows from our definition of unfolding. 

Similarly, we can optimize the core in a straightforward manner 
by isolating it using the equivalent rewriting of the above equality:

    (X) = 
      ( U^(0)⊗⋯⊗U^(N)) (G).

We refer the interested reader to the thorough review of the literature 
on tensor decompositions by <cit.>.
 


§ TENSOR CONTRACTION LAYER

One natural way to incorporate 
tensor operations into a neural network 
is to apply tensor contraction 
to an activation tensor in order to obtain 
a low-dimensional representation <cit.>. In this section, we explain how to incorporate tensor contractions
into neural networks as a differentiable layer.


We call this technique the Tensor Contraction layer (TCL).
Compared to performing a similar rank reduction
with a fully connected layer,
TCLs require fewer parameters
and less computation, while preserving the multilinear structure of the activation tensor.





 §.§ Tensor contraction layers


Given an activation tensor X of size ( S_0, I_0, I_1, ⋯, I_N ), 
the TCL will produce a compact core tensor G of smaller size ( S_0, R_0, R_1, ⋯, R_N ) defined as

    X' = 
    X×_1 V^(0)×_2  V^(1)×⋯×_N+1V^(N),

with V^(k)∈^R_k × I_k, k ∈0N. 
Note that the projections start at the second mode
because the first mode S_0 corresponds to the batch.

This layer allows us to contract the input activation tensor
without discarding its multi-linear structure. 
By contrast, a flattening layer followed 
by a fully connected layer would discard that information.
One way to see this is to recognise that a fully connected layer 
is simply a tensor contraction over the second mode of a batch 
of flattened activations, as exposed in Subsection <ref> below.

The projection factors (V^(k))_k ∈ [1, ⋯ N]  are learned end-to-end with the rest of the network by gradient backpropagation. In the rest of this paper, we denote size–(R_0, ⋯, R_N) TCL, or TCL–(R_0, ⋯, R_N)
a TCL that produces a compact core of dimension (R_0, ⋯, R_N).



 §.§ Gradient back-propagation

In the case of the TCL, 
we simply need to take the gradients with respect to the factors V^(k) for each k ∈0, ⋯, N of the tensor contraction. Specifically, we compute

    X'V^(k) = 
    X×_1 V^(0)×_2  V^(1)×⋯×_N+1V^(N)V^(k).

By rewriting the previous equality in terms of unfolded tensors,
we get an equivalent rewriting where we have isolated the considered factor:

    X'_[k]V^(k) = 
     V^(k)X_[k](⊗V^(-k))V^(k),

with 

    V^(-k)   =	V^(0)⊗⋯V^(k-1)⊗V^(k+1)⊗⋯⊗V^(N).










 §.§ Model analysis




  
Link with fully connected layers 
Let's considering an activation tensor X of size ( S_0, I_0, I_1, ⋯, I_N ).
A size–(R_0, R_1, ⋯, R_N ) TCL 
parameterized by weight factors V^(0), ⋯, V^(N) 
is equivalent to a fully connected layer 
parametrized by the weight matrix W = (V^(0)⊗⋯⊗V^(N)) and would compute

    X_[0]W =  X_[0](V^(0)⊗⋯⊗V^(N)),

where X_[0] corresponds to the unfolding of X 
along the first mode, e.g., a vectorization of each of the samples in the batch.



  
Number of parameters
Considering an activation tensor X of size
( S_0, I_0, I_1, ⋯, I_N ), 
a size–(R_0, R_1, ⋯, R_N ) TCL parameterized by weight factors V^(0), ⋯, V^(N) and
taking X as input will have 
a total of ∑_k=0^N I_k × R_k parameters.

This is to contrast with an equivalent fully connected layer (as presented above), parametrized a weight matrix W = (V^(0)⊗⋯⊗V^(N)), which would have a total of ∏k=0^N I_k × R_k parameters.

Notice how the product in number of parameters of the fully connected layer becomes a sum when using a TCL. In other words, in addition to preserving the topological structure in the activation tensor, the TCL has significantly less parameters than a corresponsing fully connected layer.

 


§ TENSOR REGRESSION LAYER


In this section, we introduce the Tensor Regression Layer, 
a new differentiable neural network layer.
In order to generate outputs, 
CNNs typically either flatten the activations 
or apply a spatial pooling operation. 
In either case, they discard all multimodal structure
and subsequently apply a fully-connected output layer.
Instead, we propose leveraging that multilinear structure 
in the activation tensor 
and formulate the output as lying in a low-rank subspace 
that jointly models the input and the output. 
We do this by means of a low-rank tensor regression, 
where we enforce a low multilinear rank 
of the regression weight tensor.




 §.§ Tensor regression as a layer

Let us denote by
X∈^S × I_0 × I_1 ×⋯× I_N
the input activation tensor corresponding to a batch of S samples (X_1, ⋯, X_S) and 
Y∈^ S × O
the O corresponding labels for each sample. 
We are interested in the problem 
of estimating the regression weight tensor 
W∈^I_0 × I_1 ×⋯× I_N × O
under some fixed low rank (R_0, ⋯, R_N, R_N+1) and a bias b∈^O, such that Y = XW_N + b, i.e.,

    Y    = XW_N + b
    subject to    W =  
        GU^(0),
    		  ⋯,
              U^(N), U^(N + 1),

with XW_N = X_[0]×W_[N+1] the contraction of X by W along their N last (respectively first) modes, G∈^R_0 ×⋯× R_N × R_N+1, 
U^(k)∈^I_k × R_k for each k in 0N
and U^(N+1)∈^O × R_N+1.
 
Previously, this setting has been studied as a standalone problem. 
In that setting, the input data is directly mapped to the output,
and the problem solved analytically
<cit.>.
However, this either limits the model to raw data 
(e.g., pixel intensities) or requires pre-processing the data 
to extract (hand-crafted) features to feed the model.
For instance, fiducial points that encode the geometry (e.g. facial landmarks) are extracted <cit.>.
In addition, analytical solutions are prohibitive 
in terms of computation and memory usage for large datasets.

 
In this work, we incorporate tensor regressions as trainable layers in neural networks, which allow to learn jointly the features (e.g. via convolutional layers) and the tensor regression.
We do so by replacing the traditional 
flattening + fully connected layers 
with a tensor regression applied directly 
to the high-order input and enforcing 
low rank constraints on the weights of the regression. 
We call our layer the Tensor Regression Layer (TRL). 
Intuitively, the advantage of the TRL comes from leveraging the multi-modal structure in the data and expressing the solution as lying on a low rank manifold encompassing both the data and the associated outputs.




 §.§ Gradient backpropagation


The gradients of the regression weights and the core 
with respect to each factor 
can be obtained by writing:

    WU^(k)=
    G×_0 U^(0)×_1  U^(1)×⋯×_N+1U^(N+1)U^(k)

Using the unfolded expression of the regression weights, we obtain the equivalent formulation:

    W_[k]U^(k)=
    U^(k)G_[k]𝐑U^(k),

with

    𝐑 = U^(0)⊗⋯U^(k-1)⊗U^(k+1)⊗⋯⊗U^(N+1).

Similarly, we can obtain the gradient with respect to the core 
by considering the vectorized expressions:

    vec(W)vec(G)=
      ( U^(0)⊗⋯⊗U^(N+1)) vec(G)vec(G).






 §.§ Model analysis

We consider as input an activation tensor X∈^S × I_0 × I_1 ×⋯× I_N, 
and a rank-(R_0, R_1, ⋯, R_N, R_N+1) tensor regression layer, 
where, typically, R_k ≤ I_k. 
Let's assume the output is n-dimensional.
A fully connected layer taking X (after a flattening layer) as input will have n_FC parameters, with

    n_FC = n ×∏_k=0^N I_k

By comparison, a rank-(R_0, R_1, ⋯, R_N, R_N+1) TRL taking X as input has a number of parameters n_TRL, with:

    n_TRL =  ∏_k=0^N+1 R_k + ∑_k=0^N R_k × I_k + R_N+1× n.





§ EFFICIENT IMPLEMENTATION OF TENSOR REGRESSION LAYERS


Based on the previous layers, we propose an equivalent,
more efficient practical implementation of the tensor regression layer.
Equation <ref> can be written:

    Y = XG×_0 U^(0)×_1  U^(1)×⋯×_N+1U^(N+1)_N + b

We can rewrite this equivalently as

    Y = X×_0 (U^(0))×_1  (U^(1))×⋯×_N(U^(N))G×_N+1U^(N+1)_N + b.

This way, most of the computation is done in the low-rank subspace 
rather than directly on the dimensions of X.

In practice, we use this formulation for implementation, 
and directly learn the pseudo-inverse of each factorV^(k)∈^I_k × R_k for each k in 0N:

    Y = X×_0 V^(0)×_1  V^(1)×⋯×_NV^(N)G×_N+1U^(N+1)_N + b

Note that this is equivalent to first applying a TCL on X,
and then applying a TRL on the result to produce Y,
with the first factors set to the identity 
(i.e., the low-rank constraint is applied only 
to the modes corresponding to the output).

In addition to this efficient formulation, 
it is also possible to achieve computational speedup via hardware acceleration 
by leveraging recent work, e.g., <cit.>,
on extending BLAS primitives. 
This would allow us to reduce the computational overhead for transpositions,
which are necessary when computing tensor contractions.
 


§ EXPERIMENTS

We empirically demonstrate the effectiveness 
of preserving the tensor structure 
through tensor contraction and tensor regression 
by integrating it into state-of-the-art architectures 
and demonstrating similar performance 
on the popular ImageNet dataset.
We show that a TRL gives equal or greater performance 
as compared to flattening followed by fully connected layers,
while allowing for large space savings.
In particular, we show that our proposed layers allow us 
to best leverage multi-linear structure in data. 
This is particularly important for MRI data. 
On the largest database available, the U.K. biobank, 
we show that this approach outperforms its traditional counterparts
by large margins on three separate prediction tasks.

We empirically verify the effectiveness of the TCL on VGG-19 <cit.>. 
We also conduct thorough experiments and ablation studies 
with the proposed layers on VGG-19, AlexNet, ResNet-50 and ResNet-101 <cit.> in various scenarios.



 §.§ Implementation details

We implemented all models using the MXNet library <cit.> 
as well as the PyTorch library <cit.>. For all tensor methods, we used the TensorLy library <cit.>s.
The models were trained with data parallelism 
across multiple GPUs on Amazon Web Services, 
with 4 NVIDIA k80 GPUs. 
For training on ImageNet, we adopt the same data augmentation procedure as in the original Residual Networks (ResNets) paper <cit.>.

When training the layers from scratch, 
we found it useful to add a batch normalization layer <cit.> before and after the TCL/TRL 
to avoid vanishing or exploding gradients,
and to make the layers more robust to changes 
in the initialization of the factors. 
In addition, we constrain the weights 
of the tensor regression 
by applying ℓ_2 normalization <cit.> 
to the factors of the Tucker decomposition. 

When experimenting with the tensor regression layer, 
instead of retraining the whole network each time 
it is possible to start from a pre-trained ResNet. 
We experimented with two settings: 
(i) We replaced the last average pooling, 
flattening and fully connected layers 
by either a TRL or a combination of TCL + TRL 
and trained these from scratch 
while keeping the rest of the network fixed;
and (ii) We investigate replacing the pooling and fully connected layers 
with a TRL that jointly learns the spatial pooling
as part of the tensor regression. 
In that setting, we also explore initializing the TRL 
by performing a Tucker decomposition 
on the weights of the fully connected layer.




 §.§ Large scale image classification

First, we report results in the typical setting 
for large scale image classification on the the widely-used ImageNet-1K dataset, 
by learning the spatial pooling as part of the tensor regression. 

The ILSVRC dataset <cit.> (ImageNet) 
is composed of 1.2 million images for training 
and 50,000 for validation, 
all labeled for 1,000 classes. 
Following <cit.>, 
we report results on the validation set 
in terms of Top-1 accuracy and Top-5 accuracy across all 1,000 classes.
Specifically, we evaluate the classification error 
on single 224 × 224 single center crop from the raw input images. 


In this setting, we remove the average pooling layer 
and feed the tensor of size 
(batch size, number of channels, height, width) to the TRL, 
while imposing a rank of 1 on the spatial dimensions 
of the core tensor of the regression.
Effectively, this setting simultaneously learns weights 
for the multilinear spatial pooling as well as the regression.




Our experimental results show that our method enables an
effective trade-off between performance and space savings
(Table <ref>).
In particular, small space savings (e.g. about 25%) 
translate in  marginal increases in performance.
It is possible to obtain more than 65% space savings without impacting accuracy, 
and to reach larger performance space savings 
with litte impact on performance (e.g. almost 80% space savings 
with less than 1% decrease in Top-1 and Top-5 accuracy).
We can express the space savings of a model M 
with n_M total parameters in its fully connected layers 
with respect to a reference model R 
with n_R total parameters 
in its fully connected layers 
as 1 - n_M/n_R (bias excluded).


To study the TRL in isolation, we consider the weights of a pre-trained model
and replace the flattening and fully connected layer with a TRL 
while keeping the rest of the network fixed.
In practice, to initialize the weights of the TRL in this setting, 
it is possible to consider the weights of the fully connected layer 
as a tensor of size 
(batch size, number of channels, 1, 1, number of classes) 
and apply a partial Tucker decomposition to it 
by keeping the first dimension (batch-size) untouched. 
The core and factors of the decomposition 
then give us the initialization of the TRL. 
The projection vectors over the spatial dimension 
are then initialized to 
1/height and1/width, respectively. 
The Tucker decomposition was performed using TensorLy <cit.>. 
In this setting, we show that we can drastically decrease 
the number of parameters with little impact on performance.



 §.§ Large scale phenotypical trait prediction from MRI data


A major challenge in neuroscience and healthcare is analysing structure-rich data, 
such as 3D brain scans <cit.>.
The human brain is a highly complex and structured organ.
It is composed of interconnected heterogeneous components, 
each of which is structurally distinct <cit.>. 
The organizational properties are critical to the brain’s,
and the individual’s, overall health.
Studies of brain topology have revealed that structural changes 
are associated with cognitive function <cit.> 
and diseases such as diabetes <cit.>.
Retaining this topological information is thus critical 
in order to maximize modelling accuracy. 
However, flattening layers followed by fully connected layers 
discards that information and are therefore sub-optimal for the task. 
By contrast, our proposed tensor regression layers naturally leverages 
and preserves the topological information, allowing for better performance.
We empirically demonstrate this on the UK biobank MRI dataset, 
the largest imaging study of its kind <cit.>, the results can be seen in Table <ref>.



We split the data into a training set containing 11,500 scans, 
a validation set of 3,800 scans and 3,800 scans for a held-out test set.
Each scan is a T1-weighted three-dimensional structural MRI with dimensions 182×218×182. 
We select three tasks: classifying gender, 
predicting body mass index (BMI) and predicting age from raw brain MRI scans. 
Predicting age is an important task as the difference 
between true and predicted age is a biomarker 
that has a number of clinical applications <cit.>. Similarly, learning to predict BMI from MRI is a valuable objective, 
as influence of obesity on brain structure has already been established <cit.>,
but its mechanism is not fully understood. 
For the third task of gender classification, 
differences between genders (more precisely biological sex) in brain functional mappings 
have already been described <cit.>. 
The reasons and implications (if any) of these differences are not known,
but understanding the biological processes underpinning them 
could provide insights into how the brain works.
For gender prediction the classes are fairly balanced,
the male:female ratio in the UK Biobank is 0.46:0.54. 

We compare a standard ResNet with 3D convolutions (3D-ResNet) 
to a 3D-ResNet where the final fully connected layer was replaced with a TRL.
The baseline ResNet contains a global-average pooling layer 
that compresses all three spatial dimensions, 
giving an output size of 512 × n_outputs. 
The models model were implemented using PyTorch <cit.> 
and TensorLy <cit.> and trained was done on a Tesla P100 GPU.
Both networks were trained from random initialization. 
The same number of iterations was used for all comparisons. 
After validating the number of iterations, 
we found the baseline and the TRL model required same number of iterations 
in order to reach convergence and the same number of iterations 
was subsequently used for all models and all experiments.

The results (table. <ref>) demonstrate that our tensor-based architecture 
is able to learn a mapping between structural MRI and phenotypes. 
It performs significantly better on all tasks compared to a 3D-ResNet
with traditional average pooling operation and a fully connected layer.
In particular, the 3D-ResNet achieved a mean absolute error (MAE) of 2.96 years 
on the age-regression task compared with 2.70 years for our TRL.
Similarly, the TRL out-performed the baseline on the gender classifying task, 
achieving an error of only 0.53 % (accuracy of 99.47 %) 
compared to an error of 0.79 % for the baseline. 
For BMI regression, the TRL also performed with a notable improvement over the baseline. 
A secondary baseline, 3D-ResNet with no average pooling, 
failed to train at all due to unstable gradients.

These three important tasks demonstrate that the TRL is leveraging 
additional information in the structural MRI data beyond that of the baseline model. 
Our method improves on previously reported brain age accuracy
in UK Biobank <cit.>, and published studies <cit.> on other brain MRI datasets. 
The method also improves on BMI prediction beyond those 
previously published for UK Biobank data <cit.>.
Improved performance gives further support for use of machine learning 
as a support tool in neuroradiological research and clinical decision making.

One question is whether this improvements are due to the ability of the TRL 
to preserve and leverage topological structure or whether 
it is simply a result of the regularizing effect of the TRL. 
To answer this question, we experimented with a TRL,
where the entire 6 × 7 × 6 full-rank activation tensor 
was used for the age prediction task. 
Using the full-rank tensor the network achieved a MAE of 2.71 years,
which is similar to the results obtained with lower-rank set ups, 
and significantly better than the baseline. 
This empirically confirms the importance of leveraging topological structure. 




 §.§ Ablation studies




  
Synthetic setting

To illustrate the effectiveness of the low-rank tensor regression, 
in terms of learning and data efficiency,
we first investigate it in isolation. 

We apply it to synthetic data 
y = vec(X) ×W 
where each sample X∈^(64  64) 
follows a Gaussian distribution 𝒩(0, 3). 
W is a fixed matrix
and the labels are generated as 
y = vec(X) ×W. 
We then train the data on X + E,
where E is added Gaussian noise 
sampled from 𝒩(0, 3). 
We compare i) a TRL with squared loss and 
ii) a fully connected layer with a squared loss. 


    
In Figure <ref>, 
we show the trained weight of both a linear regression based on a fully connected layer 
and a TRL with various ranks, both obtained in the same setting. 
The TRL is able to better recover the ground-truth weight
due to the low-rank structure imposed on these, 
which allows to leverage the multi-linear structure
and acts as an implicit regularizer. 
This is in line with findings from existing works 
focusing on analytical solution to tensor regression problems <cit.>. 
Additional regularizers such as Lasso (l1) <cit.>
can provide further advantages, especially in the face of noisy inputs. 
Going forward, we plan to investigate such additional regularization terms. 




We also compare the data efficiency of a tensor regression layer 
compared to a linear regression, in the same setting,
by varying the number of training samples.
As can be observed in Figure <ref>, 
the TRL is easier to train on small datasets and less prone to overfitting, 
due to the low rank structure of its regression weights, as opposed to typical 
(fully connected) linear regression.



  
Impact of the tensor contraction layer
We first investigate the effectiveness of the TCL 
using a VGG-19 network architecture <cit.>. 
This network is especially well-suited for our methods
because of its 138,357,544 parameters, 
119,545,856 of which (more than 80% of the total number of parameters) 
are contained in the fully-connected layers. 
By adding a TCL to contract the activation tensor 
prior to the fully connected layers, 
we can achieve large space saving. 





Table <ref> presents the accuracy obtained 
by the different combinations of TCL 
in terms of top-1 and top-5 accuracy as well as space saving. 
By adding a TCL that preserves the size of its input 
we are able to obtain slightly higher performance 
with little impact on the space saving (0.21% of space loss) 
while by decreasing the size of the TCL we got more than 65%
space saving with almost no performance deterioration.




  
Overcomplete TRL




We first tested the TRL with a ResNet-50 and a ResNet-101 architectures on ImageNet,
removing the average pooling layer 
to preserve the spatial information in the tensor.
The full activation tensor is directly passed on 
to a TRL which produces the outputs 
on which we apply softmax to get the final predictions. 
This results in more parameters as the spatial dimensions are preserved.
To reduce the computational burden 
but preserve the multi-dimensional information, 
we alternatively insert a TCL before the TRL. 

In Table <ref>, we present results obtained in this setting 
on ImageNet for various configurations of the network architecture. 
In each case, we report the size of the TCL 
(i.e. the dimension of the contracted tensor) 
and the rank of the TRL (i.e. the dimension of the core of the regression weights).



  
Choice of the rank of the TRL
While the rank of the TRL 
is an additional parameter to validate, 
it turns out to be easy to tune in practice. 
    





In Figure <ref>, 
we show the effect on Top-1 and Top-5 accuracy 
of decreasing the size of the core tensor 
of the TRL. 
We also show the corresponding space savings.
The results suggest that choosing the rank is easy
because there is a large range 
of values of the rank 
for which the performance does not decrease. 
In particular, we can obtain up to 80% 
space savings with negligible impact on performance. 
 


§ CONCLUSIONS

Deep neural networks already operate on multilinear activation tensors, 
the structure of which is typically discarded 
by flattening operations and fully-connected layers. 
This paper proposed preserving and leveraging 
the tensor structure of the activations 
by introducing two new, end-to-end trainable, layers 
that enable substantial space savings 
while preserving and leveraging the multi-dimensional topological structure. 
The TCL that we propose reduces the dimension 
of the input without discarding its multi-linear structure, 
while TRLs directly map their input tensors
to the output with low-rank regression weights. 
These techniques are easy to plug in to existing architectures
and are trainable end-to-end.

Our experiments demonstrate 
that by imposing a low-rank constraint 
on the weights of the regression, 
we can learn a low-rank manifold 
on which both the data and the labels lie. 
Furthermore these new layers act as an additional type of regularization 
on the activations (TCL) and the regression weight tensors (TRL).
The result is a compact network 
that achieves similar accuracies with far fewer parameters.
The structure in the regression weight tensor 
allows for more interpretable models while requiring less data to train.
Going forward, we plan to apply the TCL and TRL to more network architectures 
and leverage recent work to avoid computational overhead 
from transpositions when computing tensor contractions.
 


§ ACKNOWLEDGEMENTS

This research has been conducted using the UK Biobank Resource under Application Number 18545.
The authors would like to thank the editor and anonymous reviewers for the constructive feedback which helped improve this manuscript.

0.2in
54


urlstyle

[Alosco et al.(2014)Alosco, Stanek, Galioto, Korgaonkar, Grieve,
  Brickman, Spitznagel, and Gunstad]alosco2014body
Michael L Alosco, Kelly M Stanek, Rachel Galioto, Mayuresh S Korgaonkar,
  Stuart M Grieve, Adam M Brickman, Mary Beth Spitznagel, and John Gunstad.
Body mass index and brain structure in healthy children and
  adolescents.
International Journal of Neuroscience, 1240
  (1):0 49–55, 2014.

[Anandkumar et al.(2014)Anandkumar, Ge, Hsu, Kakade, and
  Telgarsky]anandkumar2014tensor
Animashree Anandkumar, Rong Ge, Daniel J Hsu, Sham M Kakade, and Matus
  Telgarsky.
Tensor decompositions for learning latent variable models.
Journal of Machine Learning Research, 150
  (1):0 2773–2832, 2014.

[Andreasen et al.(1993)Andreasen, Flaum, Swayze, O'Leary, Alliger,
  Cohen, Ehrhardt, Yuh, et al.]andreasen1993intelligence
Nancy C Andreasen, Michael Flaum, Victor Swayze, Daniel S O'Leary, Randall
  Alliger, Gregg Cohen, James Ehrhardt, William T Yuh, et al.
Intelligence and brain structure in normal individuals.
American Journal of Psychiatry, 150:0 130–130, 1993.

[Chen et al.(2015)Chen, Li, Li, Lin, Wang, Wang, Xiao, Xu, Zhang, and
  Zhang]mxnet
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao,
  Bing Xu, Chiyuan Zhang, and Zheng Zhang.
Mxnet: A flexible and efficient machine learning library for
  heterogeneous distributed systems.
CoRR, abs/1512.01274, 2015.

[Chen et al.(2018)Chen, Jin, Kang, Feng, and Yan]chen2017sharing
Yunpeng Chen, Xiaojie Jin, Bingyi Kang, Jiashi Feng, and Shuicheng Yan.
Sharing residual units through collective tensor factorization to
  improve deep neural networks.
In IJCAI, pages 635–641. International Joint Conferences on
  Artificial Intelligence Organization, 7 2018.

[Cichocki et al.(2015)Cichocki, Mandic, Lathauwer, Zhou, Zhao, Caiafa,
  and PHAN]cichocki_2015
A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa, and H. A.
  PHAN.
Tensor decompositions for signal processing applications: From
  two-way to multiway component analysis.
IEEE Signal Processing Magazine, 320 (2):0
  145–163, March 2015.

[Cichocki et al.(2009)Cichocki, Zdunek, Phan, and
  Amari]nonnegative_cichocki_2009
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-Ichi Amari.
Nonnegative Matrix and Tensor Factorizations: Applications to
  Exploratory Multi-Way Data Analysis and Blind Source Separation.
John Wiley & Sons, Ltd, 2009.

[Cohen et al.(2016)Cohen, Sharir, and Shashua]cohen2015expressive
Nadav Cohen, Or Sharir, and Amnon Shashua.
On the expressive power of deep learning: A tensor analysis.
In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir, editors,
  29th Annual Conference on Learning Theory, volume 49 of
  Proceedings of Machine Learning Research, pages 698–728, Columbia
  University, New York, New York, USA, 23–26 Jun 2016. PMLR.

[Cole et al.(2017)Cole, Poudel, Tsagkrasoulis, Caan, Steves, Spector,
  and Montana]cole2017predicting
James H Cole, Rudra PK Poudel, Dimosthenis Tsagkrasoulis, Matthan WA Caan,
  Claire Steves, Tim D Spector, and Giovanni Montana.
Predicting brain age with deep learning from raw imaging data results
  in a reliable and heritable biomarker.
NeuroImage, 163:0 115–124, 2017.

[Cole et al.(2018)Cole, Ritchie, Bastin, Hernández, Maniega, Royle,
  Corley, Pattie, Harris, Zhang, et al.]cole2018brain
James H Cole, Stuart J Ritchie, Mark E Bastin, MC Valdés Hernández,
  S Muñoz Maniega, Natalie Royle, Janie Corley, Alison Pattie, Sarah E
  Harris, Qian Zhang, et al.
Brain age predicts mortality.
Molecular psychiatry, 230 (5):0 1385, 2018.

[Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]imagenet
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR, 2009.

[Franke et al.(2018)Franke, Gaser, Roseboom, Schwab, and
  de Rooij]franke2018premature
Katja Franke, Christian Gaser, Tessa J Roseboom, Matthias Schwab, and Susanne R
  de Rooij.
Premature brain aging in humans exposed to maternal nutrient
  restriction during early gestation.
NeuroImage, 173:0 460–471, 2018.

[Gaser et al.(2013)Gaser, Franke, Klöppel, Koutsouleris, Sauer,
  Initiative, et al.]gaser2013brainage
Christian Gaser, Katja Franke, Stefan Klöppel, Nikolaos Koutsouleris,
  Heinrich Sauer, Alzheimer's Disease Neuroimaging Initiative, et al.
Brainage in mild cognitive impaired patients: predicting the
  conversion to alzheimer’s disease.
PloS one, 80 (6), 2013.

[Grefenstette and Sadrzadeh(2011)]Grefenstette2011experimental
Edward Grefenstette and Mehrnoosh Sadrzadeh.
Experimental support for a categorical compositional distributional
  model of meaning.
In Proceedings of the Conference on Empirical Methods in
  Natural Language Processing, page 1394–1404, USA, 2011. Association for
  Computational Linguistics.
ISBN 9781937284114.

[Guo et al.(2012)Guo, Kotsia, and Patras]guo2012tensor
W. Guo, I. Kotsia, and I. Patras.
Tensor learning for regression.
IEEE Transactions on Image Processing, 210
  (2):0 816–827, Feb 2012.

[Haeffele and Vidal(2015)]haeffele2015global
Benjamin D. Haeffele and René Vidal.
Global optimality in tensor factorization, deep learning, and beyond.
CoRR, abs/1506.07540, 2015.

[He et al.(2016)He, Zhang, Ren, and Sun]resnet
K. He, X. Zhang, S. Ren, and J. Sun.
Deep residual learning for image recognition.
In IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), pages 770–778, June 2016.

[He et al.(2016)He, Zhang, Ren, and Sun]he2016identity
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks.
In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,
  Computer Vision – ECCV 2016, pages 630–645. Springer International
  Publishing, 2016.
ISBN 978-3-319-46493-0.

[Hoefling(2010)]hoefling2010path
Holger Hoefling.
A path algorithm for the fused lasso signal approximator.
Journal of Computational and Graphical Statistics, 190
  (4):0 984–1006, 2010.
10.1198/jcgs.2010.09208.

[Huang et al.(2016)Huang, Sun, Liu, Sedra, and
  Weinberger]huang2016stochastic
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.
Deep networks with stochastic depth.
In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,
  Computer Vision – ECCV 2016, pages 646–661. Springer International
  Publishing, 2016.
ISBN 978-3-319-46493-0.

[Huang et al.(2017)Huang, Liu, Van Der Maaten, and
  Weinberger]densenet
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.
Densely connected convolutional networks.
In IEEE conference on computer vision and pattern recognition
  (CVPR), pages 4700–4708, 2017.

[Ingalhalikar et al.(2014)Ingalhalikar, Smith, Parker, Satterthwaite,
  Elliott, Ruparel, Hakonarson, Gur, Gur, and Verma]ingalhalikar2014sex
Madhura Ingalhalikar, Alex Smith, Drew Parker, Theodore D Satterthwaite, Mark A
  Elliott, Kosha Ruparel, Hakon Hakonarson, Raquel E Gur, Ruben C Gur, and
  Ragini Verma.
Sex differences in the structural connectome of the human brain.
Proceedings of the National Academy of Sciences, 1110
  (2):0 823–828, 2014.

[Ioffe and Szegedy(2015)]ioffe2015batch
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
  internal covariate shift.
In International Conference on International Conference on
  Machine Learning (ICML), page 448–456, 2015.

[Janzamin et al.(2015a)Janzamin, Sedghi, and
  Anandkumar]janzamin2015generalization
Majid Janzamin, Hanie Sedghi, and Anima Anandkumar.
Generalization bounds for neural networks through tensor
  factorization.
CoRR, abs/1506.08473, 2015a.

[Janzamin et al.(2015b)Janzamin, Sedghi, and
  Anandkumar]non_convexity_nn_anandkumar
Majid Janzamin, Hanie Sedghi, and Anima Anandkumar.
Beating the perils of non-convexity: Guaranteed training of neural
  networks using tensor methods.
CoRR, 2015b.

[Johnson et al.(2012)Johnson, Fox, Sperling, and
  Klunk]johnson2012brain
Keith A Johnson, Nick C Fox, Reisa A Sperling, and William E Klunk.
Brain imaging in alzheimer disease.
Cold Spring Harbor perspectives in medicine, 20
  (4):0 a006213, 2012.

[Kim et al.(2016)Kim, Park, Yoo, Choi, Yang, and
  Shin]yong2015compression
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun
  Shin.
Compression of deep convolutional neural networks for fast and low
  power mobile applications.
In ICLR, 2016.

[Kolda and Bader(2009)]kolda2009tensor
Tamara G. Kolda and Brett W. Bader.
Tensor decompositions and applications.
SIAM REVIEW, 510 (3):0 455–500, 2009.

[Kolenic et al.(2018)Kolenic, Franke, Hlinka, Matejka, Capkova,
  Pausova, Uher, Alda, Spaniel, and Hajek]kolenic2018obesity
Marian Kolenic, Katja Franke, Jaroslav Hlinka, Martin Matejka, Jana Capkova,
  Zdenka Pausova, Rudolf Uher, Martin Alda, Filip Spaniel, and Tomas Hajek.
Obesity, dyslipidemia and brain age in first-episode psychosis.
Journal of psychiatric research, 99:0 151–158, 2018.

[Kossaifi et al.(2017)Kossaifi, Khanna, Lipton, Furlanello, and
  Anandkumar]TCL
Jean Kossaifi, Aran Khanna, Zachary Lipton, Tommaso Furlanello, and Anima
  Anandkumar.
Tensor contraction layers for parsimonious deep nets.
In The IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR) Workshops, July 2017.

[Kossaifi et al.(2019)Kossaifi, Panagakis, Anandkumar, and
  Pantic]tensorly
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic.
Tensorly: Tensor learning in python.
Journal of Machine Learning Research, 200
  (26):0 1–6, 2019.

[Lebedev et al.(2015)Lebedev, Ganin, Rakhuba, Oseledets, and
  Lempitsky]lebedev2014speeding
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S.
  Lempitsky.
Speeding-up convolutional neural networks using fine-tuned
  cp-decomposition.
ICLR, 2015.

[Mills and Tamnes(2014)]mills2014methods
Kathryn L Mills and Christian K Tamnes.
Methods and considerations for longitudinal structural brain imaging
  analysis across development.
Developmental cognitive neuroscience, 9:0 172–190,
  2014.

[Ning et al.(2018)Ning, Zhao, Matloff, Sun, and
  Toga]ning2018association
Kaida Ning, Lu Zhao, Will Matloff, Fengzhu Sun, and Arthur Toga.
Association of brain age with smoking, alcohol consumption, and
  genetic variants.
bioRxiv, page 469924, 2018.

[Novikov et al.(2015)Novikov, Podoprikhin, Osokin, and
  Vetrov]novikov2015tensorizing
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov.
Tensorizing neural networks.
In Proceedings of the 28th International Conference on Neural
  Information Processing Systems, NIPS'15, pages 442–450, 2015.

[Papalexakis et al.(2016)Papalexakis, Faloutsos, and
  Sidiropoulos]tensor_mining_papalexakis
Evangelos E. Papalexakis, Christos Faloutsos, and Nicholas D. Sidiropoulos.
Tensors for data mining and data fusion: Models, applications, and
  scalable algorithms.
ACM Trans. Intell. Syst. Technol., 80 (2):0
  16:1–16:44, October 2016.

[Paszke et al.(2017)Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin,
  Desmaison, Antiga, and Lerer]paszke2017automatic
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary
  DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.
Automatic differentiation in PyTorch.
In NeurIPS Autodiff Workshop, 2017.

[Rabusseau and Kadri(2016)]rabusseau2016low
Guillaume Rabusseau and Hachem Kadri.
Low-rank regression with tensor responses.
In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett,
  editors, Advances in Neural Information Processing Systems 29, pages
  1867–1875. Curran Associates, Inc., 2016.

[Raji et al.(2010)Raji, Ho, Parikshak, Becker, Lopez, Kuller, Hua,
  Leow, Toga, and Thompson]raji2010brain
Cyrus A Raji, April J Ho, Neelroop N Parikshak, James T Becker, Oscar L Lopez,
  Lewis H Kuller, Xue Hua, Alex D Leow, Arthur W Toga, and Paul M Thompson.
Brain structure and obesity.
Human brain mapping, 310 (3):0 353–364, 2010.

[Salimans and Kingma(2016)]salimans2016weight
Tim Salimans and Durk P Kingma.
Weight normalization: A simple reparameterization to accelerate
  training of deep neural networks.
In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett,
  editors, Advances in Neural Information Processing Systems 29, pages
  901–909. Curran Associates, Inc., 2016.

[Sedghi and Anandkumar(2016)]sedghi2016training
Hanie Sedghi and Anima Anandkumar.
Training input-output recurrent neural networks through spectral
  methods.
CoRR, abs/1603.00954, 2016.

[Sharir and Shashua(2018)]sharir2017expressive
Or Sharir and Amnon Shashua.
On the expressive power of overlapping architectures of deep
  learning.
In International Conference on Learning Representations
  (ICLR), 2018.
URL <https://openreview.net/forum?id=HkNGsseC->.

[Shi et al.(2016)Shi, Niranjan, Anandkumar, and Cecka]shi2016tensor
Y. Shi, U. N. Niranjan, A. Anandkumar, and C. Cecka.
Tensor contractions with extended blas kernels on cpu and gpu.
In 2016 IEEE 23rd International Conference on High Performance
  Computing (HiPC), pages 193–202, Dec 2016.

[Sidiropoulos et al.(2017)Sidiropoulos, De Lathauwer, Fu,
  Huang, Papalexakis, and Faloutsos]tensor_decomposition_sidiropoulos
N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E.
  Papalexakis, and C. Faloutsos.
Tensor decomposition for signal processing and machine learning.
IEEE Transactions on Signal Processing, 650
  (13):0 3551–3582, 2017.

[Simonyan and Zisserman(2015)]vgg
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
In International Conference on Learning Representations
  (ICLR), 2015.

[Sporns et al.(2005)Sporns, Tononi, and Kötter]sporns2005human
Olaf Sporns, Giulio Tononi, and Rolf Kötter.
The human connectome: a structural description of the human brain.
PLoS computational biology, 10 (4), 2005.

[Sudlow et al.(2015)Sudlow, Gallacher, Allen, Beral, Burton, Danesh,
  Downey, Elliott, Green, Landray, et al.]sudlow2015uk
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John
  Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, et al.
Uk biobank: an open access resource for identifying the causes of a
  wide range of complex diseases of middle and old age.
PLoS medicine, 120 (3):0 e1001779, 2015.

[Tai et al.(2015)Tai, Xiao, Wang, and E]tai2015convolutional
Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E.
Convolutional neural networks with low-rank regularization.
CoRR, abs/1511.06067, 2015.

[Tibshirani and Taylor(2011)]tibshirani2011solution
Ryan J. Tibshirani and Jonathan Taylor.
The solution path of the generalized lasso.
Ann. Statist., 390 (3):0 1335–1371, 06 2011.
10.1214/11-AOS878.
URL <https://doi.org/10.1214/11-AOS878>.

[Vakli et al.(2020)Vakli, Deák-Meszlényi, Auer, and
  Vidnyánszky]vakli2020predicting
Pál Vakli, Regina J Deák-Meszlényi, Tibor Auer, and Zoltán
  Vidnyánszky.
Predicting body mass index from structural mri brain images using a
  deep convolutional neural network.
Frontiers in Neuroinformatics, 14:0 10, 2020.

[Vasilescu and Terzopoulos(2002)]tensor_faces
M. A. O. Vasilescu and Demetri Terzopoulos.
Multilinear analysis of image ensembles: Tensorfaces.
In Proceedings of the 7th European Conference on Computer
  Vision-Part I, ECCV '02, pages 447–460, London, UK, UK, 2002.
  Springer-Verlag.

[Yang and Hospedales(2017)]yang2016deep
Yongxin Yang and Timothy M. Hospedales.
Deep multi-task representation learning: A tensor factorisation
  approach.
ICLR, 2017.

[Yu and Liu(2016)]yu2016learning
Rose Yu and Yan Liu.
Learning from multiway data: Simple and efficient tensor regression.
In International Conference on Machine Learning (ICML),
  volume 48, pages 373–381, New York, New York, USA, 20–22 Jun 2016. PMLR.

[Zhou et al.(2013)Zhou, Li, and Zhu]zhou2013tensor
Hua Zhou, Lexin Li, and Hongtu Zhu.
Tensor regression with applications in neuroimaging data analysis.
Journal of the American Statistical Association, 1080
  (502):0 540–552, 2013.


 

