when using common tensor frameworks. Backpropagation through tensors The backpropagation algorithm computes the gradients of a loss function through multiple layers of a neural network. The loss function is typically defined as the error between the input data and network output over a training set. To develop tensor-based frameworks, it is necessary to define the backpropagation step over tensors as well. Since tensors are the element-wise extension of vectors and matrices to multiple dimensions, derivatives can also be performed element-wise over tensors. The tensor object defines the partial derivatives over inputs, outputs and weights. Tensor objects, like scalars, can be chained through repeated application of the chain rule. Tensor factorization An important contribution of tensors in machine learning is the ability to factorize tensors to reduce the number of elements or parameters to be learned. This is a form of data reduction which preserves the meaning of the data by making use of the rank of the tensor to eliminate redundancies in the data. There are several different types of decomposition. Tucker decomposition Tucker decomposition takes a tensor, for example a 3-mode tensor and decomposes the tensor into three matrices and a smaller tensor . The shape of the matrices and new tensor are such that the total number of elements is reduced. The new tensors have shapes Then the original tensor can be expressed as the tensor product of these four tensors: In the example shown in the figure, the dimensions of the tensors are : I=8, J=6, K=3, : I=8, P=5, : J=6, Q=4, : K=3, R=2, : P=5, Q=4, R=2. The total number of elements in the Tucker factorization is The number of elements in the original is 144, resulting in a data reduction from 144 down to 110 elements, a reduction of 23% in parameters or data size. For much larger initial tensors, and depending on the rank (redundancy) of the tensor, the gains can be more significant. The work of Rabanser et al. provides an introduction to tensors with more details on the extension of Tucker decomposition to N-dimensions beyond the mode-3 example given here. Tensor trains Another technique for decomposing tensors rewrites the initial tensor as a sequence (train) of smaller sized tensors. A tensor-train (TT) is a sequence of tensors of reduced rank, called canonical factors. The original tensor can be expressed as the sum-product of the sequence. Developed in 2011 by Ivan