the activation function : The vectors and of output values can be expressed as a mode-1 tensors, while the hidden weights can be expressed as a mode-2 tensor. In this example the unit values are scalars while the tensor takes on the dimensions of the network layers: In this notation, the output values can be computed as a tensor product of the input and weight tensors: which computes the sum-product as a tensor multiplication (similar to matrix multiplication). This formulation of tensors enables the entire layer of a fully connected network to be efficiently computed by mapping the units and weights to tensors. In convolutional layers A different reformulation of neural networks allows tensors to express the convolution layers of a neural network. A convolutional layer has multiple inputs, each of which is a spatial structure such as an image or volume. The inputs are convolved by filtering before being passed to the next layer. A typical use is to perform feature detection or isolation in image recognition. Convolution is often computed as the multiplication of an input signal with a filter kernel . In two dimensions we have where is the width of the kernel. This definition can be rephrased as a matrix-vector product in terms of tensors that express the kernel, data and inverse transform of the kernel. where and are the inverse transform, data and kernel. The derivation is more complex when the filtering kernel also includes a non-linear activation function such as sigmoid or ReLU. The hidden weights of the convolution layer are the parameters to the filter. These can be reduced with a pooling layer which reduces the resolution (size) of the data, and can also be expressed as a tensor operation. Design of tensor frameworks The design of tensor frameworks has enabled machine learning modelers to build neural networks without knowing the details of tensors or backpropagation. With the development of unified frameworks such as PyTorch and TensorFlow, the construction of neural networks from the bottom up, including backpropagation and optimization, are not typically undertaken by most modelers. These frameworks support automatic differentiation, which computes the gradients of the learning loss function through multiple layers of a complex tensor network. The construction of tensor frameworks requires a deeper understanding of this optimization process in neural network learning. These topics are relevant to framework developers but are not strictly necessary for network modelers to understand