What is the point of tensors in CNNs? Why not simply reshape the data into matrices? Take the following tensor:
$$
\left[\begin{array}{cc}
a &amp; b &amp; c\\
d &amp; e &amp; f\\
g &amp; h &amp; i\\
\end{array}\right] 
$$
$$
\left[\begin{array}{cc}
j &amp; k &amp; l\\
o &amp; n &amp; m\\
p &amp; q &amp; r\\
\end{array}\right]
$$

Where each matrix represents a channel.

This could be reshaped fairly easily into a vector:
$$
[a,d,g,b,e,h,c,f,i,j,o,p,k,n,q,l,m,r]
$$

And then concatenated row-wise with other vectorized tensors to form a typical flat-file dataset of dimension $N \times P$, where $N$ is the number of training samples and $P$ is the product of the tensor dimensions.

Rather than futzing with a convolutional layer, one could simply constrain ones weights to be zero in the subsequent layer.  

If $X$ is a flat $N\times P$ dataset of concatenated vectorized tensors, then the convolutional weights would form a sparse matrix, with the first two columns of a $P \times 4$ convolutional "layer" being $2\times 2\times 2$ filter being
$$
\left[\begin{array}{c}
w 0\\ w0\\ 00\\ ww\\ ww\\ 00\\ 0w\\0w\\00\\w0\\ w0\\00\\ww\\ww\\0w\\0w\\0w\\00
\end{array}\right]
$$

This seems to me more intuitive than the tensor formulation, and could be computed fairly simply using sparse matrix packages.  Perhaps it is partly matter of taste.  But I'm curious: is there anything special about the tensor paradigm -- either mathematically or computationally -- that is superior to the flattened representation?  I understand that computers convert matrix algebra to for-loops "under the hood", but doesn't the advent of the GPU make such explicit looping irrelevant?
Tensors come pretty natural in convolutionals networks.


Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This information gets lost when you reshape an image into a vector. Look how a convolutional layer works.





Training usually is done in batches, which is another dimension as far as a neural network is concerned. You don't want to mix different images into a matrix, they are totally independent.
Finally, there is depth channel. Initially it's one of R, G, B channels of the input image, and then each channel corresponds to a filter applied to the previous conv layer. Once again, filters are independent, it doesn't make sense to mix them up until the final layer.


So, in total there are 4-rank tensors going through the conv net. It's not only more intuitive (each dimension has a meaning), but also results in higher accuracy, because it employs all meaningful information from the images.