Why is the kernel of a Convolutional layer a 4D-tensor and not a 3D one? I am doing my final degree project on Convolutional Networks and trying to understand the explanation shown in Deep Learning book by Ian Goodfellow et al.
When defining convolution for 2D images, the expression is:
$$S(i,j) = (K*I)(i,j) = \sum_{m}\sum_{n}I(i+m,j+n)K(m,n)$$
where $I$ is the image input (two dimensions for widht and height) and $K$ is the kernel.
Later it states that the input is usually a 3D tensor. This is because usually the input image has multiple channels (say, red, green and blue channels). Furthermore, it says that it can also be a 4D-tensor when the input is seen as a batch of images, where the last dimension represents a different example, but that they will omit this last dimension for the sake of simplicity.  I understand this.
Let now $I_{i,j,k}$ be the input element with row $j$, column $k$ (height and width) and channel $i$. The output of the convolution is a similarly-structured 3D-tensor $S_{i,j,k}$.
Then, the generalized convolution expression for this 3D-tensor input is
$$S_{i,j,k} = \sum_{m,n,l} I_{l,j+m-1,k+n-1}K_{i,l,m,n}$$
where &quot;the 4-D kernel tensor $K$ with element $K_{i,j,k,l}$ giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input, with an oﬀset of $k$ rows and $l$ columns between the output unit and the input unit&quot;.
I am completely lost in this definition. Why is the Kernel 4-D and not 3-D (which is a logical generalization of the first formula)? What is the analogous to the sentence &quot;giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input&quot; in the initial 2D Kernel tensor? I think that is the main things that has to be understood to understand the 4D-kernel.
The first formula you quote is for an image with one input channel and one output channel, it just focuses on height and width. In this case, if we consider a 5x5 convolution, the Kernel will just have size 5x5, $m$ and $n$ and going from -2 to +2.
Now if our input has 3 channels (RGB, but could be feature maps). we need to use each channel as an input, and the weights will be different for each input map. So K becomes 3-D : 3x5x5. This new dimension corresponds to the $l$ in your formula.
If we want to have 10 outputs, we need 10 different ways of convoluting the input feature maps, so our Kernel will have one more dimension, leading to a 10x3x5x5 size for the Kernel. This last dimension corresponds to the $i$ of your formula.
To recap, the 4 dimensions of the Kernel $i, l, m, n$ stand for :

$i$ : Output dimension
$l$ : Input dimension
$m, n$ : height and width

About this sentence &quot;giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input&quot;.
I feel it is wrong as I would rather interpret it as : &quot;giving the connection strength between a unit in channel $i$ of the output and a unit in channel $l$ of the input&quot;