Why should each filter have different weights for each input channel? From the answers to this question In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?, I got the fact that each filter has different weights for each input channel. But why should that be the case? What if we apply the same weights to each input channel? Does it work or not?
For simplicitly, let's consider only the first convolutional layer, that is, the one applied to the image. If you consider an RGB image, then there are $3$ channels: the red channel, the green channel and the blue channel. Thus, a kernel that is applied to this image will also have $3$ channels: the red channel, the green channel and the blue channel. In general, the distributions of the intensity of the red, green and blue colors in the image are different, so, in general, the red, green and blue channels of the kernel will also be different because they need to keep track of different information.