How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps? I understand that for the first layer (assuming we have a grayscale image) we calculate the convolution of 3*3 receptive field as a weighted sum of receptive weights with pixels
$ x1 · w1 + x2 · w2 + x3·w3 + ... + xn · wn$
But for the second layer, where we already have $N$ feature maps in the last convolution layer, how would we calculate the convolution(for a particular pixel/cell)? should we take an average of the $N$ weighted sums we have from feature maps?
If the question isn't clear, I have tried to highlighted it from the image taken from famous 3d visualization.

In the image, for the pixel in question(hovered and squared) we have inputs coming from 4 feature maps. I think they are four integers (weighted sums of the receptive field of the bottom right corner from each feature map).
How is the value (convolution)  for this particular pixel/cell calculated? showed we take average like below?
(I can add more details if the question is not clear)
weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 + weighted_sum_fmap1 / 4
No, you don't average across all feature maps.
When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter &quot;covers&quot; the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.
Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

The same happens with feature maps with N channels: we need a filter with N channels.