How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps?
No, you don't average across all feature maps.
When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter &quot;covers&quot; the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.
Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

The same happens with feature maps with N channels: we need a filter with N channels.