Should zero element be considered to calculate mean when smoothing spectrogram by window?
You don't seem to exactly perform smoothing, but instead averaging by non-overlapping blocks of 3 and downsampling. A sound answer cannot be given without the  knowledge of what you want to do from this data simplification. 

However, using the zeros possesses the highest risk for me, as this biases the result  with values that actually are unknown. Using the non-zero values only is in fact assuming the following model (0-order): the expected value for unknown ones is the average of the known values: if you replace zeros by such a mean, you don't change the result.

Using higher-order models or additional assumptions, you might be able to get more interesting smoothing.