Why averaging in Spectral magnitude domain not in complex domain to estimate spectrum of a process
Averaging includes the introduction of errors, be it in the temporal OR in the frequency domain.

But let's address your specific concern. My opinion is that if the SNR is good, neglecting the phase distortion created by the noise does not introduce a big error. So IMHO averaging in time or in frequency (signal amplitude) will give good results in both cases if SNR is good, and bad results, also in both cases.