Spectral averaging without phase information
The short and most common answer is that you can average without phase but there are cases where you can average phase as well.

Cross correlation between
2 sigmals where one has a delayed component of the other that is of a "long" duration can exploit complex averaging.

Spectral analysis using the periodogram averages magnitude or magnitude squared.

There are numerous examples of where you do or don't use complex averaging.  It depends on the problem being solved and is typically going to be relatively obvious if you should or shouldn't.  If you expect the phase to evolve in a predictable way over your observations, you can use that to your advantage.