What's the difference between calculating the FFT of a constant sound for 5 seconds versus every 1 second and taking the average?
I would say that you answered part of your own question by noting the reduction of variance with averaging.

Most books tend to present signals that are either random or deterministic but  signals can be somewhat of both character to varying extent.

If your signal tends towards the deterministic, and high SNR, there isn’t a lot variation to reduce so the higher resolution of the longer FFT would probably be more preferable in most applications.

If you have a low SNR tone, you want to do as much averaging as other considerations would permit to reduce that noise.

One can also overresolve with long FFTs.  Pure steady tones are rare. Real tones have some bandwidth and you want your FFT bins to match that bandwidth.

Averaging has some other advantages like reducing the down stream data rates like if you are displaying real time STFT displays. 

Depending on your application, there are other more specific tradeoffs between averaging and FFT length.