Given a spectrogram, how do I calculate loudness over time?
Yes, in fact it's not a bad idea at all to use some kind of spectrogram representation to calculate a loudness envelope. Depending on what exactly you need you can either just sum the spectral magnitude over all frequencies at a given time, or first apply some kind of perception based weighting functions 
to make your loudness measure more related to actual auditory perception.

So if you give some more detail about the application you have in mind you will likely receive a more accurate answer.