Transformer Open-Circuit & Short-Circuit Tests
but I am unsure if I should use the rms or peak values for the
measured voltage &amp; current in the calculations of the losses

To measure power properly you cannot (or should not) compute power by measuring voltage and current independently (RMS or otherwise). You should compute power as instantaneous voltage multiplied by instantaneous current and take the average of that waveform. Examples: -

All easily done in most simulators.