What is the correct way to measure DC power at the output of a boost converter in a Simulink simulation?
The method I used to calculate the average value of the power was to
  take the instantaneous value (just the multiplication of V and I)


That's how digital power meters do it and then average the result using a low pass filter for instance. This works for both AC and DC circuits providing harmonic distortion does not cause aliasing artefacts.

However, it is likely that the DC voltage will have very little ripple current so you can make a fair-to-reasonable estimate of average power by multiplying voltage by low-pass filtered current. 

RMS of voltage or RMS of current are irrelevant to measuring power unless pure sine waves are involved and you know the power factor BUT this technique applies to AC circuits and is pointless for DC circuits.