SinePWM - Modulation Index calculation for a 3phase igbt inverter with ac load
The modulation index is something you impose, yourself. What you're calculating is the maximum (peak) value of the sine for a 100% modulation. For example, if  m=0.9 (90% modulation) then for your Udc = 650 V the peak sine would be m*650 = 585 V. You have a 3-phase application, so the phase to phase (LL) RMS value will be m*650/sqrt(2) = 413.7 V, and for the phase to ground (LN) the RMS will be m*650/sqrt(6) = 238.8V.