RMS value calculation of 3 phase line voltages for power calculations
You're not doing an RMS transformation. You're converting L-N (RMS) to L-L (RMS).
You can use either but you have to remember that the phase to phase voltage is \$ \sqrt{3} \$ times the phase to neutral voltage.

Figure 1. The phasor 3-phase and neutral diagram.
The \$ \sqrt{3} \$ term just comes from the trigonometric relationship between the voltages in Figure 1. (Remember that the \$sin(60) = \frac {\sqrt 3} 2 \$.)