How to calculate RMS of total noise from digital Oscilloscope (FFT)?
If you have a single frequency spectral point at say 2kHz and it is -55 dBu then it has a voltage level of 0.001377449 Vrms (see this for the conversion). If you also had another spectral point at 2.1kHz and it was -60 dBu (0.000774597 Vrms), then you'd calculate the RMS value of these two points by this: -

RMS Value of A and B together = \$\sqrt{A^2 + B^2}\$

You'd get a value of 0.001580305. That's two points.

Now extend this method to all the points within the spectral band of interest. If it means summing 10,000 points then taking the sq root of this sum, then that's what you have to do.