Root mean square bandwidth of a signal with flat spectrum
Your formula for the RMS bandwidth makes sense for (perfectly band-limited) low pass signals, i.e., for signals with a spectrum centered around $f_0=0$. The bandwidth of low pass signals is defined as the support of their spectrum at positive frequencies. So your integration limits must be $-B$ and $B$. This results in

$$B^2_{\rm rms}=\frac{\displaystyle\int_{-B}^{B}f^2df}{\displaystyle\int_{-B}^Bdf}=\frac{2B^3/3}{2B}=\frac{B^2}{3}\tag{1}\\$$

which is the expression that you're looking for.

Note that the RMS bandwidth is also defined for signals that are not ideally band-limited. In that case you have to integrate from $-\infty$ to $\infty$, as pointed out in MBaz's answer.

If you have a (not necessarily perfectly band-limited) band pass signal centered around $f_0\neq 0$, you must use the following formula for the RMS bandwidth:

$$B^2_{\rm rms}=\frac{4\displaystyle\int_{0}^{\infty}\lvert H(f)\rvert^2(f-f_0)^2df}{\displaystyle\int_{0}^{\infty}\lvert H(f)\rvert^2df}\tag{2}\\$$

If $\lvert H(f)\rvert$ is perfectly band-limited and if it is constant ($\lvert H(f)\rvert=c$) in the interval $[f_0-B/2,f_0+B/2]$, its RMS bandwidth is

$$B^2_{\rm rms}=\frac{4c^2\displaystyle\int_{f_0-B/2}^{f_0+B/2}(f-f_0)^2df}{c^2\displaystyle\int_{f_0-B/2}^{f_0+B/2}df}=\frac{4c^2\displaystyle\int_{-B/2}^{B/2}f^2df}{c^2B}=\frac{B^2}{3}\tag{3}\\$$

which is the same result as for the corresponding low pass signal.

Note the factor $4$ in Eq. $(2)$, which is caused by the fact that the RMS bandwidth (without the correction factor) measures the "width" (variance) of the function $|H(f)|^2$. For low pass signals this width is measured across the whole spectrum (positive and negative frequencies), whereas for band pass signals, this width is measured only at positive frequencies. If this width is the same for a low pass signal and for a band pass signal, then the bandwidth of the band pass signal must be twice the bandwidth of the corresponding low pass signal, because bandwidth is measured at positive frequencies only. This is the reason why the squared bandwidth in the case of a band pass signal needs a factor $4$ to comply with the standard definition of bandwidth for low pass and band pass signals.