Does dividing the magnitude spectrum of white noise by sqrt(2) give an RMS magnitude spectrum?
However, an FFT of a noise signal indicates magnitude in bins at equally spaced frequencies, which means the magnitude represented in each bin, is the magnitude of a periodic signal (sinusoid).


No; it just means that when projecting the noise signal onto a sinusoidal signal, you don't get nothing.

Assuming your noise realization in each is independent identically distributed (usually the way we model noise!), then the (rotated) sum of such noise samples approximates a normal distribution, no matter how the (finite-variance) original distribution looked like. The magnitude in each bin is an approximation (because by definition, your DFT has finite length!) to the noise power density; in an uncorrelated noise process, that's the variance==power==rms².