Johnson Noise Measurement
Johnson noise is the white noise of resistors. 

1) You can measure the integrated noise voltage: sample the voltage with a digital multimeter and process the statistics. The standard deviation will be the integrated voltage noise. You might need to know the bandwidth of the multimeter and its own noise if the noise levels are very low.
1) b) use an oscilloscope and check the noise in the waveform. They can calculate the RMS noise as well. Though the resolution of an oscilloscope is inferior to a multimeter, but it is much faster.

2) spectrum analyser, i.e. bandpass filtering many small parts of the spectrum and measure the energy in there. Basically it is the same method you described as well. If you built it for yourself you need to know the noise added by your components and calibrate them out from your measurement.

+1: noise is messy. It is hard to distinguish where the deviations are coming from. They can be radio stations (I've debugged some strange weak waveforms in a circuit of mine for more than an hour until finally I checked its FFT and it became clear that it picked up FM radio), wifi, bluetooth, etc. or SMPS in nearby computers or instruments...
Therefore you need to shield your DUT and its connection to your instruments as much as you can.