What Determines the Noise Bandwidth of a DC Voltmeter?
If you want to know what the bandwidth is, it should be in the datasheet, if its not then it might be hard to find it. One thing you might be able to do is limit the bandwith yourself. 

Measuring noise bandwidth is not going to work with a DC volt meter, its measuring near 0Hz (whatever the update rate is). RMS values measure noise across many frequencies and use a bandpass filter. 

There are two problems of measuring resistor thermal noise:
1) Other noise sources are going to make it difficult, meters, cables and amplifiers are all noise sources and need to be accounted for when measuring noise
2)  The bandwidth of the noise and sampling of the noise needs to be accounted for because we cannot measure the distribution of white noise directly (that would involve measuring across all times to infinity, but we can approximate it). 



The noise of a resistor looks like this:

$$ \frac{v_n^2}{\Delta f} = 4k_BTR$$

The voltage source is modeled as the noise power and that power changes depending on how much noise sources frequency you sample, like you did in the oscilloscope example. 

No matter the instrument, you will need to know the bandpass of the noise or you will not be able to determine the noise power.

It might also be worth noting that you can ground out the inputs and measure the baseline noise (of the pre-amplifier or meter) if your using them and find out how much noise you have from the system. Then connect the resistor that you wish to measure, the RMS noise from the resistor and the system will add by the sum of the squares. 

If you want to measure noise with a DC voltmeter you would need a pre amplifier and a way to convert the total noise power to an analog measurement at DC. There are circuits you could use to do this, but most of them would be hard to build yourself, and probably not worth your time. Some manufactures make IC's such as this one(AD737) 
that can calculate the RMS power, then run that value into the DC Meter.

If the meter has an RMS (AC) button, then just use the AC button after the amplifier and use a preamplifier, the bandwidth of the RMS measurment must be known to calculate the noise power from the resistor.

If your just using a DC meter, your limited on what noise you can measure based on the accuracy of the meter.


Here are some other good resources:
Noise sampling
Low level measurements Handbook