How to measure uV voltages buried in noise?
anywhere from 10-10kHz.
Would I need to use a band pass?

Sounds like you at the very least would want to use a low-pass filter; that'd remove the noise power from above 10 kHz.

In noisy, low voltage situations like this, how do I go about measuring this signal?

You'd apply as much gain as necessary for your measurement device or ADC to measure the signal. There's not much you can do if you know nothing else about the signal! If your signal is unknown, and so is your noise, you simply can't tell them apart.
I do, however, get the feeling that you might know a lot more about the signal in question than you describe in your question. In that case, signal processing can help you tell signal from noise, and estimate what you need to estimate. As an example:
The millivolts you're getting are pretty much signal, when you consider that GPS receivers work with signals down to -160 dBm – that's 10⁻¹⁹ watts, or ca. 30 nV amplitude if we assume that the GPS receiver has an input resistance of 100 Ω. Most importantly, that's far below thermal noise floor over the same bandwidth! Still, GPS works reliably by employing processing gain based on knowledge of the signal structure.