Oversampling while maintaining noise PSD I'm designing a digital PID for control and stabilization of laser amplitude. The noise PSD of the laser intensity is around -140dB/Hz, and I want to servo that between DC and 100kHz. I think I've found a solution on the input side, but would like to make sure my understanding of noise spectral density vs SNR is correct.

I'm thinking about using an ADC at 5MSPS with a SNR of 100dB. My understanding is that this means that the power spectral density for a full-scale signal will have a noise floor of -163 dBFS/Hz, calculated form the SNR and the bandwidth. Since I only want to servo to 100kHz, I can use a digital FIR filter to maintain that noise floor so that I have a healthy amount of SNR between the noise that I want to servo and the intrinsic noise of the ADC.

What I'm a bit confused about is what the FIR filter does to the noise spectrum. I'm pretty sure that the filter will simply attenuate everything past the passband, and maintain the 167 dBFS/Hz - the SNR increases, but the effective bandwidth decreases. Is this actually correct?
One important point you may be missing is assuming that the noise of the ADC is truly white over your sampling bandwidth- it isn't! You will have spurs from signal correlations and ADC non-linearities that will limit the noise floor you can achieve. This will be specified by the SFDR (spurious-free-dynamic-range) of the ADC for guidance as to what you may be able to achieve. Since the location of the spurs will be difficult to predict, you should assume that the strongest spur could be within your bandwidth of interest, in which case no linear filtering will reduce the power of the spur which exists at a single point in frequency.

That said, the filter will reduce white noise components, and the best metric is the EQNBW (equivalent noise bandwidth) for your filter; which is the equivalent bandwidth of a brickwall filter that would remove the same amount of noise. In this case the SNR improvement (processing gain) is 10Log10(2EQNBW/fs) assuming a low pass filter where EQNBW is the bandwidth from DC to the equivalent cutoff frequency.