Complete ADC SNR formula when signal is not full scale
You're overthinking this making things (in your mind) more complex than they are.
Suppose we have a 10-bit ADC, then the will be roughly 60 dB (yes, theoretically 61.96 dB but I like round numbers, also: feel free to show me an ADC that works exactly according to theory, but I digress). So SNR = 60 dB
Let's say that this ADC has a full scale of 0 dBV, that's 0 dB Volts and in my world that's an amplitude of 1 Vpeak.
So then the noise level will be 60 dB below that 0 dBV so noise = -60 dBV (that's 1 mVp).
Now if we feed the ADC a smaller signal of for example 100 mVp = -20 dBV, the noise will still be at -60 dBV. So that means the SNR (for that smaller signal) will be -20 dBV - -60 dBV = 40 dB. The SNR became 20 dB smaller, which makes sense as the signal was made 20 dB smaller.
Does that match with your second formula?

SNR(V) =? 20log(V/Vfs) + 6.02N +1.76 (dB)

Yep, I think it does.
This isn't written in articles because it is (in my opinion) quite obvious.
An AGC would complicate things but not much, the AGC would simply add gain and increase the amplitude of the signal (not the noise). But an AGC will affect the dynamic range of your signal (it will be lowered), often that can harm your signal.