Arduino ADC Reference Voltage if it is Battery Powered
... won't the ADC reference voltage constantly be dropping with the battery?


Yes, which is why you either use or measure an internal bandgap reference instead.

Use the analogReference() function to select a reference appropriate for the board in use. Note that you will need to use a voltage divider to reduce the battery voltage to a value below that of the selected reference if you wish to measure it.

To measure the bandgap voltage instead (using AVCC as a reference and working "backwards") you will need to set MUX[3:0] in ADMUX to 0b1110 and then perform an ADC reading directly (set ADSC in ADCSRA and wait until it resets, then read from ADC[H:L]).

As always, see the MCU datasheet for details.