Arduino analogReference() switching during run time
Changing the reference voltage will affect the ADC readings until the internal ADC capacitor has charged/discharged. Depending on how long that takes to charge/discharge is how long it will take until you can get accurate readings. I would try increasing the delay time more and see if that helps, or see how many readings you have to take before it stabilizes. Here's a link to Arduinos analogReference() page.
https://www.arduino.cc/reference/en/language/functions/analog-io/analogreference/