Gain on an ADS1115 using differential input
For those who might face a similar problem in the future, after some digging I believe the answer to my issue is the follows:

When using the LED simply as originally described (Cathode to A1, Anode to A0), the voltage depends on the resistance between the pin, which in the ADS1115 is gain-dependent. This is why the gain alters the voltage and ADC values in a weird way.

Wiring the LED up as suggested by Andy aka solves the problem to a degree, but the resistance of the resistor between anode and ground in the reverse-biased LED must be quite high (~5MOhms in my case), and the change in resistance of the ADC at different gains can still impact the circuit and should be watched out for.

The weird noise, after careful timestamping of the measurements and Fourier analysis is revealed to simply by 50Hz mains noise bleeding through the RPi from its power supply. Still working on solving that issue, but as it has nothing to do with the ADC, considering this question answered.