Inaccurate output voltage from the Analog Multiplexer
if the voltage drop of the first branch shunt resistor is 2.236mV the
output of the multiplexer is 2.399mV

The MAX337's input leakage current rapidly rises as the input closes-in on either supply rail. It can be as high a 1 μA (top of page 3) and, given that the on-resistance is around 400 Ω, that could account for ± 400 μV of extra input offset voltage you were not expecting to see.
Input (or port) leakage current needn't be regarded as flowing out of a pin; it can be regarded as flowing back through the multiplexer hence the volt-drop caused by its internal on-resistance. Clearly it can't obviously flow into the InAmp (except for reasons at the bottom of this answer) hence it has to flow somewhere and the only route is via the internal chip to the LED driver or the diodes.
You might also be approaching or exceeding the input voltage range of the InAmp too.