Photodiode operational amplifier's strange temperature behaviour
It seems there are possibly a few interactions going on.

Unbalanced source impedance to gnd, causing poor CMRR to noise
the Pd has an open circuit voltage Vo of 320mV that is temperature dependent -3mV/‘C.
The PD capacitance reduces with Vr negative bias 80pF at V=0 and 50pF at V=-3V

Suggestion

Reverse bias the PD to &gt;=5V and use your differential trans-impedance amplifier on a current shunt to provide some gain. Excessive gain will require RC matching feedback to source parasitic and cable capacitance.
Using 1nA dark current at room temp, choose your shunt R based on some Noise spec.
Try to balance your PD stray capacitance to ground on each pin.
Choose your interface cable short for low capacitance but also twisted for CMRR.