LM324N transimpedance amp for photodiode: Too low output, smaller feedback resistor gives greater output, not lower
For Vout to successfully drag \$V_{IN-}\$ to equal \$V_{IN+}\$ (at 0 volts) it has to sink current from the photodiode (via \$R_F\$) towards a negative voltage. That can't happen because the lowest negative rail on the LM324 is 0 volts hence the output cannot force the circuit into equilibrium.

The red box in the drawing show the problem - the LM324 needs to have a negative rail when the photodiode is connected as shown.
On the other hand, if you changed the pins on the photodiode, it should begin to work because then the op-amp has to source current into the photodiode via \$R_F\$ and all should be well. Now you will need the photodiode cathode connected to ground: -