Precision Current Sink for LED Modulation
The LM324 datasheet specifies the output voltage max as Vsupply - 1.5V, so given your 5V power supply, 3.5V out (max) is exactly what you would expect.  The LM324 should be able to drive the MOSFET at low frequencies, although the gate is a capacitive load so stability can be an issue.

You will do better using a rail-to-rail output opamp, although even those don't get all the way to the 5V output in your circuit.