Peltier with Mosfet - Heat Issues and Alternatives
You don't show the circuit, but it seems you are trying to linearly regulate the Peltier current.  That is going to cause significant dissipation in the pass element.

A more efficient way is to use PWM, although the pulses need to be filtered so that the Peltier sees the average.  I go into a lot more detail about this here.