Smooth 12V pwm signal running at 4 KHz
You probably want to use an LR, with RC you'll have quite some losses. Have a look at all those class D amplifier output stages, you should have no trouble with reversing as far as the filtering goes. Of course you'll need an H-bridge to reverse your peltier. You might want to look at some of the popular motor drivers.