How much heat loss through bathroom fan vent?
You probably want to install some kind of a damper or louvers that will minimize air exchange through the vent when the fan is shut off, right? I would expect that the fan assembly itself might have a damper built into it, but maybe the housing is binding the damper. Or maybe it doesn't have one.

http://www.batticdoor.com/bathfandraftblocker.html

http://www.askthebuilder.com/bathroom-exhaust-fan-leaks-cold-air/