How can I prevent cold air from dropping down my bath fan duct?
I am going to answer this even though i should probably wait for more information.
Bathroom, and dryer vent, outlet caps that exit a wall are generally attached to the exterior wall with screws, one at each corner. There may be some silicone caulking as well. They have a male crimped pipe that fits into the duct coming from the source.
You should be able to remove the screws and pull it directly out, turn 180 degrees and re-attach.
Here is YouTube video of one with a spring loaded flapper.

3&quot; or 4&quot; Round Duct Black Wall Cap w/ Damper, supplyhouse.com