Is 4" Ducting Used for a 4" Bath Fan?
If your ductwork is in an uncontrolled environment, you truly should insulate it. This for the reasons you list - to help prevent vapor from becoming water prematurely.

While it is not necessarily a lot, any is bad since gravity pulls it right back down the pipe into the ceiling, or the fan, etc.

If your air column is smooth, the work necessary to move it is less. The goal is to get the water out, not let it stay in. If the fan is properly sized for the room and there arent many obstructions (rigid tubing, 90 degree elbows, or even wind blowing a flapper shut...) you have a recipe for success.

The insulation is just 1 more defense in avoidancehelping to promote the vapor escaping.

This site has quite a bit on it:
info on others experiences with venting