Cold bathroom that has exhaust
Exhaust ducts can make areas dramatically colder.

In a cold climate, there should be a vent flap near the fan that closes to prevent air flowing into the room - but these commonly only do a so-so job.

A better fix is installing a vent cap with flappers on the exterior of the duct. Because these have much more room to work with, and endure exterior conditions, they're much more effective and of better quality.

I'd probably do:
Step 1: check/clean exhaust fan and housing.
2: smoke test windows in the bathroom area.
3: since my washer/dryer are in bathroom, check that ducting too.
4: check exterior flaps