Bathroom Vent is tied to ceiling vent in adjacent room
Yikes, this is really the first time I have ever heard of this configuration. Could this have been part of a previous central forced hot air system? It is not common to tie exhaust vents from a bathroom to any other living space. A dedicated exhaust fan vented to outdoors with a one way damper is a very important way to remove excess moisture from bathrooms and help prevent mold and mildew growth. It is not acceptable to vent a bathroom to another living space or into an attic. It is equally not right to tie this type of vent to any other air plenum. If you are feeling cold outside air through these vents, you are getting huge heat losses. There is no need to introduce outdoor air into your house intentionally unless it is done with specifically designed air exchange equipment in an extremely airtight designed house. I think you need to determine the routing of these "phantom" vents and come up with a plan to isolate the bath vents to the out of doors with a proper bathroom fan unit and block your heat form other areas from being wasted out an open plenum.