Switching Serial Connections among several devices?
In the general case, you need a 2:1 multiplexer in front of each RX input.

If you know the unconnected device will not be lowering its TX output (I'm assuming the line idle level is high), then just a 2-input AND gate in front of each RX input will do it.

Or, you can use the AND gate trick with a OR gate on each TX output.  That explicitly masks off the unused device by forcing the OR output high when that device is not in use.  The extra input to each OR gate enables that device when low and disables it when high.