Using BILATERAL SWITCH properly?
You should be able to do it the way you propose, but you can also get away with just one IC:



I believe this is also the method @Mishony mentions in the comments (just noticed it as I am posting this).

Both inputs would be pulled high by resistor (e.g. 10k - 100k). In this state all switches would be closed and your FC and OSD are connected TX to RX.

When you pull down either X or Y it will disconnect one or other of the devices. For example if Y is low then the FC is disconnected. In this state you can then plug on a debug device or programming device or something like to configure the OSD. When you pull X low the OSD is disconnected and you can now configure the FC.

Just remember to reverse the TX and RX signals when swapping between which device you are debugging. It shouldn't matter too much if you forget because the switch has typically around \$470\Omega\$ on resistance for a 5V supply. This should mean if you connect TX to TX the switch itself will protect your devices from shorting out the TX pins. The current flowing as a result of shorting them out would be limited to about \$10\mathrm{mA}\$ which is within the operating limits of the switch. Best to avoid doing so, but shouldn't hurt.

Also that on resistance won't be an issue. I frequently use 1k resistors between two UART devices at speeds of 115200 Baud without issue. It's a useful thing to have as I say because it avoids the risk of damage from TX/RX reversal.



Another option which actually wouldn't take up too much space would be something like this:



You put jumpers over the connectors when in flight mode, and then when you need to configure it you can simply connect on your debug to the corresponding pins. You can move one of the ground pins to the other side to make the connector for the debug rotationally symmetrical.