How to turn off a higher supply from a lower one?
First create a truth table. With two switches there are four combinations but you have only specified three.

A    B    Board
-----------------------
0    0    not specified
0    1    12 V
1    0     5 V
1    1     5 V


Then it becomes easy to work out the switch logic.







simulate this circuit &ndash; Schematic created using CircuitLab

Figure 1. Electrical schematic from the switch logic. Note that this implementation cuts feed to the board when both switches are off.

Regarding tapping off one battery: as I mentioned in the comments, tapping off one battery is generally a bad idea because you now have uneven discharge.