Advices on building a USB switch
Try 







simulate this circuit &ndash; Schematic created using CircuitLab

The switches don't carry any signifigant current. Closing either of them turns off the output, at the cost of 50uA current from the source (trivial unless you're trying to run off a small battery for a long time).

I've not done the work of picking a suitable P-channel MOSFET. You should choose one with a low Rdson.