how to let dc only go one way
I wouldn't attempt the diode solution. The commenters are correct in that wiring the two power supplies together like that is likely to cause problems; I've myself seen a PC that won't shut down properly if back-fed 5V through USB from a bad hub.

What I would suggest is connect both the grounds together, but connect only one of the power supplies. That obviously requires that PC to be on, so you can't use the gizmo if only the other one is on.

An alternative that doesn't have that limitation is to use an externally powered hub on the side after the switch, and connect NEITHER of the power lines but BOTH the ground lines.