Switch two USB ports via relay
There is some room for improvement in your circuit.

First of all, as suggested in the comments you can tie all grounds together and get rid of one of the switches.

You also probably want to disconnect and connect pins in a given order, usually power is the first to come and the last to leave, this is enforced in (apparently not all) USB connectors by making gnd and Vdd pins longer.

I would go with a small P mos for the Vdd rail, something in the 10V Vsd 1V Vth 5V Vsg max range. Drain on the slave, and you need two of them of course.

For the diff pair the story is somewhat different. You need a passgate, it must be fast, and so on. But luckily enough somebody had your need before you, search for 'usb switch ic' and you will get many ready made chips that do just what you need, and are designed for that.

To get the power sequence right either you throw a small micro on it (nice side project - yay!) or you can probably get away with RC delays and a bit of luck.