USB Receiver switch between two PCs
Unfortunately, yes, you missed something important.
USB uses fast digital signals. How fast depends on which version of USB (Low Speed, Full Speed, High Speed, Super speed) but they vary from very quite fast to ludicrously fast.
Fast signals need special care when laying out PCBs, or making cables.  This is a complicated subject, so I'm not going to try to explain it fully here, but you will need to use a carefully designed PCB to do something like this. Even if you can get that right, a basic mechanical switch could still cause problems. It will help if you can fool the devices into using USB low speed, as that will be more tolerant of imperfect cables/traces.
There are a couple of other things here which aren't great too, but which might not stop it working.  It looks like you're only switching the digital lines, not power and ground.  Differences in ground potential between the two host devices could cause problems, so it would be better to switch those too. Also there is supposed to be a strict order for connecting pins for USB: ground and power, then the digital lines.  Disconnection should be the reverse. Doing them in the wrong order has a chance of breaking things.
All in all, USB is a very tricky bus to work with. The good news is that there are some special ICs out there which will do what you want, but even then, laying out the PCB will be tricky.