DYI USB Hub - It is possible to just connect ports in parallel?
A hub is an active device, wiring data lines in parallel will not work, 

Your idea of a conventional switch is likely to struggle but may still work for you. if it does you may be able to switch a normal hub.