Build own custom USB 'regulator' adapter board to solve Desktop boot hangs with KVM
This problem will not be fixed with "a few resistors and capacitors to stabilize the power". The problem is that the KVM does not respond correctly to the initial BIOS handshakes between the computer and the KVM. The most obvious fix is to try some other KVM. But if you are set on building something then you have to simulate what does appear to work!

You have observed that the computer boots fine with the KVM disconnected. And then I have to presume that the KVM works fine when subsequently plugged in once the system has booted. So what you would have to come up with is a device that delays the USB connection of the KVM. This delay device could be something that follows this general diagram:



This would break into the USB cable and route all four wires through the NO (normally open) contacts of a 4-pole signal relay. The relay coil would be driven from some buffered drive signal that activated the coil once the target computer was booted up. The nature of that circuit would be something you would have to devise since I am not privy to your computer setup. Most likely you would want to power the relay driver circuit via 5V extracted from the another USB port coming out of that same computer so that when the computer is off the the relay circuit shuts itself off. 

One possible solution for the relay driver may be to use an Arduino board that connects by USB to the computer. When the Arduino board powered up and the computer had finished boot you could have the computer startup script send a command to the Arduino to switch the relay.