Remove Y caps from PSU
The Y caps form a voltage divider between phase and neutral with the unloaded voltage at their center point being at Vmains/2.  Measurement with a voltmeter  will usually show a lower voltage due to meter loading.  (Try measuring Phase to midpoint and Neutral to midpoint. The result will not sum to Vmains/2).
The correct solution is to ground the Y resistor center point. As this is almost always connected to power supply ground this is usually achieved by using a 3 wire power cord with ground wire AND a power outlet with ground.
The problems usually occur when this ground connection is missing.
This is a safety hazard in its own right. Ungrounded outlets are more common in some countries than in others - but are less desirable than a grounded outlet for safety reasons.
The major task of the Y capacitors is to help remove 'noise' from the mains input - mainly from the power supply itself but they also filter incoming mains borne noise.
Removing the Y caps makes the power supply non-compliant to regulatory requirements and make it more liable to cause EM (electromagnetic) interference.
It may also make the device somewhat less resistant to mains borne EM interference. An X cap serves much the same purpose. The Y caps are balanced to ground IF ground is provided, providing a notionally superior result.
____________________________
Consequences:
The voltage mentioned is due to the Y caps acting as a voltage divider across P-N and there being no earth connection.

Worst case this can destroy products - often either a port or a peripheral connected to a port. I have seen a printer destroyed in this way.

Laptops will often give users minor but annoying shocks if touched if their psu has no ground connection and incoming 'ground' is DC connected or via  a capacitor to the DC output ground.