How is it possible for voltage to exist when only a single voltage source terminal is connected?
You know what you have there?

A radio receiver.

The power supply isn't supplying power (as such), but the wires and things are acting like an antenna.

Look, same effect using just me and a diode:



It's kind of like the old crystal set.  No power source there - just radio waves:



Since there is no (intentional) tuning and filtering you are most likely seeing the most powerful source of EM waves in the locality - the ubiquitous 50 (or 60) Hz hum from the local mains system.