Confusion while diagnosing an open ground
The funny voltages you are seeing are due to the lamps in the 3-lamp tester. There is a hot-ground lamp (yellow) and a neutral-ground lamp.  They are in a 3-way tug of war with the voltmeter.  So wacky numbers like that are not unusual.  But it means the ground wire is floating (not attached to anything) so it is easily influenced by whatever it's attached to. 

When you removed the socket, you no longer needed the 3-lamp tester so the voltmeter won the tug of war, pulling the floating ground to whatever voltage you were testing to.   

The ground wire is simply not working.  This would be of serious concern if the circuit was not GFCI protected.  But it is.  So, shrug, there you go.