Does voltage in my PC's case indicate a circuit grounding problem?
I concur with your theory that the case is under power.  This may be due to a ground fault in the PC's power supply.  It could also be some other appliance raising ground due to its ground fault combining with a gap in the grounding system.  

It's possible to have "islanded grounds" where a bunch of equipment grounds connect to each other, but do not connect to the actual earth. In an islanded-ground situation, you have the lowest common denominator; a ground fault on any of them lights up all of them. 

The Equipment Safety Grounding System (outlet to panel), combined with the Grounding Electrode System (panel to ground rods), should be handling that properly.  I'd say it's time to give both systems a heavy inspection with a 10 foot pole... sorry, I mean with a fine-tooth comb. 

If your grounding electrode system is faulty, that's where the 10-foot poles come in.