Why does my lamp only work when plugged the "right" way?
It must be a matter of stray leakage. 

Either intentionally (e.g., GFCI circuitry) or unintentionally (e.g., stray voltages removing bias from a node) the circuit stops working. 

Turning the plug around changes the relationship of any reference node to the stray capacitance to “the world” through the lamp’s body. This leads to more or less stray currents in different nodes in the circuitry. 

If the lamp has a Triac driving circuit for a dimmer, this stray current might be enough to cause this circuit to stop operating. 

Measure the AC voltage from the lamp housing to ground. You might be able to see a difference depending on the orientation of the plug.