Light flickers when turned off and power disconnected
This is serious business. This is a parallel arcing fault, meaning it is not arcing across a gap, but arcing between a hot conductor and an appliance conductor, effectively bypassing the switch.  

I would expect to see/feel the damaged area, because it will be warm to the touch.

It is a hazardous situation because it is quite likely to start a fire.  To avoid these, try using an AFCI breaker in the panel.