Understanding voltage
When you state:


  if voltage is the amount of work needed to move a charge from A -> B not the actual the amount of work being spent across A -> B but the work needed [if] we were to move a charge from A -> B


"Amount of work" is referring to energy. As you said on your post, Voltage is not an energy, energy would be described as Wh which is voltage * current * time.

I feel your confusion, and the best way to understand voltage, current and such is the parallel with the water. It behaves very similarly.

Think of a waterfall:
- The height of the waterfall is the voltage. Higher is the height, more force the water will get.
- The rate of flow of the water is the current.
- The power is the rate of water * the height.
- The energy is power * time.

Now you can think of a waterfall without any water going through, that is just basically a lake atop a mountain. This is potential energy.



To your question:


  In this situation, if it takes V - 1 volts to move a charge from B ->
  C, but we are not moving any charge since there is no current, is the
  input 'B' still damaged?
  The Simple answer is Yes, because it is the "pressure" that will damage your component and as the metaphor above, you can burst a pipe by increasing the pressure without having the water (electron) to actually move.


You can also think of electron as balls inside a tube. 

You can push the first ball harder and harder until the lead pops (your device breaks). Until the lead pops, the pressure would have increased but nothing actually moved.