Why does high voltage imply a high "pushing force" on electrons?
What exerts a force on charged particles is the electric field, which is proportional to the gradient of the potential. 

So when the voltage (or potential) is changing rapidly between nearby points in space, that's when you have a strong force exerted on charges. 


  Things closest to it [a battery] are being pushed with the greatest force.


This isn't necessarily true.

I could connect two wires to the two terminals of the battery and run them a long distance away, and then place the ends of the two wires very close to each other (but not touching). Then )very nearly) the full potential difference of the battery would be found across the small gap between the two wires, and the field strength between the wire ends would be much stronger than the field strength near the actual battery terminals. 

Schematically, here's the situation I'm trying to describe:







simulate this circuit &ndash; Schematic created using CircuitLab

(Imagine the battery symbols represent several cells within a battery)