Square wave from high voltage power supply (5kV)
The classical circuit for flipping polarities of a DC current is the H-Bridge. (Really, plenty of material on that out there.)
The trick will be finding a semiconductor switch rated for 5 kV (and more, considering there will be flyback voltage spikes).
There's IGBTs rated for that voltage, but chances are you won't even have enough current to make them operate reliably. Also, cost...
So, I guess you'll want to use a stack of MOSFETs on each leg of your H-Bridge, and control them using optocouplers. There's also optocouplers with MOSFETs outputs, but I don't know whether these exist for your voltages.