Converting 12V capacitors from parallel to series: do they need to be rated for 250V?
If the capacitors are close to the same value they need only be rated at somewhat more than 12V. It's common to do this with Cockroft-Walton voltage multipliers (you will be avoiding the series diode drops by using mechanical contacts).  

If you have quite different values, then the voltage can distribute unevenly when the output discharges (the lower value capacitors would have more voltage across them).