Constant static thrust from a brushed motor powered by minimal weight rechargeable battery
You will need independent constant current sources for each motor. The power source should be 12 VDC. Google "constant current supplies" and you will find many off the shelf, ready to use.
You need a 12 volt source because by definition a constant current source puts out whatever voltage is needed to maintain a certain current setting. You still have adjustable speed control but now it is done by adjusting the current to the motors.
Due to resistance and internal voltage drop to be sure your motors get 9 volts (if they need it) the voltage source should be around 12 volts or so.
You can use most any 12 volt battery, from a marine deep-cycle battery (very heavy if 100 A/H type) to 3 Li-Po batteries in series.
For a given type of battery be sure to have the correct charger. Li-Po batteries must be charged individually as they need a charger specific to that battery voltage (3.8 volt, 4.2 volt, etc). The 12 volt marine battery will need a more conventional 12 volt charger, but may not charge as fast as the Li-Po batteries using a fast charge mode.
At 5 amps of current for both motors (I assume that will be a normal setting you would want) the marine battery will last for days, But you may have weight and space issues with such a heavy battery. The Li-Po batteries will weigh much less but last only as long as their amp-hour rating divided by the expected run time before you must charge them.
The safety issue is not so much the marine battery but the Li-Po batteries. They cannot be charged in series as one cell will overcharge while the others are under charged. A over charged Li-Po battery can get hot enough to catch on fire, hence the reason for more elaborate charging procedures.


EDIT: Others have the opinion that for best performance you should get motors designed for a servo-loop where current and voltage are based only on the motor RPMs you want or need. Another suggestion is a PWM constant wattage supply, which may or may not be cheaper. I wrote my answer assuming you wanted to keep the motors you had and control motor speed in a safe way.

5 amps at 9 volts is only 45 watts. With some conversion losses a 60 watt power source would be enough, and that is not going to be that expensive. No more than a 100 A/H marine battery plus its charger.