Compatible driver motor for JGA25-370
Yes the battery is the main problem together with an undersized motor torque. Meaning your mechanical load >>2W motor rating and must be changed to exceed Load.  Physics 101.

Stall start current is 1A and if reversed at full speed 2A surge with normal max rated load 0.2A

This motor is temperature rated for 2W MAX = V*I

Since drivers are bipolar they drop at least 1.5 to 2.5V depending on current . So if you want max torque use and voltage use 2S Li-ion cell’s with current sensing and MOSFET 2A full bridge drivers with PWM speed/ power regulation. Determine Wh capacity you need.

Specs
    6V
 58
 0.06
 50
 0.4
 2.5
 20
 2
 1.5
 103

Load speed
rpm/min 
Load currentA 
Rated torque
kg.cm 
Stall torquekgcm 
Stall currentA 
Power
W 
Reduction ratio1:00 
Gearbox long mm