Thin wire overheats, thick wire allows too much current to flow
Thick wire is good, the low resistance gives you low losses.
If the torque of the motor is too high, reduce the voltage. Preferably you use a 12V source, that you can adjust to the value you want (that is basically what you are doing right now with the different wire resistance, but this results, as you noticed, in a lot of power losses in the wire).
You can either use an adjustable 12V power supply that you can plug in the outlet, or combine your 12V battery with a DC/DC converter. There are ready-to-buy modules with a potentiometer to adjust the voltage.