Is there a way to calculate the proper wire gauge?
There is no single formula which will do what you want to do. Rather, it is a series of steps.


What is your total power? - We already know the answer: 12 volts @ 40 amps, or 480 watts.
How much power are you willing to waste in your wire? - Let's use 5% as an illustration. As a check, this is 12 watts. Dissipated over a 10-meter cable, this is about 1.2 watts / meter, which seems reasonable. The wire will get warm but not hot, as long as the cable is not contained in an enclosed, insulating cable run.
5% of 12 volts is .6 volts. Since the cable runs to the heater and also back from the heater, the voltage drop on a single conductor is .3 volts in 10 meters, or .03 volts/meter.
Since the voltage drop in the cable occurs at 40 amps, the resistance of the wire must follow Voltage = Current x Resistance, and Resistance = Voltage/Current = .03/40 = .00075 ohms/meter. Note that this is the maximum resistance allowable. Any greater value produces more drop in the wire and more power dissipated.
From a standard table http://www.daycounter.com/Calculators/AWG.phtml you can see that a 3-guage wire will have a resistance of .000647 ohm/meter, while a 4-guage wire has .000815 ohms/meter.


Is this pretty heavy wire? Yup. Can you do better? Well, only if you're willing to heat up the cable more. The problem here is that your supply voltage is so low that you can't afford much resistance. As illustration of this, consider 480 watts at 120 volts. Then the current is 4 amps while the allowable (5%) voltage is 6 volts. Then you can repeat the calculations above and see that you only need 23 guage wire. 

Oh yes, and at 3 ga, you want stranded, not for durability, but just so you can bend the suckers.