DC motor in vacuum, heat generation
Interesting question!

Your worst case is that you have a perfect vacuum and no heat path out of your chamber.  600 rpm at 1000 gcm is about 6 watts of mechanical power.  I wouldn't choose a big motor because it would be less efficient; rather, choose a "right-sized" brushless motor and gearbox for your speed and load.  You should be able to achieve at least 50% efficiency, meaning that 6 watts will be going into your load and 6 watts will be lost as heat.  If you run for 10 minutes (or 600 seconds), 6 watts will produce 3600 joules or 860 calories.  You only need to attach a heat sink capable of absorbing 860 calories.  Aluminum has a specific heat of 0.215 calories/gram-Kelvin.  Do the math, and 200 grams of aluminum heat sink will have a temperature rise of 20 degrees K.  So attach a heat sink and you can play around with the time, losses, and thermal mass until you get what you need.  Of course, you must cool it after each run by allowing air in for convection or some other heat path out of your chamber.  Otherwise the temperature will continue to rise with each run.

Good luck!