Transistor is getting hot on low duty pwm cycle
Put a reverse biased diode across the motor to dissipate the inductive flyback spike. The transistor is doing that at the moment and that could be leading to your heating problem.
The transistor is not designed to do this and it could lead to failure of the transistor.