Potentiometer warm up issue
Controlling a 10W motor with a 5W pot isn't great in the first place.  At certain speeds, you could already be getting close to 5W, and that won't be spread evenly across the whole length of the pot.

The more mechanical load you place on a motor, the more current it draws.  More current means more heat in the potentiometer.  If you are turning the pot towards one end to increase the speed to compensate for the extra load, then the heat is now being dissipated in an even smaller part of the pot's track.