Potentiometer getting warm
A 2 watt pot of 100ohms has a maximum permitted current of 140mA (\$I^2R\$=P, 0.14 * 0.14 * 100 = 2).

You are overloading the portion of track that's carrying the current, whether or not the total dissipation is below 2 watts.