Fan voltage fluctuating with potentiometer
You cannot use a 10K pot to control fan speed that way.

If the pot is set to 50% (top half is 5K), and the fan draws 170 mA, you would have 850 Volts across the top half of the pot, and the pot would be dissipating 144 Watts!!!

You should use an LM317 or similar adjustable regulator, and use the pot as one of the adjustment resistors (even then, you'll want a much lower resistance pot.)