Case fan making humming noise when used with potentiometer
Figure 1. The fan. Image source: PChub.
This type of fan does not control a simple brushed DC motor. Instead, they contain a BLDC (BrushLess DC) motor and drive electronics.

Figure 2. A typical BLDC fan driver circuit. Image source: EE web.
By powering the fan through a resistance you are probably charging up some internal capacitance and the driver chip turns on at some internal voltage threshold. It in turn starts up the motor which will promptly discharge the internal energy store, stop the motor and the cycle starts again.
Instead you should be powering the fan at the rated voltage and using the PWM input to control the speed.

Figure 3. PWM signal transitioning from high pulse width (75%) to low (25%) and back again should result in 75% and 25% of full speed. Image source: LEDnique.com.
You'll find plenty of PWM control examples on a web search.