Replace fan speed controller with Dimmer switch
The color (as long as it isn't white, green, or bare) doesn't matter; they're just pig-tails on switch legs. You're good to go here with a direct swap out and arbitrarily connecting the wires to the new device (assuming it doesn't have a neutral contact. If it does it probably matters which contact has incoming power. Read the directions ;).

Make sure those nuts are still snug after you're done.