how do i clean rust flecks off an alloy wheel
If it's truly an alloy rim, it isn't rust. Use a metal polish, like you find at the auto parts store for polishing alloy wheels. 

It will clean of and smooth out any surface issues without removing to much material. 

It takes elbow grease, or you can use a Dremel with a jewelry polishing wheel on it to do it faster.