Print contaminated by chips of previous filament colors
My bet is your extruder knurls are scratching the filament. If it happens for specific filament then probably it doesn't keep its diameter. It means you can avoid these grains by using better filament.

If it happens always then you could do 3 things


reduce stress on the spring which pushes filament to extruder gear
make extruder gear less sharp
eventually change the great itself