Why does camouflage make some planes fly slower?
Same reason gliders keep their wings waxed.  It wasn't the camo per se, it was the dull matte field-applied paint finish that included all sorts of imperfections, and to a small degree, the weight of the paint vs natural finish.  At a microscopic level, the surface of the matte finish was much rougher than a gloss or unpainted metal surface.  The microscopically rougher surface would have a stronger ability to inhibit air molecules flowing right against the surface (skin friction).  Add to that the surface imperfections introduced by the typical wartime spray-and-get-it-out-the-door process, vs smooth bare aluminum.  

Think of the inhibiting effect on wind velocity close to the ground of a forested area vs a grassy plain, and shrink it down to the micro level.  The effect is most pronounced where the flow is relatively smooth, or laminar against the surface, which is most areas there the cross section is getting fatter (leading edges of wings and control surfaces, the nose, etc).

The Mustang would suffer a bit more than others from this with flat paint because of its (somewhat) laminar wing that had a maximum thickness well aft, which means it would have more surface area that is sensitive to skin friction, than a non-laminar wing aircraft like a Spitfire or Typhoon.  So if the Mustang was penalized 5kt with flat paint, a Spitfire might have only been penalized 3kt because it had less surface area that was (relatively) laminar in the first place.