How would I make the unpainted part of a texture paint image not affect object color when mixing?
Use a math node set to "Less than" with a very low threshold, and use that in the factor input of the mix RGB.

Illustration :