How does Blender deal with partially textured .obj object?
My solution to this would be to move all the uv coordinates of the non uv-mapped faces to the very same point.

Having a few pixels set to the color that you want your mesh to be and putting your uvs there instead can be a way to get rid of unwanted colors.