How to hide vertex colors in Texture Paint?
Add a Vertex Color layer that is solid white and keep that one activated when you are not doing any Vertex Painting.  

It does waste a layer and definitely counts as a work-around but it's the only way I know of to do this and it works.