"Vertex paint" or Cycles "Mix node" problem?
Having such problems with mask created by Vertex Paint:


First check whether there are double vertices (W >
Remove doubles). 
Recalculate normals with Ctrl + N. The result
is better but still weird, something like:


Continue using Vertex Paint, simply drawing over these gaps and
smoothing boundaries and it's done.