Histogram Backprojection in OpenCV?
I cannot say much about how can we increase the accuracy of your blob detection by your current method till i exactly understand method and implementation; However, i feel it is quite certain that color (alone) quite often vulnerable to false detection due to the fact that many things in the world corresponds to closer to skin color. 

The best best, i believe is to use more information - hopefully an independent feature. For example, you can use shape - and apply the classifier who can detect hand/no-hand based on most possible counter shapes. some times the hand shape may have a round blob - but most often the fingers etc would bring the uniqueness necessary for classification. 
you can run across many possible scenarios to create a good model. 

Think about it, if you would have given this task to a human being - how would they have guessed it?