What to do when we can't trust our human classifiers? Suppose we want to design a neural network that can diagnose skin cancer. We want this neural network to consider the possibility that the doctor we hired misclassified some of our images while labeling. How can we design our neural network?
First, neural networks are good in dealing with"label noise". I'm currently on mobile/vacation, so remind me to search the paper on Friday.

Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.

I see the ways: 

(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%

(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)

(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable