What to do when we can't trust our human classifiers?
First, neural networks are good in dealing with"label noise". I'm currently on mobile/vacation, so remind me to search the paper on Friday.

Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.

I see the ways: 

(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%

(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)

(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable