How to treat the datasets with unreliable labels
You have a common issue in which your learning data is noisy. Of course if you can do anything to make your data better/use more data this is the way to go. But I assume manual cleaning at large is not possible.

Reasons for the noise?

There are different approaches in machine learning to deal with noisy data. It depends a lot on where your noise is coming from. It helps to understand why your labels are incorrect. In your case it sounds like there is a structural mistake where you systematically misclassify under certain conditions.

ML methods to solve the noisy data issue

Besides simple rules (if applicable) there are more complex ways to approach the issue. There are many papers (e.g. this, this and this) that suggest different variants of the following:


Manually create a small subset of cleaned labels
Train the network on clean and noisy labels
Use an additional network (or layer) to either create a mapping from noisy to clean labels or to learn about the noise distribution


This way you try to make the network recognizing its own mistake. Some of these models are rather complex, others are quite complex. There is even a paper that trains the network to also classify the noise and than make predictions based on features + noise classification.

I am not sure if there are any best practises yet which architecture works best in which case. You may need to try out some alternatives.