Can I use confident predictions to correct incorrect labels?
At the moment, you don't really know the performance of your model because you have quite a few wrong labels in your test set.

You mentioned that you want to use the new probabilities to correct the validation and test sets. However, if you do that, you will of course get higher results because you are using the labels coming from your own model. However, if you verify by hand than it is ok.

Therefore, what I would suggest is:


Train the classifier with all your data. Don't do any splits.
After training, pass all your data through the model. Separate all the data that the model gets wrong.
Manually check all the mistakes. You will have a few scenarios: The model gets is wrong but the label was correct) in this case, leave the label as it was. The model gets it wrong but was wrong) this are the interesting cases for you, where the model is wrong but in fact it is correct since the original labels were wrong. Correct these.
Now perform train/validation/test splits.
Retrain your model with the train/validation splits.
Test your model with the test split.