What is the best practice for tuning hyperparameters using validation data?
If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example.
First, I'd double check the data to make sure the train, validation and test sets are definitely distinct, and that all three sets have roughly the same number of positive and negative examples. If this isn't obviously the problem, then most likely I'd guess that you are overfitting the hyperparameters to the validation dataset.
Using cross-validation would most likely help you to see whether this is the case, as you'd see more dramatic variation between each fold if the parameters were overfit towards one specific validation fold.
If you repeat the process on the test dataset, then that data is no longer &quot;unseen&quot; and is part of the model, even if you just use it to tune hyperparameters. You should keep the test dataset completely separate from the model building process so that when you come to measuring model performance, you get a true estimation of what would happen on unseen data.