Does using your test set ultimately burn your data set in case of failure? Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.
I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.
I am now stuck with two options:

Throw everything away and start new with the same data set. This however has been likened to p-hacking.
Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.

Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?
Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?
I would try one more time to split the data &amp; train a new algo. This is similar to p-hacking as you've stated so try to avoid doing this more than once but if the alternative is the project stops there doesn't seem to be much more you can do.
This time when you split the data I would try stratifying it by some feature (the target? Some category of the input variables?). This ensures the training/validation/testing sets have the same proportion of some criteria which might help your performance be a little more consistent when comparing the validation set to the testing set.