What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
I no longer really use validation that much, but rather only training and testing.  Why? Because I mostly follow Ron Kohavi's (Stanford Univ) approach to CV.  I have done a lot of validation but it seemed to be overkill, essentially causing me to ask why I have this very small-sampled parameter watch on the side from which I am supposed to learn from. You know if an ANN is overfitting when the training error diverges away from testing error -- it's an indication the model is breaking up. See my blog on this.