What is the best way to compare these small distributions?
To assess the effects of hyperparameters on the neural network results, you need to eliminate confounding variables. Let's say you have network $A_t$ and network $B_t$ corresponding to trial/sample $t$. The randomly initialized weights of $A_t$ should equal $B_t$. If you are using a stochastic optimizer (e.g. SGD), you need to ensure that the random instance selection is the same between training network $A_t$ and $B_t$.
Once you've eliminated confounding variables, then you effectively have paired samples, which you can compare via a Wilcoxon signed-rank test. For small sample sizes, it can be preferable over a paired t-test because 1. you can't verify that the samples are normally distributed and 2. the Wilcoxon test effectively evaluates median rather than mean. The median is more robust against outliers, which are particularly influential if you only have a few samples.