Asynchronous Hyperparameter Optimization - Dependency between iterations
hyperopt proceeds sequentially, unless you let it use a parallel backend. Is N the max_evals parameter? Yes, you always want to select the hyperparams with the best validation loss. That's what it returns in the end. It may be that it found the best hyperparameters well before the final trial. It does not mean anything is wrong. It learns as it goes the distribution of the loss conditional on hyperparameters, and on purpose explores less-certain parts of the space that are most likely to yield improvement, but may not, especially towards the end of the search. This happens with entirely well-defined loss functions.