Strategies for automatically tuning the hyper-parameters of deep learning models I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and suggestions/critiques on my proposed strategy below. I realise that experimentation may be the best way forward but I don't want to unnecessarily duplicate somebody else's work.

I know that a popular approach is to assume that the function of model error wrt. HPs is a bayesian process, the parameters of which are updated as new points are evaluated (ie. an expensive full training of the underlying deep learning model) so that a more rapid convergence on the optimal HPs can be achieved by quickly predicting better candidates for the next HPs to be evaluated than a simple grid search would be able to. Ilija Ilievski et al report more efficient convergence by using an RBF surrogate model (with pySOT) instead of bayesian https://github.com/ilija139/HORD.

Given that changes in some HPs (let's call them cheap) eg. ADAM initial learning rate and learning rate decay won't affect training time wheras doubling some HPs (let's call them expensive) eg. SGD batch size or total number of steps will double training time, does it make sense to approach this problem by trying to quickly refine cheap HPs whilst the object function is cheap(er) to evaluate and then allowing the object function to become more expensive gradually as the expensive HPs inflate?

One way of implementing this startegy might be to hold expensive HPs at relatively low values whilst refining cheap HPs and then allowing expensive HPs to grow and then iterating this process until some precondition is met, ie. model-error no longer goes down or processor-time budget is exceeded.

An alternative and possibly more 'organic' approach that I imagine would effectively achieve something similar would be to give each evaluation of the object function a certain processor time budget that would gradually increase. So when the object function over-ran its budget, it would be (gracefully) stopped and whatever error result that model gave, would be taken alongside whatever HPs were used (with number of steps being the actual number of steps taken, rather than what was requested). The idea being that this would pressure the HP surrogate to rapidly refine the cheap HPs initially whilst allowing the expensive HPs to be refined over time.

Does this make sense?
I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network. 

My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).

I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures

Ymmv, perhaps.