Strategies for automatically tuning the hyper-parameters of deep learning models
I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network. 

My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).

I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures

Ymmv, perhaps.