advanced deep-learning best practices in the right direction updating hyperparameterson the other handis extremely challenging consider the followingcomputing the feedback signal (does this set of hyperparameters lead to high-performing model on this task?can be extremely expensiveit requires creating and training new model from scratch on your dataset the hyperparameter space is typically made of discrete decisions and thus isn' continuous or differentiable henceyou typically can' do gradient descent in hyperparameter space insteadyou must rely on gradient-free optimization techniqueswhich naturally are far less efficient than gradient descent because these challenges are difficult and the field is still youngwe currently only have access to very limited tools to optimize models oftenit turns out that random search (choosing hyperparameters to evaluate at randomrepeatedlyis the best solutiondespite being the most naive one but one tool have found reliably better than random search is hyperopt (library for hyperparameter optimization that internally uses trees of parzen estimators to predict sets of hyperparameters that are likely to work well another library called hyperas (with keras models do check it out one important issue to keep in mind when doing automatic hyperparameter optimization at scale is validation-set overfitting because you're updating hyperparameters based on signal that is computed using your validation datayou're effectively training them on the validation dataand thus they will quickly overfit to the validation data always keep this in mind note overallhyperparameter optimization is powerful technique that is an absolute requirement to get to state-of-the-art models on any task or to win machine-learning competitions think about itonce upon timepeople handcrafted the features that went into shallow machine-learning models that was very much suboptimal nowdeep learning automates the task of hierarchical feature engineering--features are learned using feedback signalnot hand-tunedand that' the way it should be in the same wayyou shouldn' handcraft your model architecturesyou should optimize them in principled way at the time of writingthe field of automatic hyperparameter optimization is very young and immatureas deep learning was some years agobut expect it to boom in the next few years model ensembling another powerful technique for obtaining the best possible results on task is model ensembling ensembling consists of pooling together the predictions of set of different modelsto produce better predictions if you look at machine-learning competitionsin particular on kaggleyou'll see that the winners use very large ensembles of models that inevitably beat any single modelno matter how good licensed to