convolutions and xception in my paper "xceptiondeep learning with depthwise separable convolutions hyperparameter optimization when building deep-learning modelyou have to make many seemingly arbitrary decisionshow many layers should you stackhow many units or filters should go in each layershould you use relu as activationor different functionshould you use batchnormalization after given layerhow much dropout should you useand so on these architecture-level parameters are called hyperparameters to distinguish them from the parameters of modelwhich are trained via backpropagation in practiceexperienced machine-learning engineers and researchers build intuition over time as to what works and what doesn' when it comes to these choices-they develop hyperparameter-tuning skills but there are no formal rules if you want to get to the very limit of what can be achieved on given taskyou can' be content with arbitrary choices made by fallible human your initial decisions are almost always suboptimaleven if you have good intuition you can refine your choices by tweaking them by hand and retraining the model repeatedly--that' what machinelearning engineers and researchers spend most of their time doing but it shouldn' be your job as human to fiddle with hyperparameters all day--that is better left to machine thus you need to explore the space of possible decisions automaticallysystematicallyin principled way you need to search the architecture space and find the bestperforming ones empirically that' what the field of automatic hyperparameter optimization is aboutit' an entire field of researchand an important one the process of optimizing hyperparameters typically looks like this choose set of hyperparameters (automaticallybuild the corresponding model fit it to your training dataand measure the final performance on the validation data choose the next set of hyperparameters to try (automaticallyrepeat eventuallymeasure performance on your test data the key to this process is the algorithm that uses this history of validation performancegiven various sets of hyperparametersto choose the next set of hyperparameters to evaluate many different techniques are possiblebayesian optimizationgenetic algorithmssimple random searchand so on training the weights of model is relatively easyyou compute loss function on mini-batch of data and then use the backpropagation algorithm to move the weights see note above licensed to