advanced deep-learning best practices that elephants are like snakesand they would forever stay ignorant of the truth of the elephant diversity is what makes ensembling work in machine-learning termsif all of your models are biased in the same waythen your ensemble will retain this same bias if your models are biased in different waysthe biases will cancel each other outand the ensemble will be more robust and more accurate for this reasonyou should ensemble models that are as good as possible while being as different as possible this typically means using very different architectures or even different brands of machine-learning approaches one thing that is largely not worth doing is ensembling the same network trained several times independentlyfrom different random initializations if the only difference between your models is their random initialization and the order in which they were exposed to the training datathen your ensemble will be low-diversity and will provide only tiny improvement over any single model one thing have found to work well in practice--but that doesn' generalize to every problem domain--is the use of an ensemble of tree-based methods (such as random forests or gradient-boosted treesand deep neural networks in partner andrei kolev and took fourth place in the higgs boson decay detection challenge on kaggle (www kaggle com/ /higgs-bosonusing an ensemble of various tree models and deep neural networks remarkablyone of the models in the ensemble originated from different method than the others (it was regularized greedy forestand had significantly worse score than the others unsurprisinglyit was assigned small weight in the ensemble but to our surpriseit turned out to improve the overall ensemble by large factorbecause it was so different from every other modelit provided information that the other models didn' have access to that' precisely the point of ensembling it' not so much about how good your best model isit' about the diversity of your set of candidate models in recent timesone style of basic ensemble that has been very successful in practice is the wide and deep category of modelsblending deep learning with shallow learning such models consist of jointly training deep neural network with large linear model the joint training of family of diverse models is yet another option to achieve model ensembling wrapping up when building high-performing deep convnetsyou'll need to use residual connectionsbatch normalizationand depthwise separable convolutions in the futureit' likely that depthwise separable convolutions will completely replace regular convolutionswhether for dor applicationsdue to their higher representational efficiency building deep networks requires making many small hyperparameter and architecture choiceswhich together define how good your model will be rather than basing these choices on intuition or random chanceit' better to systematically search hyperparameter space to find optimal choices at this licensed to