getting the most out of your models ensembling relies on the assumption that different good models trained independently are likely to be good for different reasonseach model looks at slightly different aspects of the data to make its predictionsgetting part of the "truthbut not all of it you may be familiar with the ancient parable of the blind men and the elephanta group of blind men come across an elephant for the first time and try to understand what the elephant is by touching it each man touches different part of the elephant' body--just one partsuch as the trunk or leg then the men describe to each other what an elephant is"it' like snake,"like pillar or tree,and so on the blind men are essentially machine-learning models trying to understand the manifold of the training dataeach from its own perspectiveusing its own assumptions (provided by the unique architecture of the model and the unique random weight initializationeach of them gets part of the truth of the databut not the whole truth by pooling their perspectives togetheryou can get far more accurate description of the data the elephant is combination of partsnot any single blind man gets it quite rightbutinterviewed togetherthey can tell fairly accurate story let' use classification as an example the easiest way to pool the predictions of set of classifiers (to ensemble the classifiersis to average their predictions at inference timeuse four different models to compute initial predictions preds_a model_a predict(x_valpreds_b model_b predict(x_valpreds_c model_c predict(x_valpreds_d model_d predict(x_valthis new prediction array should be more accurate than any of the initial ones final_preds (preds_a preds_b preds_c preds_dthis will work only if the classifiers are more or less equally good if one of them is significantly worse than the othersthe final predictions may not be as good as the best classifier of the group smarter way to ensemble classifiers is to do weighted averagewhere the weights are learned on the validation data--typicallythe better classifiers are given higher weightand the worse classifiers are given lower weight to search for good set of ensembling weightsyou can use random search or simple optimization algorithm such as nelder-meadpreds_a model_a predict(x_valpreds_b model_b predict(x_valpreds_c model_c predict(x_valpreds_d model_d predict(x_valthese weights ( are assumed to be learned empirically final_preds preds_a preds_b preds_c preds_d there are many possible variantsyou can do an average of an exponential of the predictionsfor instance in generala simple weighted average with weights optimized on the validation data provides very strong baseline the key to making ensembling work is the diversity of the set of classifiers diversity is strength if all the blind men only touched the elephant' trunkthey would agree licensed to