Ensemble modelling using model's probabilities
Let's say you fit n models on given dataset, to put them together, you have in fact few options:


Take the average (I'd play with weighted or geometric)
Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
You can go further for blending, boosting or bagging, there is really nice article about these methods here.