Deep model ensemble giving different results
Getting varying results at the level of F1 score is possible but should be high, as your are essentially taking the average over many values (assuming a reasonably sized dataset).

There are however many sources of random behaviour that could be creeping in. From most obvious to most obscure:


Are the definitions and parameters of each of your models identical each time?
Do you use the same train/validation/test splits for each session?
Are your training hyper-parameters (number of epochs, learning rate, early stopping) the same each time?
Do you set the random seed in Numpy/Keras/Tensorflow?
Did you set any Nvidia CUDA flags for determinism?


Regarding the final point, GPU determinism - this is a very new thing in the context of deep learning and many people believed it was impossible, but there is good progress being made. Check out these video/slides from Nvidia GTC conference this year.

These are really just the components of each individual building block i.e. each deep model. You could also introduce random behaviour perhaps in the way you chain these models together to create your ensemble.