Models that converged before aren't converging anymore in Keras
It was actually just the random weights and incredible luck that I got good convergence the first three tries in a row while didn't most other times afterwards.

I tried setting the seed and found certain seeds that gave me good results every time whereas most other seeds won't converge for many epochs.

from numpy.random import seed
seed(5)
from tensorflow import set_random_seed
set_random_seed(42)