Weight-initialisation and model instability Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction. 

What could be the causes of such a behavior ? What would be the solutions ? 

*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.
As you have noticed weight initialization is extremely important, If for instance, you initialize your network with zeros, then it's guaranteed you will not learn from data. there are many ways to initialize your data:

Most simple approach (beginner approach) : 
just use random initialization

Xavier Initialization:
this might be a more intermediate approach if you are interested on this check the following link: 
https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79

I'd say that you gotta keep in mind when your network is "learning" it is basically navigating in a hyper dimension. most people do not actually standardize data. Standardizing data is essential since you make learning easier for your algorithm.

So short answer: standardize your data, your algorithm will converge faster into a minimum (of course not always global).