How to explain that a same DNN model have radically different behaviours with each new initialization and training?
I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.

Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform').

... &quot;unusual&quot; element to point here;  I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.

There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously too smooth.

I've also noticed that the losses provided by history.history are different

Yes, so this is a little gotcha in the keras implementation. For training loss, keras does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. (link)
That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by $\sim1/2$  of an epoch.
The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated (&quot;dying ReLus&quot;) almost instantly.
Couple of suggestions that I can make:

I don't see you setting up a learning rate. Try making it smaller (like, much smaller). And see if there's any difference.

MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?



Does the number of neurons, hence the number of weights influence the phenomena of &quot;dying ReLus&quot; ?

Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on https://playground.tensorflow.org/ - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.