How to check for dead relu neurons Background:
While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as stated here. (What is the &quot;dying ReLU&quot; problem in neural networks?)

Question:
What im hoping to do is to implement a check in the code itself to check if the neurons are dead. After that, the code could refit the network if needed. 

As such, what is a good citeria to check for dead neurons? Currently im thinking of checking for low variance in the prediction as a citeria. 

If it helps, im using keras.
A dead ReLU pretty much just means that its argument value is negative such that the gradient stays at 0; no matter how you train it from that point on. You can simply have a look at the gradient during training to see whether a ReLU is dead or not.

In practice you may simply want to use leaky ReLUs, i.e. instead of f(x) = max(0,x) you set f(x) = x if x > 0 and f(x) = 0.01x if x &lt;= 0. This way you always allow a small non-zero gradient and the unit should not get fully stuck in training anymore.