We used the Leaky Rectified Linear Unit (Leaky ReLU) as an activation function. It has been proposed as a way to resolve the dying units problem of ReLU, by preventing the unit from saturating, thus enabling a small gradient to always flow through the unit, potentially recovering extreme values of the weights 41 ; it can be described as follows: