Why does it speed up gradient descent if the function is smooth? I now read a book titled "Hands-on Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, it has the following description on the explanation of ELU (Exponential ReLU).


  Third, the function is smooth everywhere, including around z = 0, which helps speed up Gradient Descent, since it does not bounce as much left and right of z = 0.




The z means the x-axis on the graph above. I understand the derivative is smooth since the z &lt; 0 line has a curve and in that realm the derivative is no longer equal to 0. 

However, why is it the case that if the function is "smooth everywhere, including around z=0", it speeds up Gradient Descent?
My guess it is due to the derivative, as ReLU has a discontinuous derivative at 0. Hence if you use the definition:

$$
f' \approx \frac{f(x+\epsilon) -f(x-\epsilon)}{2 \epsilon}
$$

and $x$ is very close to 0, you will get many of those 'jumps'.