local minima vs saddle points in deep learning
Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition for this critical point to be a minimum is that the Hessian matrix is positive definite. As the Hessian is a symmetric matrix, we can diagonalize it. If we write the diagonal matrix corresponding to the Hessian as:
$$
  D =
  \begin{bmatrix}
    d_{1} &amp; &amp; \\
    &amp; \ddots &amp; \\
    &amp; &amp; d_{n}
  \end{bmatrix}
$$
the Hessian being positive definite is equivalent to $d_1 &gt; 0, \dots, d_n&gt;0$.

Now let's think about deep learning cost functions. Deep learning cost functions depend on lots of parameters in a very complicated manner, so the Hessian will have a complicated expression itself. For this reason, we can think that the values of $d_1,\dots,d_n$ are not biased towards negative or positive values. For this reason, given any critical point, the probability of every value $d_i$ to be positive can be assumed to be $1/2$. Moreover, it is reasonable to assume that the values of $d_i$ do not depend easily on the values of $d_j$, due to the high non-linearity of the Hessian matrix, so we will take the probabilities of them being positive as independent events.

For this reason, given a critical point, the probability of it being a minimum is:
$$
P(d_1 &gt; 0, \dots, d_n &gt; 0) = P(d_1 &gt; 0)\cdot \cdots \cdot P(d_n &gt; 0) = \frac{1}{2^n}
$$

The probability of any critical point being a minimum decreases exponentially with the dimension of the input space. In deep learning,
this space can range from 1000 to $10^8$, and in both cases $1/2^n$ is ridiculously small. Now we are convinced that, given any critical point that we come across, it is very unlikely that it is a minimum.

But what about maxima?

The maxima of a function are the minima of minus the function. For this reason, all the arguments used previously can be used to minus the cost function and we conclude that every critical point has probability of $1/2  ^n$ to be a maximum.

For this reason, given a critical point, the probability of it being a saddle point is 
$$P(saddle) = 1 - P(maximum) - P(minimum) = 1 - \frac{1}{2^n} - \frac{1}{2^n} = 1 - \frac{1}{2^{n-1}}$$

Which is very close to 1 if $n$ is large enough (which typically is in deep learning).