What Is Saturating Gradient Problem
If you use sigmoid-like activation functions, like sigmoid and tanh, after some epochs of training, the linear part of each neuron will have values that are very big or very small. This means that the linear part will have a big output value regardless of its sign. Consequently, the input of sigmoid-like functions in each neuron which adds non-linearity will be far from the center of these functions.

In those locations, the gradient/derivative value is very small. Consequently, after numerous iterations, the weights get updated so slowly because the value of the gradient is very small. This is why we use the ReLU activation function for which its gradient doesn't have this problem. Saturating means that after some epochs that learning happens relatively fast, the value of the linear part will be far from the center of the sigmoid and it somehow saturates, and it takes too much time to update the weights because the value of gradient is small. You can take a look at here as a solution for this problem.
If I want to explain the math part, suppose that you are using sigmoid as the activation function. If $\sigma$ represents sigmoid, its gradient is $\sigma (1-\sigma$). Now suppose that your linear part, the input of sigmoid is a positive number which is too large, then sigmoid which is:

$$ \frac{1}{1+e^{-x}} $$

will have a value near to one but smaller than that. On the other hand $\sigma$ will be so close to zero, multiplying $\sigma$ and $1 - \sigma$ will result in a small value, which means the value of the gradient is very small. If the value of the linear part is very small, then $\sigma$ will be close to zero and $1- \sigma$ will be close to 1 but smaller than that. Again, multiplying these will lead to a small value as the gradient.