What does it mean for an activation function to be "saturated/non-saturated"? For context, in this paper


  Several RNN variants such as the long short-term memory (LSTM) [10,
  18] and the gated recurrent unit (GRU) [5] have been proposed to
  address the gradient problems. However, the use of the hyperbolic tan-
  gent and the sigmoid functions as the activation function in these
  variants results in gradient decay over layers. Conse- quently,
  construction and training of a deep LSTM or GRU based RNN network is
  practically difficult. By contrast, ex- isting CNNs using
  non-saturated activation function such as relu can be stacked into a very deep network (e.g. over 20 layers using the basic convolutional
  layers and over 100 lay- ers with residual connection
Please see this answer.

An activation function is considered non-satured if

$$
\lim_{z \rightarrow \infty} f(z) = \infty
$$

A saturated activation function has a compact range such as $[-1,1]$ for $\tanh$ or $[0,1]$ for the sigmoid.