message: [TF 2.0 API Docs] Added Documentation to tf.keras.activations.selu
diff: @@ -89,12 +89,37 @@ def elu(x, alpha=1.0):
  def selu(x):
  """Scaled Exponential Linear Unit (SELU).
  - SELU is equal to: `scale * elu(x, alpha)`, where alpha and scale
  - are pre-defined constants. The values of `alpha` and `scale` are
  + The Scaled Exponential Linear Unit (SELU) activation function is:
  + `scale * x` if `x > 0` and `scale * alpha * (exp(x) - 1)` if `x < 0`
  + where `alpha` and `scale` are pre-defined constants
  + (`alpha = 1.6732632423543772848170429916717`
  + and `scale = 1.0507009873554804934193349852946`).
  + The SELU activation function multiplies `scale` > 1 with the
  + `[elu](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/activations/elu)`
  + (Exponential Linear Unit (ELU)) to ensure a slope larger than one
  + for positive net inputs.
  +
  + The values of `alpha` and `scale` are
  chosen so that the mean and variance of the inputs are preserved
  between two consecutive layers as long as the weights are initialized
  - correctly (see `lecun_normal` initialization) and the number of inputs
  - is "large enough" (see references for more information).
  + correctly (see [`lecun_normal` initialization]
  + (https://www.tensorflow.org/api_docs/python/tf/keras/initializers/lecun_normal))
  + and the number of inputs is "large enough"
  + (see references for more information).
  +
  + ![](https://cdn-images-1.medium.com/max/1600/1*m0e8lZU_Zrkh4ESfQkY2Pw.png)
  + (Courtesy: Blog on Towards DataScience at
  + https://towardsdatascience.com/selu-make-fnns-great-again-snn-8d61526802a9)
  +
  + Example Usage:
  + ```python3
  + n_classes = 10 #10-class problem
  + model = models.Sequential()
  + model.add(Dense(64, activation='selu', input_shape=(28, 28, 1)))
  + model.add(Dense(32, activation='selu'))
  + model.add(Dense(16, activation='selu'))
  + model.add(Dense(n_classes, activation='softmax'))
  + ```
  Arguments:
  x: A tensor or variable to compute the activation function for.
  @@ -103,11 +128,14 @@ def selu(x):
  The scaled exponential unit activation: `scale * elu(x, alpha)`.
  # Note
  - - To be used together with the initialization "lecun_normal".
  - - To be used together with the dropout variant "AlphaDropout".
  + - To be used together with the initialization "[lecun_normal]
  + (https://www.tensorflow.org/api_docs/python/tf/keras/initializers/lecun_normal)".
  + - To be used together with the dropout variant "[AlphaDropout]
  + (https://www.tensorflow.org/api_docs/python/tf/keras/layers/AlphaDropout)".
  References:
  - - [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
  + [Self-Normalizing Neural Networks (Klambauer et al, 2017)]
  + (https://arxiv.org/abs/1706.02515)
  """
  alpha = 1.6732632423543772848170429916717
  scale = 1.0507009873554804934193349852946
  