) where a superscript T denotes a transpose, 𝐰 . and 𝐛 . are respectively the connection weights and biases of layer l, and 𝑔 is the hidden-layer activation function. Nonlinear hidden layer activation functions like rectified linear unit (ReLU) or Gaussian Error Linear Unit (GELU) are suggested to prevent the vanishing gradient problem. Note that GELU is an improved version of ReLU with a global continuous behavior. The GELU activation function is defined as