# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Optional: Different Actication Functions
#
# It is __optional__ but recommended for you to implement these activation functions in ```exercise_code/networks/layer.py```, both forward and backward pass, as a choice of hyperparameter. 
#
# __Note__: We provide you with the sigmoid activation function for your convenience, which you've already worked with in previous exercises. 

# Recall that activation functions introduce more non-linearity to the network. Here we introduce several kinds of activation functions:
#
# * Sigmoid
#
# $$Sigmoid(x) = \frac{1}{1 + exp(-x)}$$
#
# <img src=https://pytorch.org/docs/stable/_images/Sigmoid.png alt="Figure4" width="400"/>
#
# * ReLU
#
# $$ReLU(x) = max(0, x)$$
#
# <img src=https://pytorch.org/docs/stable/_images/ReLU.png alt="Figure2" width="400"/>
#
# * Leaky ReLU
#
# $$LeakyReLU(x) = max(0.01x, x)$$
#
# <img src=https://pytorch.org/docs/stable/_images/LeakyReLU.png alt="Figure3" width="400"/>
#
# * Tanh
#
# $$Tanh(x) = \frac{exp(x) - exp(-x)}{exp(x) + exp(-x)}$$
#
# <img src=https://pytorch.org/docs/stable/_images/Tanh.png alt="Figure3" width="400"/>

# %load_ext autoreload
# %autoreload 2

# +
from exercise_code.tests.layer_tests import *

print(ReluTest()())
print()
print(LeakyReluTest()())
print()
print(TanhTest()())
# -

# __Hint__:
#
# If you have implemented the extra activation functions, please run the following cell to check whether you have did it the right way. 
#
# Otherwise just skip the cell.

StarcoderdataJupyterScriptsDedupFiltered