# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # # Course 2: Week 2

# ## ## Mini-batch gradient descent
#
# To process your whole training set at once would demand a lot of memory. Instead, we will split our training set inti mini-batchs (smaller portions) of size $t$.
#
# Then, we run our model on each portion at a time, until it has run all of them.
#
# [...] "$epoch$ is a word that means a single pass through the training set."

# **Example:**
#
#
# One of the many parameters that we used was `number_of_iterations`. 
#
# Assume,
#
# `number_of_iterations`$ = 10$,
#
# and that we split our trainig data ($X$) in 50 parts.
#
# Then, for each mini-batch $0 < i < 50$, $X^{\{i\}}$, we will run $X^{\{i\}}$ through our model a `number_of_iterations`-times.
#
# Whenever all 50 parts are done, we'll have **1 $epoch$**
#
# Which means we just got a new parameter to tune: `number of epochs`.

# ![](files/media/mini.png)
#
# Pay attention to $X^{\{i\}}$'s and $Y^{\{i\}}$'s shapes:
#
# $X := (n_x, t)$ and $Y := (1, t)$

# Applying vectorization:
#
# ![](files/media/mini2.png)
# ![](files/media/mini3.png)
# ![](files/media/mini4.png)

# Plotting $J$, we should come by something like this:
#
# ![](files/media/mini5.png)
#
# (Notice that one the mini-batch scenario, each iteration over $X^{\{i\}}$ is as if we'd computed $J$ for our entire trainig set, and then dit it again, and appended the graphs)
#
# ![](files/media/mini6.png)
#
# [...] "In Stochastic Gradient Descent, you **use only 1 training example** before updating the gradients. When the training set is large, SGD can be faster. But the parameters will "oscillate" toward the minimum rather than converge smoothly"
#
# ![](files/media/sgd.png)

# ## ## Exponentially weighted (moving) averages

# **Example:**
#
# $\theta_i$ is the i-est day temperature
#
# ![](files/media/exp.png)
#
#
# $V_t = 0.9 * V_{t-1} + 0.1 * \theta_t$
#
# Let's define and start adjusting a parameter $\beta$:
#
# $V_t = \beta * V_{t-1} + (1 - beta) * \theta_t$
#
# ![](files/media/exp2.png)
#
# ![](files/media/exp3.png)
#
# ![](files/media/exp4.png)
#
# Thus, having a **exponentially weghted average**

# ## ### Bias correction
#
# ![](files/media/exp5.png)

# ## ## Gradient descent with momentum
#
# ![](files/media/gmm.png)
#
# The best behaviour for us is
#
# ![](files/media/gmm1_2.png)
#
# [...] "And this up and down oscillations slows down gradient descent and
# prevents you from using a much larger learning rate"
#
# [...] "if you were to use a much larger learning rate you might end up over shooting and end up diverging"
#
# ![](files/media/gmm2.png)
#
# [...] "So what this does is smooth out the steps of gradient descent."
#
# ![](files/media/gmm3.png)
#
# [...] "So, in the vertical direction, where you want to slow things down, this will average out positive and negative numbers, so the average will be close to zero. Whereas, on the horizontal direction,
# all the derivatives are pointing to the right of the horizontal direction, so the average in the horizontal direction will still be pretty big. So that's why with this algorithm, with a few iterations
# you find that the gradient descent with momentum ends up eventually just taking steps that are much smaller oscillations in the vertical direction, but are more directed to just moving quickly in the horizontal direction. And so this allows your algorithm to take a more straightforward path, or to damp out the oscillations in this path to the minimum"
#
# ![](files/media/gmm4.png)
#
# To apply the `bias correction`,
#
# $Vdw = \frac{Vdw}{(1-\beta^t)}$
#
#
# $Vdb = \frac{Vdb}{(1-\beta^t)}$

# ## ## Root mean squared (RMS) prop

# ![](files/media/rms.png)
#
# To avoid denominator explosion, we may add $\epsilon$ to the square root,
#
# $w := w - \alpha * \frac{dw}{\sqrt{Sdw + \epsilon}}$
#
# $b := b - \alpha * \frac{db}{\sqrt{Sdb + \epsilon}}$

# ## ## Adam optimization algorithm
# *(ADAM: Adaptive moment estimation)*
#
# ![](files/media/adam.png)
#
# ![](files/media/adam2.png)
#
# ![](files/media/adam3.png)
#
# $Vdw = \frac{Vdw^{corrected}}{(1-\beta^t)}$
#
#
# $Vdb = \frac{Vdb^{corrected}}{(1-\beta^t)}$
#
# About our parameters,
#
# $\alpha$: needs to be tune
#
# And good (practice) starting values are
#
# $\beta_1 = 0.9$
#
# $\beta_2 = 0.999$
#
# $\epsilon = 10^{-8}$
#

# ## ## Learning rate decay
#
# Slowly reduce the learning rate to avoid wandering around the global minimum.
#
# Note that you'll probably be using mini-batch, where each $X^{\{i\}}$ could be interpreted by the algorithm as another data set. Then, to avoid $W, b$ to wander around, we could adjust $\alpha$ as epochs go and optminize $J$ faster.
#
# ![](files/media/ld.png)
#
# ![](files/media/ld2.png)
#
# $\alpha = \frac{1}{1 + decayRate * epochNum} * \alpha_0$

# ## ### Other decay methods
#
# `Exponential decay`:
#
# $\alpha = 0.95^{epochNum} * \alpha_0$
#
# `Discrete decay`:
#
# $\alpha = \frac{k}{\sqrt{epochNum}} * \alpha_0$

# <div class="alert alert-block alert-info">
# I would encourage you, after optimizing a problem, to get creative and try to think of other method(s) to speed up learning.
#     
# A good first step, would be to try and plot your solution to check for possible vanishing-exploding situations
# </div>

# ## ## Local optima problem

# [...] "a plateau is a region where the derivative is close to zero for a long time"
#
# ![](files/media/ld3.png)
#
# ![](files/media/ld5.png)
#
# To avoid beeing stuck on these regions for a long time, that's why we use optmizations.

# <div class="alert alert-block alert-info">
# <b>Regarding this week's assignment:</b>
# <div class="alert alert-block alert-warning">
# I just learned that we can use another method to initialize a matrix with zeros.
#
# Instead of,
#
# **np.zeros(vec.shape)**
#
# One can use,
#
# **np.zeros_like(vec)**
#
# </div>
# </div>

import numpy as np

# +
vec = np.array([
    [[1, 11, 111], [2, 22, 222], [3, 33, 333]],
    
    [[4, 44, 444],[5, 55, 555],[6, 66, 666]],
    
    [[7, 77, 777],[8, 88, 888],[9, 99, 999]]
])

# vec is just like the RGB channels from a picture
# remember course 1

vec
# -

vec.shape

# +
vec_re = vec.reshape(vec.shape[0], -1).T

vec_re
# -

vec_re.shape

z_ = np.zeros(vec_re.shape)
z_

z_.shape

z__ = np.zeros_like(vec_re)
z__

z__.shape



StarcoderdataJupyterScriptsDedupFiltered