# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + _cell_guid="79c7e3d0-c299-4dcb-8224-4455121ee9b0" _uuid="d629ff2d2480ee46fbb7e2d37f6b5fab8052498a" id="v0BtAX1--7l_" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302344, "user_tz": -330, "elapsed": 4165, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Import Numpy & PyTorch
import numpy as np
import torch

# + [markdown] _uuid="34f006aa7eb4bbc683c39b7059021da900180908" id="tUurNfvF-7mc" colab_type="text"
# A tensor is a number, vector, matrix or any n-dimensional array.

# + [markdown] _uuid="0b65b6bb4d15127b1d51f09abf616cfd29fa48b4" id="DAOgSWEp-7oF" colab_type="text"
# ## Problem Statement

# + [markdown] _uuid="c1beecda01bc332596edd193cade30006e3f6cbf" id="-Fi1M6pd-7oJ" colab_type="text"
# We'll create a model that predicts crop yeilds for apples (*target variable*) by looking at the average temperature, rainfall and humidity (*input variables or features*) in different regions. 
#
# Here's the training data:
#
# >Temp | Rain | Humidity | Prediction
# >--- | --- | --- | ---
# > 73 | 67 | 43 | 56
# > 91 | 88 | 64 | 81
# > 87 | 134 | 58 | 119
# > 102 | 43 | 37 | 22
# > 69 | 96 | 70 | 103
#
# In a **linear regression** model, each target variable is estimated to be a weighted sum of the input variables, offset by some constant, known as a bias :
#
# ```
# yeild_apple  = w11 * temp + w12 * rainfall + w13 * humidity + b1
# ```
#
# It means that the yield of apples is a linear or planar function of the temperature, rainfall & humidity.
#
#
#
# **Our objective**: Find a suitable set of *weights* and *biases* using the training data, to make accurate predictions.

# + [markdown] _uuid="c24b8195c0e9c6e8e13e169d264484f1f9b3b1ae" id="h0dmV2Fc-7oL" colab_type="text"
# ## Training Data
# The training data can be represented using 2 matrices (inputs and targets), each with one row per observation and one column for variable.

# + _uuid="dfda99005fc6daf3a49ae1cdd427ccac0aa446b1" id="MaIf33bV-7oN" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302348, "user_tz": -330, "elapsed": 4147, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Input (temp, rainfall, humidity)
inputs = np.array([[73, 67, 43], 
                   [91, 88, 64], 
                   [87, 134, 58], 
                   [102, 43, 37], 
                   [69, 96, 70]], dtype='float32')

# + _uuid="bf56faf74f7e29c9ed7523308718a9ab1acc0667" id="1tnPriBD-7oa" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302352, "user_tz": -330, "elapsed": 4134, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Target (apples)
targets = np.array([[56], 
                    [81], 
                    [119], 
                    [22], 
                    [103]], dtype='float32')

# + [markdown] _uuid="70d48f83ae4fce7aba7dd78fd58dddc77c598bfd" id="MyJm3YtE-7oo" colab_type="text"
# Before we build a model, we need to convert inputs and targets to PyTorch tensors.

# + _uuid="931c1bad8788e607fa100d4338e1b1fe120e2339" id="KZyKnyCc-7oq" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302357, "user_tz": -330, "elapsed": 4106, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Convert inputs and targets to tensors


# + [markdown] _uuid="652647cd90bd0784ec4dc53472410f7358ee18c9" id="y0RLCJnb-7o4" colab_type="text"
# ## Linear Regression Model (from scratch)
#
# The *weights* and *biases* can also be represented as matrices, initialized with random values. The first row of `w` and the first element of `b` are use to predict the first target variable i.e. yield for apples, and similarly the second for oranges.

# + _uuid="6f788ae559355b3f01667be1554a5d2bdcade8db" id="OjToROni-7o5" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302360, "user_tz": -330, "elapsed": 4083, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Weights and biases


# + [markdown] _uuid="3579a065997cae41f7f504916b6bc07878ac768c" id="8qNNejI9-7pH" colab_type="text"
# The *model* is simply a function that performs a matrix multiplication of the input `x` and the weights `w` (transposed) and adds the bias `b` (replicated for each observation).
#
# $$
# \hspace{2.5cm} X \hspace{1.1cm} \times \hspace{1.2cm} W^T \hspace{1.2cm}  + \hspace{1cm} b \hspace{2cm}
# $$
#
# $$
# \left[ \begin{array}{cc}
# 73 & 67 & 43 \\
# 91 & 88 & 64 \\
# \vdots & \vdots & \vdots \\
# 69 & 96 & 70
# \end{array} \right]
# %
# \times
# %
# \left[ \begin{array}{cc}
# w_{11} & w_{21} \\
# w_{12} & w_{22} \\
# w_{13} & w_{23}
# \end{array} \right]
# %
# # +
# %
# \left[ \begin{array}{cc}
# b_{1} & b_{2} \\
# b_{1} & b_{2} \\
# \vdots & \vdots \\
# b_{1} & b_{2} \\
# \end{array} \right]
# $$

# + _uuid="b1119f5ae9688a5f31dba438c7f78ca382deb7e3" id="5G_d0Ka--7pJ" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302364, "user_tz": -330, "elapsed": 4075, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SS<KEY>aC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Define the model


# + [markdown] _uuid="8e0a4644cb1c4ed68a3bcf67a8a156341ac7c853" id="nT94e2ZK-7pb" colab_type="text"
# The matrix obtained by passing the input data to the model is a set of predictions for the target variables.

# + _uuid="b042a3cf8f16f4c4380cccbac9d0892719c24190" id="VUpnkKlO-7pd" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302367, "user_tz": -330, "elapsed": 4054, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Generate predictions


# + _uuid="5551ef933de7902c8b5a38ae3d8e4795cb244f38" id="KuIPDbJV-7po" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302371, "user_tz": -330, "elapsed": 4031, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Compare with targets


# + [markdown] _uuid="2c4a9cf2b3c9152f2f832176bce9a87381e2419c" id="Q-NuYiwI-7p4" colab_type="text"
# Because we've started with random weights and biases, the model does not perform a good job of predicting the target varaibles.

# + [markdown] _uuid="edaae7266f5d47c5e970e1438a812f10d8d35fb4" id="hiNOZ2g1-7p7" colab_type="text"
# ## Loss Function
#
# We can compare the predictions with the actual targets, using the following method: 
# * Calculate the difference between the two matrices (`preds` and `targets`).
# * Square all elements of the difference matrix to remove negative values.
# * Calculate the average of the elements in the resulting matrix.
#
# The result is a single number, known as the **mean squared error** (MSE).

# + _uuid="dbf5bca8cbf2a3831089b454c70469e3748e9682" id="_wY9fW06-7p9" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302375, "user_tz": -330, "elapsed": 4024, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# MSE loss


# + _uuid="90da6779aad81608c40cdca77c3c04b68a815c11" id="V__m5zOU-7qH" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302378, "user_tz": -330, "elapsed": 4000, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Compute loss


# + [markdown] _uuid="3ab3acadf389f30430b55c26c7979dcffaa974a5" id="j-TOY_7g-7qS" colab_type="text"
# The resulting number is called the **loss**, because it indicates how bad the model is at predicting the target variables. Lower the loss, better the model. 

# + [markdown] _uuid="c61acf9c3cff205d769fc52ed3b1b76f5ae66233" id="kbQQKg0_-7qU" colab_type="text"
# ## Compute Gradients
#
# With PyTorch, we can automatically compute the gradient or derivative of the `loss` w.r.t. to the weights and biases, because they have `requires_grad` set to `True`.
#
# More on autograd:  https://pytorch.org/docs/stable/autograd.html#module-torch.autograd

# + _uuid="ef66710c6ef1944567c4dc033e1ca316f35490ab" id="jMUIxzeO-7qW" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302380, "user_tz": -330, "elapsed": 3991, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Compute gradients


# + [markdown] _uuid="6504cddcfb4bfb0817bf03ef460f08f3145a9091" id="CtacVbsp-7qk" colab_type="text"
# The gradients are stored in the `.grad` property of the respective tensors.

# + _uuid="5943d1cef604a178c95f5e8d255519d42d9f9982" id="RWG8jqkG-7qo" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302382, "user_tz": -330, "elapsed": 3966, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Gradients for weights


# + _uuid="47278e318b156c6a5812e0842dbc4164c8362562" id="SzeDazjr-7qx" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302384, "user_tz": -330, "elapsed": 3944, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Gradients for bias


# + [markdown] _uuid="466dc3a2cc2d4bd2c10ae4cf59cf4627b5cc9c75" id="6HohuU-I-7q_" colab_type="text"
# A key insight from calculus is that the gradient indicates the rate of change of the loss, or the slope of the loss function w.r.t. the weights and biases. 
#
# * If a gradient element is **postive**, 
#     * **increasing** the element's value slightly will **increase** the loss.
#     * **decreasing** the element's value slightly will **decrease** the loss.
#
#
#
#
# * If a gradient element is **negative**,
#     * **increasing** the element's value slightly will **decrease** the loss.
#     * **decreasing** the element's value slightly will **increase** the loss.
#     
#
#
# The increase or decrease is proportional to the value of the gradient.

# + [markdown] _uuid="35ed968bfc135bd44eeb100ae401d0628fbc5c63" id="oRgBMWgV-7rB" colab_type="text"
# Finally, we'll reset the gradients to zero before moving forward, because PyTorch accumulates gradients.

# + _uuid="5f02dc376c21857d4e545d98413952c5ac73039b" id="lwkeQev0-7rD" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302387, "user_tz": -330, "elapsed": 3929, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}


# + [markdown] _uuid="5501c66c9729c4954e9b798a0634a9d84487e639" id="pjKbQIcT-7rN" colab_type="text"
# ## Adjust weights and biases using gradient descent
#
# We'll reduce the loss and improve our model using the gradient descent algorithm, which has the following steps:
#
# 1. Generate predictions
# 2. Calculate the loss
# 3. Compute gradients w.r.t the weights and biases
# 4. Adjust the weights by subtracting a small quantity proportional to the gradient
# 5. Reset the gradients to zero

# + _uuid="ef0d2bd2d9c5acb60992e238439ee00c2223319f" id="NbJYF_oB-7rP" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302390, "user_tz": -330, "elapsed": 3913, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Generate predictions


# + _uuid="302ee8226da4ee5d0dad137c638573a79f8abded" id="yt9A0Bzw-7rb" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302392, "user_tz": -330, "elapsed": 3896, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Calculate the loss


# + _uuid="01c596aecf87e4670033ddd4ed36e26b97e2f9ab" id="X3U2apNf-7rp" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302399, "user_tz": -330, "elapsed": 3896, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Compute gradients


# + _uuid="ec1e2bdc8f91523e556fad55ee8c01eb5431ae24" id="Gi8Iw67j-7rz" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302405, "user_tz": -330, "elapsed": 3895, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Adjust weights & reset gradients


# + _uuid="1d61b6f61f49b19099d29d1be8ec5ae4967bbd51" id="sB17H1hr-7sD" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302408, "user_tz": -330, "elapsed": 3878, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
#print(w)

# + [markdown] _uuid="6af10c29db7cb0d6e869b2c30966a34a48a011e2" id="YbNSWsxh-7so" colab_type="text"
# With the new weights and biases, the model should have a lower loss.

# + _uuid="c542b5fe75d82454f34cac13cdcff8b48dd1945c" id="tzhhX8xh-7su" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302412, "user_tz": -330, "elapsed": 3863, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Calculate loss


# + [markdown] _uuid="5201901695f3ea13d7fdd5d985da7e0761c541d0" id="JvUhV8nQ-7s9" colab_type="text"
# ## Train for multiple epochs
#
# To reduce the loss further, we repeat the process of adjusting the weights and biases using the gradients multiple times. Each iteration is called an epoch.

# + _uuid="9f5f0ffeee666b30c5828636359f0be6addbef7c" id="rX0ZllBO-7tJ" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302416, "user_tz": -330, "elapsed": 3860, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Train for 100 epochs


# + _uuid="c4820ca48b78f4dc242d80a9ec3ec6aca1aef671" id="ym2eslp8-7ta" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302418, "user_tz": -330, "elapsed": 3842, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Calculate loss


# + _uuid="bbcd65fa7094cec187565e54c2107e683bea787b" id="l7z9q9g9-7to" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302418, "user_tz": -330, "elapsed": 3825, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Print predictions


# + _uuid="addec2c4eca8edfcae5544ea2cc717182c21d90f" id="tEjLn-IO-7ty" colab_type="code" colab={} executionInfo={"status": "ok", "timestamp": 1596557302420, "user_tz": -330, "elapsed": 3807, "user": {"displayName": "Prof. <NAME>", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggt3sg6X_951s0boD3SSJvqRng4AQaC3MhTBtGQ9Q=s64", "userId": "16159546014304882594"}}
# Print targets


StarcoderdataJupyterScriptsDedupFiltered