# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] colab_type="text" id="EhimMQAvYp5e"
# # Tensorflow Keras Image Classification using Convolution Neural Network
#

# + colab_type="code" id="Z7xhxleXLcvU" outputId="61cfede0-3d92-43f2-99ea-ed2d790f6269" colab={"base_uri": "https://localhost:8080/", "height": 80}
from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf

tf.keras.backend.clear_session()  # For easy reset of notebook stat
print(tf.__version__)

# + [markdown] colab_type="text" id="1HJ3_JgZRDOB"
# ## Sample network and training on MNIST dataset
#
# Below is a sample code from the tensorflow keras library to model a cnn architecture, training it with a dataset and running inference.
#
# Please go through it and try to understand the different classes and functions available in tensorflow keras to model a cnn architecture.
#
#

# + colab_type="code" id="MSPWa5z03cmD" outputId="acd202e4-e34a-4ffe-d1f9-2abeb95dba83" colab={"base_uri": "https://localhost:8080/", "height": 693}
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
print("train",train_images.shape)
print("test",test_images.shape)
train_images = (train_images.reshape(train_images.shape[0], 28, 28, 1).astype('float32'))/255
test_images = (test_images.reshape(test_images.shape[0], 28, 28, 1).astype('float32'))/255
print("train",train_images.shape)
print("test",test_images.shape)
train_images = train_images.astype('float32') / 255.
test_images = test_images.astype('float32') / 255.

one = train_images[3]
import matplotlib.pyplot as plt
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i].reshape((28,28)), cmap=plt.cm.binary)
    plt.xlabel(train_labels[i])
plt.show()


# + colab_type="code" id="I9cV5d_jYYYw" outputId="1859a076-4010-49aa-c6e1-0ec71283121a" colab={"base_uri": "https://localhost:8080/", "height": 1000}
inputs = tf.keras.Input(shape=(28,28,1), name='img')
x = tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu')(inputs)
x = tf.keras.layers.MaxPooling2D()(x)
x = tf.keras.layers.Conv2D(filters=64, kernel_size=3, activation='relu')(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dropout(rate=0.5)(x)
outputs = tf.keras.layers.Dense(units=10, activation='softmax')(x)
model = tf.keras.Model(inputs, outputs, name='vanila cnn')
model.summary()
tf.keras.utils.plot_model(model, 'vanilla_cnn.png', show_shapes=True)


# + colab_type="code" id="30bkUEM2p-0-" outputId="aa2ccc28-879e-4daa-924c-7a97eac76223" colab={"base_uri": "https://localhost:8080/", "height": 1000}

#y_train = tf.keras.utils.to_categorical(y_train, 10)
#y_test = tf.keras.utils.to_categorical(y_test, 10)

model.compile(optimizer=tf.keras.optimizers.RMSprop(1e-3),
              loss='sparse_categorical_crossentropy',
              metrics=['acc'])
_history = model.fit(train_images, train_labels,
          validation_data=(test_images, test_labels),
          batch_size=64,
          epochs=30,
          validation_split=0.2)


# + colab_type="code" id="Mp9jMQxOrsxy" outputId="35dbaa1c-3a18-4a29-cd1b-05695a0cca53" colab={"base_uri": "https://localhost:8080/", "height": 337}
import matplotlib.pyplot as plt
plt.figure(figsize=(8,5))
plt.plot(_history.history['loss'])
plt.plot(_history.history['val_loss'])

# + [markdown] colab_type="text" id="gv-QwVF3RVBH"
#
# ## Questions: Understanding Tensorflow Keras API  [2x5 = 10 points]

# + [markdown] colab_type="text" id="w8pwRi1slTpv"
# * What does "?" signify in the network diagram or None in the the model summary?
#

# + [markdown] colab_type="text" id="r3mdas6DEnK8"
# Here None specifies the batch size of the input. It is not necessary to mention the batch size while mentioning the input shape. If batch size is given while defining the input shape then it would replace the 'None' in the model summary and '?' in the network diagram.

# + [markdown] colab_type="text" id="sVHVXRc_Em_x"
# * Why does layer conv2d_19 (Conv2D)  have 2 dimension less than the upper layer?
#

# + [markdown] colab_type="text" id="AvsWWM9vEm2X"
# the shape of the output of the convolution layer is slightly lesser because, the kernel has a width greater than one and we compute the cross-correlation where the kernel fits completely with in the image. the output size of the convolution layer is determined by the input shape $(A X B)$ minus the convolution kernel size $(a x b)$ as $(A-a+1)\;and \;(B-b+1)$. in our case the input size is $(28 x 28)$ and kernel size is $(3x3)$, so the output size will be $(26x26)$

# + [markdown] colab_type="text" id="wKwvV3cJEmij"
# * what is stride?
#

# + [markdown] colab_type="text" id="JRQSLWylEms8"
# Stride is referred as the number of rows and columns traversed per slide by the kernel. generally for computing the cross correlation, we slide the kernel through all locations starting from top left corner to bottom right right corner. and sometimes to increase the computational efficiency or downsample we move more than one location at a time skipping the intermediate location. and this movement is referred as stride. 
#
# For example, consider a cross-correlation operation with stride of 3 vertically and 2 horizontally, if the second element of the first column is the output then the kernel is slides down three rows and if the second element in the first row is the output, the the kernel slides two colums to the right.
#

# + [markdown] colab_type="text" id="wyOIgyHEEmZ2"
#

# + [markdown] colab_type="text" id="NM6oMspDEmOq"
# * Difference between MaxPooling2D and GlobalAveragePooling2D ?
#

# + [markdown] colab_type="text" id="mUEzIdt8El6F"
# like convolutional layers, pooling operators consists of similar fixed shape windows that traverse over all the locations either to compute maximum or average of the elements in that pooling window.
#
# Maxpooling2D is a maximum pooling window of 2 dimension, which computes the maximum of the elements in that particular window. The results are downsapled or max pooled feature maps that highlight the most present feature in that window. This works better for image classification problems.
#
# Global Average Pooling is an operation that calculates the average output of each feature map in the previous layer. This fairly simple operation reduces the data significantly and prepares the model for the final classification layer. it helps in the removal of a large number of trainable parameters from the model.
#
# Max Pooling also performs as a Noise Suppressant. It discards the noisy activations altogether and also performs de-noising along with dimensionality reduction. On the other hand, Average Pooling simply performs dimensionality reduction as a noise suppressing mechanism. Hence, we can say that Max Pooling performs a lot better than Average Pooling.
#

# + [markdown] colab_type="text" id="6s8xZWN5EeSO"
# * What is sparse_categorical_crossentropy loss function ?

# + [markdown] colab_type="text" id="Kd8QHQeaEy48"
# it is a loss function which performs the calculation of the error without requiring that the target variable be one hot encoded values.
#
# for example, if the target variables are one-hot encoded like [1,0,0] , [0,1,0], [0,0,1] then use the categorical-crossentropy loss function,
#
# if the target variables are integers, say [1], [2], [3]..., then we use sparse categorical crossentropy loss functions.
#

# + [markdown] colab_type="text" id="fvqxGHte16q5"
# # Build different convolutions in Numpy
#
# In the questions below you will learn the workings of different convolution network by implementing them in python numpy.
#
# There are different kinds of convolution which has been implemented in different neural architectures. The most popular ones are :
#
# 1. Simple Convolution
# 2. Atrous Convolution
# 3. Depth wise Convolution
# 4. Point wise Convolution
# 5. Depth wise separable Convolution
#
# Good summary can be found here
# [convolutions-types]( https://ikhlestov.github.io/pages/machine-learning/convolutions-types/). 
#
# The task is to read about them and implement them using numpy

# + colab_type="code" id="X6JWEk6Z3sI0" outputId="ddd5d499-07fc-47c8-bef3-286092efc0d1" colab={"base_uri": "https://localhost:8080/", "height": 51}
# Helper functions
import matplotlib.pyplot as plt

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()
train_images = train_images.reshape(train_images.shape[0], 32, 32, 3).astype('float32')
test_images = test_images.reshape(test_images.shape[0], 32, 32, 3).astype('float32')

train_images = train_images.astype('float32') / 255.
test_images = test_images.astype('float32') / 255.

one = train_images[3]
def plot_convolution_images(image, convoled_image):
        
    plt.figure(figsize=(10,10))
    plt.subplot(1,2,1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.title("Image")
    plt.imshow(image.reshape((32,32,3)))
    plt.subplot(1,2,2)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(convoled_image.reshape((32,32)))
    plt.title("Convolution")
    plt.show()


# + [markdown] colab_type="text" id="fVrvSLzduVwX"
# ### Sample: Simple Convolution 

# + colab_type="code" id="RPD9Ap_Y1_zL" outputId="3ea3d449-8333-4d30-f794-7c417d2eb416" colab={"base_uri": "https://localhost:8080/", "height": 338}
import numpy as np

def myConv2D(image: np.ndarray, 
             filter_size: int, 
             kernel_size: int):
    """
    myConv2D convolutions.
    Args:
        image: (hi, wi, cin).
        filter_size: (fs). The number of output filters
        kernel_size: (ks)
    Returns:
        (fs, hi, wi)
    """
    print(image.shape)
  
    kernels = np.random.rand(kernel_size, kernel_size, image.shape[-1], filter_size )
    
    out = np.zeros([filter_size, image.shape[0], image.shape[1]])
    pad = kernel_size // 2
    image_padded = np.pad(image, [(pad,), (pad,), (0,)], 'constant')
    
    for idx_c_out in range(filter_size):
        for idx_row in range(image.shape[0]):
            for idx_col in range(image.shape[1]):
                kernel = kernels[:, :, :, idx_c_out]
                patch = image_padded[idx_row: idx_row + kernel_size, idx_col: idx_col + kernel_size]
                out[idx_c_out, idx_row, idx_col] += (kernel * patch).sum()
    
    return out

# Create inputs
#image = np.random.rand(28, 28, 1).astype(np.float32)
kernel_size = 3
filters = 64
convolved = myConv2D(one, filters, kernel_size)
print (convolved.shape)
plot_convolution_images(one, convolved[0])

# + id="i-xoBMTYI891" colab_type="code" colab={}
np.pad()


# + [markdown] colab_type="text" id="OUiwDeMAvBQ9"
# ## Question: Extend Simple Convolution to accomodate Atrous/Dilated Convolution in numpy [10 points]

# + colab_type="code" id="RrCRGsAKu_Xo" outputId="1df56212-4958-4f27-a4c7-2a00beec9104" colab={"base_uri": "https://localhost:8080/", "height": 625}
def myConv2DWithDilation(image: np.ndarray, 
             dilation_rate: int, 
             filter_size: int,
             kernel_size: int):
    """
    myConv2D convolutions.
    Args:
        image: (hi, wi, cin)
        dilation_rate: (dr) 
        filter_size: (ds)
        kernel_size: (ks)
    Returns:
        ( ?????Fill this ???? )
    """
    kernels = np.random.rand(kernel_size, kernel_size, image.shape[-1], filter_size )
    print(kernels.shape)
    n_ex, in_rows, in_cols, n_in = kernels.shape
    r_ix = np.repeat(np.arange(1, in_rows), dilation_rate-1)
    c_ix = np.repeat(np.arange(1, in_cols), dilation_rate-1)
    Xd = np.insert(kernels, r_ix, 0, axis=1)
    Xd = np.insert(Xd, c_ix, 0, axis=0)
    out = np.zeros([filter_size, image.shape[0], image.shape[1]])
    pad = kernel_size*dilation_rate // 2
    image_padded = np.pad(image, [(pad,), (pad,), (0,)], 'constant')
    
    for idx_c_out in range(filter_size):
        for idx_row in range(image.shape[0]):
            for idx_col in range(image.shape[1]):
                kernel = Xd[:, :, :, idx_c_out]
                patch = image_padded[idx_row: idx_row + (kernel_size*dilation_rate - (dilation_rate-1))
                                     , idx_col: idx_col + (kernel_size*dilation_rate - (dilation_rate-1))]
                out[idx_c_out, idx_row, idx_col] += (kernel * patch).sum()
    return out, kernels

kernel_size = 3
filters = 64
dilation = 2
convolved, kernels = myConv2DWithDilation(one, dilation, filters, kernel_size)
plot_convolution_images(one, convolved[0])
dilation = 3
convolved, kernels = myConv2DWithDilation(one, dilation, filters, kernel_size)
plot_convolution_images(one, convolved[0])


# + [markdown] colab_type="text" id="i4bnqbWu6Rpb"
# ## Question: Implement a DepthwiseConvolution  in numpy [10 points]

# + colab_type="code" id="y7nY5aj0823G" outputId="efc9d005-ba93-4a65-f8d8-ef15d9e3fc16" colab={"base_uri": "https://localhost:8080/", "height": 303}
def myDepthWiseConv2D(image: np.ndarray, 
             depth_multiplier: int, 
             kernel_size: int):
    """
    myConv2D convolutions.
    Args:
        image: (hi, wi, cin).
        depth_multiplier: (ds).The number of depthwise convolution output channels for each input channel
        kernel_size: (ks)
    Returns:
        (ds * cin, hi, wi)
    """

    kernels = np.random.rand(kernel_size, kernel_size, image.shape[-1], depth_multiplier )
    # c = kernels.shape[-1]
    out = np.zeros([image.shape[2] * depth_multiplier, image.shape[0], image.shape[1]])
    out1 = np.zeros([image.shape[0], image.shape[1],image.shape[2] * depth_multiplier])
    pad = kernel_size // 2
    image_padded = np.pad(image, [(pad,), (pad,), (0,)], 'constant')

    for idx_cin in range(image.shape[2]):
      for idx_row in range(image.shape[0]):
        for idx_col in range(image.shape[1]):
          patch = image_padded[idx_row: idx_row + kernel_size, idx_col: idx_col + kernel_size]
          for idx_cmul in range(depth_multiplier):
            kernel = kernels[:, :, idx_cin, idx_cmul]
            out[idx_cin * depth_multiplier + idx_cmul, idx_row, idx_col] = (kernel * patch).sum()
            out1[idx_row, idx_col,idx_cin * depth_multiplier + idx_cmul] = (kernel * patch).sum()


    return out, out1


kernel_size = 3
filters = 64
convolved, test = myDepthWiseConv2D(one, filters, kernel_size)
plot_convolution_images(one, convolved[0])


# + [markdown] colab_type="text" id="-b1wDzPRUNik"
# ## Question: Combine Depthwise and Pointwise and implement a Depthwise Separable Convolution in numpy [10 points]

# + colab_type="code" id="0y5cSYJsu523" outputId="5a4e67b1-6714-4fb4-c623-9211c5c980dc" colab={"base_uri": "https://localhost:8080/", "height": 321}
def myDepthWiseSeparableConv2D(image: np.ndarray,
                              depth_multiplier: int,
                              filters_pointwise: int,
                              kernel_size: int):
    """
    Depthwise separable convolutions.
    Args:
        image: (hi, wi, cin).
        depth_multiplier: (ds).The number of depthwise convolution output channels for each input channel
        filters_pointwise: (ps). The number of pointwise convolution output channels for each input channel
        kernel_size: (ks)
    Returns:
        (???? Fill this ????? )
    """
    convolved, img = myDepthWiseConv2D(image, depth_multiplier, kernel_size)
    out = myConv2D(img, depth_multiplier, filters_pointwise)

    return out


kernel_size = 3
filters = 64
pointwise_filters = 16
dilation = 2
convolved = myDepthWiseSeparableConv2D(one, filters, pointwise_filters, kernel_size)
plot_convolution_images(one, convolved[0])


# + [markdown] colab_type="text" id="3P_dMeYsvzPP"
# ## Questions: Understanding Convolution [2x5 = 10 points]

# + [markdown] colab_type="text" id="9GjiIX2kbM9j"
# * What are the advantages of Atrous/Dilated convolutions? For which tasks are they used?

# + [markdown] colab_type="text" id="6-5R7GoQbhoV"
# These convolutions introduces a new parameter to convolution layers called dialation rate. this specifies the spaceing between the values in kernel.
#
# This delivers a wider field of view at the same computational cost.
#
# Dilated convolutions have been shown to decrease blurring in semantic segmentation maps, and are purported to work at least in part by extracting long range information without the need for pooling.
#
# tasks which uses dilated convolutions:
#
# semantic image segmentation.\
# object detection\
# audio generation\
# video modeling\
# machine translation
#
#

# + [markdown] colab_type="text" id="ukSc9ceBbijn"
# * What are advantages of Depthwise convolution over normal convolution?

# + [markdown] colab_type="text" id="th_ohwdHbo_P"
# Amount of computations in Depthwise convolution is less than the normal convolution. which means that depthwise convolution are faster than normal convolutions. This also makes the depthwise convolutions computationally cheaper.
#
# the total number of parameters in case of normal convolution is more, which increases the risk of over fitting of the data, which is not the case of depthwise convolution.
#
#
#
#

# + [markdown] colab_type="text" id="rim6fxLC_MD2"
# # Build different Activation function
#
# In this section we will learn about different activation functions.

# + [markdown] colab_type="text" id="gyFb8qKJwB_X"
# ## Question: Softmax Activation [5 points]

# + colab_type="code" id="_SQaRygGwHob" colab={}
def softmax(x: np.ndarray):
    ## YOUR CODE HERE

    y = np.exp(x)
    out = np.exp(x)/np.sum(y)

    return out


# + id="Wn6gg8r8lSQz" colab_type="code" outputId="0168a888-9578-4927-f555-67393adc02ed" colab={"base_uri": "https://localhost:8080/", "height": 265}
# plot inputs and outputs
import matplotlib.pyplot as plt
 
# define a series of inputs
series_in = [x for x in range(-10, 11)]
# calculate outputs for our inputs
series_out = [softmax(x) for x in series_in]
# line plot of raw inputs to rectified outputs
plt.plot(series_in, series_out, label='SoftMax')
plt.legend()
plt.show()


# + [markdown] colab_type="text" id="N88zQImlwIRn"
# ## Question: Build a leaky RELU Activation function [5 points]

# + colab_type="code" id="_pDZk_MEwVuz" colab={}
def leakyrelu(x: np.ndarray,
              leaky_rate: float):
    
    out = np.maximum(leaky_rate*x, x)

    return out


# + id="AGEvYuqIjY1s" colab_type="code" outputId="1c81b1a8-b4db-4807-ba74-a258906d758e" colab={"base_uri": "https://localhost:8080/", "height": 265}
# plot inputs and outputs
import matplotlib.pyplot as plt
 
# rectified linear function
def rectified(x):
	return max(0.0, x)
 
# define a series of inputs
series_in = [x for x in range(-10, 11)]
# calculate outputs for our inputs
series_out = [rectified(x) for x in series_in]
series_out1 = [leakyrelu(x, 0.1) for x in series_in]
# line plot of raw inputs to rectified outputs
plt.plot(series_in, series_out, label='ReLU')
plt.plot(series_in, series_out1, label='LeakyReLU')
plt.legend()
plt.show()


# + [markdown] colab_type="text" id="Si0dJGEgwOY0"
# ## Question: Build a Swish Activation function [10 points]
#
# Details of the swish activation function can be found in section 4 of the paper [SEARCHING FORACTIVATIONFUNCTIONS](https://arxiv.org/pdf/1710.05941.pdf)

# + colab_type="code" id="xnORrGPOwZlC" colab={}
def swish(x: np.ndarray,
          beta: float):
    
    out = []
    for item in x:
               #(the sigmoid function)
        out.append(np.dot(item, (1/(1+math.exp(beta * -item)))))

    return out


def sigmoid(x):
    a = []
    for item in x:
               #(the sigmoid function)
        a.append(1/(1+math.exp(-item)))
    return a


def firstderswish(x: np.ndarray,
                  beta: float):
    out=[]
    for item in x:
      f1 = np.dot(item, (1/(1+np.exp(-beta*item))))
      sig = (1/(1+np.exp(-beta*item)))
      f = beta*f1 +  np.dot(sig, (1-beta*f1))
      out.append(f)

    return out


# + id="igoMZRHKecJU" colab_type="code" outputId="9ca3df4a-e99f-41ef-e1bc-ac05c5166d49" colab={"base_uri": "https://localhost:8080/", "height": 545}
import math
import matplotlib.pyplot as plt
import numpy as np

x = np.arange(-10., 10., 0.2)

y = sigmoid(x)
y1 = swish(x, 0.1)
y2 = swish(x, 1.0)
y3 = swish(x, 10)
y4 = firstderswish(x, 0.1)
y5 = firstderswish(x, 1.0)
y6 = firstderswish(x, 10)


a = plt.figure(1)
plt.title('swish')
plt.plot(x,y, label='sigmoid')
plt.plot(x,y1, label='beta=0.1')
plt.plot(x,y2,label='beta=1.0')
plt.plot(x,y3,label='beta=10')
plt.legend()
a.show()
b = plt.figure(2)
plt.title('first derivative of swish')
plt.plot(x,y4, label='beta=0.1')
plt.plot(x,y5,label='beta=1.0')
plt.plot(x,y6,label='beta=10')
plt.legend()
b.show()

# + [markdown] colab_type="text" id="VwZDpyw-gnKX"
# # Explain a fractional Max pooling function [10 points]
#
# Fractional Max pooling was introduced by the paper [<NAME>, Fractional Max-Pooling](https://arxiv.org/abs/1412.6071)
#
# Explain below the difference between normal max pooling and fractional max pooling ?

# + [markdown] colab_type="text" id="wcafTm8phhSa"
# Is the fractional version of max-pooling in which multiplicative factor($ \alpha $) is allowed to take non-integer value within a range of $ 1< \alpha<2 $.
#
# The size of hidden layers is reduced by a fractional factor which provides an oppurtunity to view an image at a different scale.
#
# FMP introduces randomness which is associated with the choice of pooling region due to its stochastic nature. The region of pooling can either be overlapped or disjoint which is selected either randomly or pseudo-randomly with the use of dropout and training data augmentation.
#
# The use if fractional max pooling with overlapped region of pooling works better than the disjoint one.
#
# It is also noted that pseudorandom selection of pooling region with data augmentation performs better when compared to random selection.
#
# Max pooling:
# * Its statistical properties makes it capable of fitting sparse representation.
# *Perform betters when coupled with spare coding and simple linear classifiers.
# * Due to elimination of non-maximal components, computation for upper layers are reduced.
# * Is deterministic in nature
# * The discerning features get disappeared when majority of the elements in the pooling area are present in high magnitudes.
#
# Fractional Max-out Pooling:
# * Is stochastic in nature
# * Randomness or pseudorandomness in selection of pooling region.
# * Good performance of pseudorandomness selection with data augmentation.
# * Superior result is obtained by overlapping instead of disjoint fractional max pooling
# * random selection of pooling region reduces the model performance.
# * Disjoint fractional max pooling causes performance degradation.

# + [markdown] colab_type="text" id="4_ZEh3iCicmy"
# ## Total hours spent
#
# Please report the total time invested in solving the assignment below

# + colab_type="code" id="xoSjtR3aivOI" colab={}
total_time_invested = 30

StarcoderdataJupyterScriptsDedupFiltered