"""
# **Artificial Neural Networks: Street View Housing Number Digit Recognition**


### **Context:** 
--------------

One of the most interesting tasks in deep learning is to recognize objects in natural scenes. The ability to process visual information using machine learning algorithms can be very useful as demonstrated in various applications.

The SVHN dataset contains labeled digits cropped from street level photos. It is one of the most popular image recognition datasets. It has been used in neural networks created by Google to improve map quality by automatically transcribing the address numbers from a patch of pixels. The transcribed number with a known street address helps pinpoint the location of the building it represents. 

### **Objective:**
----------------

Build a feed foward neural network model that can identify the digits in the images. 

### **Note:**
----------------
ANN is not the best choice to deal with image data. CNN's has proven much better performance. This notebook is just for practice.
"""
"""
## **Importing libraries**
"""
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, BatchNormalization
from tensorflow.keras.utils import to_categorical
"""
## **Load the dataset**
- Let us now load the dataset that is available as a .h5 file.
- Split the data into train and the test dataset
"""
import h5py

# Open the file as read only
h5f = h5py.File('../input/street-view-house-nos-h5-file/SVHN_single_grey1.h5', 'r')

# Load the training and the test set
X_train = h5f['X_train'][:]
y_train = h5f['y_train'][:]
X_test = h5f['X_test'][:]
y_test = h5f['y_test'][:]


# Close this file
h5f.close()
"""
Let's check the number of images in the training and testing data.
"""
len(X_train), len(X_test),X_train.shape, X_test.shape
"""
- There are 42,000 images in the training data and 18,000 images in the testing data. 
"""
"""
## **Visualizing images**
- Use X_train to visualize the first 10 images
- Use Y_train to print the first 10 labels
"""
# visualizing the first 10 images in the dataset and their labels
plt.figure(figsize=(10, 1))

for i in range(10):
    plt.subplot(1, 10, i+1)
    plt.imshow(X_train[i], cmap="gray")
    plt.axis('off')

plt.show()
print('label for each of the above image: %s' % (y_train[0:10]))
"""
## **Data preparation**
"""
# Shape of the images and the first image

print("Shape:", X_train[0].shape)
print()
print("First image:\n", X_train[0])
# Reshaping the dataset to flatten them. Remember that we are trying to reshape the 2D image data into a 1D array

X_train = X_train.reshape(X_train.shape[0], 1024)
X_test = X_test.reshape(X_test.shape[0], 1024)
# Normalize inputs from 0-255 to 0-1

X_train = X_train/255
X_test = X_test/255
# New shape 

print('Training set:', X_train.shape, y_train.shape)
print('Test set:', X_test.shape, y_test.shape)
# one hot encode output
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
print(X_train.shape)
print(y_train.shape)
"""
## **Model Building**

Now that we have done the data preprocessing, let's build an ANN model.
"""
#Fixing the seed for random number generators
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
"""
#### **Build and train a ANN model**
"""
#Importing losses and optimizers modules
from tensorflow.keras import losses
from tensorflow.keras import optimizers

#Define the function
def nn_model_1():
    model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(1024, )),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(10, activation = 'softmax')
    ]) 
    
    #declare adam optimizer with learning rate of 0.001 
    adam = optimizers.Adam(learning_rate=0.001)
    
    #compile the model
    model.compile(optimizer='adam', loss='categorical_crossentropy',  metrics = ['accuracy'])
    return model
# Build the model
model_1 = nn_model_1()
#Print the summary
model_1.summary()
# Fit the model
history_model_1 = model_1.fit(X_train, y_train, epochs=20, validation_split=0.2, batch_size=128, verbose = 1)
"""
### **Plotting the validation and training accuracies**
"""
# plotting the accuracies

dict_hist = history_model_1.history
list_ep = [i for i in range(1,21)]

plt.figure(figsize = (8,8))
plt.plot(list_ep,dict_hist['accuracy'],ls = '--', label = 'accuracy')
plt.plot(list_ep,dict_hist['val_accuracy'],ls = '--', label = 'val_accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')
plt.legend()
plt.show()
"""
**Observations:_______**
- The Accuracy of both training and validation is almost symetric. This is a good indicator that the model is not overfitting the training dataset.
- The Accuracy starts to increase steadily until epoch 6, then it increases in a lower rate until epoch 17 where we observe minimum increase afterward.
- We can see that the accuracy of validation dataset is slightly higher than the training dataset until epoch 10, where they start replacing places. This could give a sign that the model starts to fit the training dataset better, but not to the point of overfitting yet.
- Overall, the model is not overfit and is giving accuracy of 0.7 on training dataset and roughly the same on validation dataset.
"""
"""
Let's build one more model with higher complexity and see if we can improve the performance of the model.
"""
#Clearing keras backend
from tensorflow.keras import backend
backend.clear_session()
#Fixing the seed for random number generators
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
"""
#### **Build and train the new ANN model as per the above mentioned architecture**
"""
#Importing losses and optimizers modules
from tensorflow.keras import losses
from tensorflow.keras import optimizers

#Define the function
def nn_model_2():
    model = tf.keras.Sequential([
      tf.keras.layers.Dense(256, activation='relu', input_shape=(1024, )),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dropout(rate = 0.2),
      tf.keras.layers.Dense(64, activation='relu'),
      tf.keras.layers.Dense(64, activation='relu'),
      tf.keras.layers.Dense(32, activation='relu'),
      tf.keras.layers.BatchNormalization(),
      tf.keras.layers.Dense(10, activation = 'softmax')             
    ]) 
    
    #declare adam optimizer with learning rate of 0.0005 
    adam = optimizers.Adam(learning_rate=0.0005)
    
    #compile the model
    model.compile(optimizer=adam, loss= 'categorical_crossentropy', metrics= ['accuracy'])
    
    return model
# Build the model
model_2 = nn_model_2()
#Print the model summary
model_2.summary()
# Fit the model
history_model_2 = model_2.fit(X_train,y_train, epochs=30, validation_split=0.2, batch_size=128, verbose = 1)
"""
### **Plotting the validation and training accuracies**
"""
# plotting the accuracies

dict_hist = history_model_2.history
list_ep = [i for i in range(1,31)]

plt.figure(figsize = (8,8))
plt.plot(list_ep,dict_hist['accuracy'],ls = '--', label = 'accuracy')
plt.plot(list_ep,dict_hist['val_accuracy'],ls = '--', label = 'val_accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epochs')
plt.legend()
plt.show()
"""
**Observations:_______**
- Accuracy of both training dataset and validation dataset is symetric to some level. This rejects any overfitting concerns.
- Accuracy of both is growing rapidly as the model progresses with epochs to 8, then accuracy still improves, but slowly.
- After epoch 7 we can see small ups and downs in the validation dataset accuracy. This is not a concern as the overall trend is positive.
- Overall the model is better than the previous with accuracy of 0.77 compared to 0.70.

"""
"""
## **Predictions on the test data**
"""
test_pred = model_2.predict(X_test)

test_pred = np.argmax(test_pred, axis=-1)
#Converting each entry to single label from one-hot encoded vector
y_test = np.argmax(y_test, axis=-1)
"""
#### **Print the classification report and the confusion matrix**
"""
#importing required functions
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

#Printing the classification report
print(classification_report(y_test, test_pred))

#Plotting the heatmap using confusion matrix
cm = confusion_matrix(y_test, test_pred)
plt.figure(figsize=(8,5))
sns.heatmap(cm, annot=True,  fmt='.0f')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
"""
#### **Observations:__________**
 - The classification report tells us that numbers 0 and 4 have the highest f1-score (0.81) meaning they have the best chances of being accurately recognized. Whereas, numbers 3 and 8 have the lowest f1-score of (0.77).
 - Number 3 has the lowest precision and 0 has the highest, this means that the model is classifing other numbers as 3 which not valid. Whereas 0 has the lowest chances of being invalidly positive.
 - Number 4 has the highest recall, whereas 5 and 6 have the lowest. It indicates that the model is struggling to identify all 5's and 6's as what they are. Opposed to 4, which the model idenifies in high rates of completion. 
 - The confusion matrix shows that the model confused 5 with 3 and 6 with 8. This complements the above obervations about 3, 5 and 6. which assures that they are having some prediction issues in general.

"""