"""
![image](https://neurohive.io/wp-content/uploads/2018/11/vgg16.png)
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
"""
## VGG16 And ImageNet
"""
"""
The pre-trained model we'll be working with to classify images of cats and dogs is called VGG16, which is the model that won the 2014 ImageNet competition.

In the ImageNet competition, multiple teams compete to build a model that best classifies images from the ImageNet library. The ImageNet library houses thousands of images belonging to 1000 different categories.

We’ll import this VGG16 model and then fine-tune it using Keras. The fine-tuned model will not classify images as one of the 1000 categories for which it was trained on, but instead it will only work to classify images as either cats or dogs.

Note that dogs and cats were included in the ImageNet library from which VGG16 was originally trained. Therefore, the model has already learned the features of cats and dogs. Given this, the fine-tuning we'll do on this model will be very minimal
"""
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense, Flatten, BatchNormalization, Conv2D, MaxPool2D
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import KFold
import itertools
import os
import shutil
import random
import glob
import matplotlib.pyplot as plt
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
%matplotlib inline
train_path = '/kaggle/input/cat-and-dog/training_set/training_set'
# valid_path = 'data/dogs-vs-cats/valid'
test_path = '../input/cat-and-dog/test_set/test_set'
train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input) \
    .flow_from_directory(directory=train_path, target_size=(224,224), classes=['cats', 'dogs'], batch_size=10)
test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input) \
    .flow_from_directory(directory=test_path, target_size=(224,224), classes=['cats', 'dogs'], batch_size=10, shuffle=False)
imgs, labels = next(train_batches)
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 10, figsize=(20,20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()
"""
Let's first check out a batch of training data using the plotting function
"""
plotImages(imgs)
print(labels)    
"""
When we [previously](https://www.kaggle.com/bavalpreet26/cnn-tutorial-keras-nb2) inspected these images, we briefly discussed that the color data was skewed as a result of preprocessing the images using the tf.keras.applications.vgg16.preprocess_input function.
"""
"""
To understand what preprocessing is needed for images that will be passed to a VGG16 model, we can look at the [VGG16 paper](https://arxiv.org/pdf/1409.1556.pdf).
"""
"""
Under the 2.1 Architecture section, we can see that the authors stated that, "The only preprocessing we do is subtracting the mean RGB value, computed on the training set, from each pixel."

This is the preprocessing that was used on the original training data, and therefore, this is the way we need to process images before passing them to VGG16 or a fine-tuned VGG16 model.

This processing is what is causing the underlying color data to look distorted
"""
"""
# Part-1 Building A Fine - Tuned Model
"""
"""
Now, let's begin building our model.
"""
vgg16_model = tf.keras.applications.vgg16.VGG16()
"""
The original trained VGG16 model, along with its saved weights and other parameters, is now downloaded.

We can check out a summary of the model just to see what the architecture looks like.
"""
vgg16_model.summary()
"""
In contrast, recall how much simpler the CNN was that we worked with in the last [notebook](https://www.kaggle.com/bavalpreet26/cnn-tutorial-keras-nb2). VGG16 is much more complex and sophisticated and has many more layers than our previous model.
"""
"""
Notice that the last `Dense` layer of VGG16 has `1000` outputs. These outputs correspond to the 1000 categories in the ImageNet library.

Since we’re only going to be classifying two categories, cats and dogs, we need to modify this model in order for it to do what we want it to do, which is to only classify cats and dogs.

Before we do that, note that the type of Keras models we’ve been working with so far in this series have been of type `Sequential`.
"""
"""
If we check out the type of model `vgg16_model` is, we see that it is of type `Model`, which is from the Keras’ `Functional` API.
"""
type(vgg16_model)
"""
We’ve not yet worked with the more sophisticated Functional API, although we will work with it in later notebooks using the MobileNet model.
"""
"""
For now, we’re going to go through a process to convert the `Functional` model to a `Sequential` model, so that it will be easier for us to work with given our current knowledge.

We first create a new model of type `Sequential`. We then iterate over each of the layers in `vgg16_model`, except for the last layer, and add each layer to the new `Sequential` model.
"""
model = Sequential()
for layer in vgg16_model.layers[:-1]:
    model.add(layer)
"""
Now, we have replicated the entire `vgg16_model` (excluding the output layer) to a new `Sequential` model, which we've just given the name `model`.

Next, we’ll iterate over each of the layers in our new `Sequential` model and set them to be <font color='red'>non-trainable</font>. This freezes the weights and other trainable parameters in each layer so that they will not be trained or updated when we later pass in our images of cats and dogs.
"""
for layer in model.layers:
    layer.trainable = False
"""
The reason we don’t want to retrain these layers is because, as mentioned earlier, cats and dogs were already included in the original ImageNet library. So, VGG16 already does a nice job at classifying these categories. We only want to modify the model such that the output layer understands only how to classify cats and dogs and nothing else. Therefore, we don’t want any re-training to occur on the earlier layers.

Next, we add our new output layer, consisting of only 2 nodes that correspond to cat and dog. This output layer will be the only trainable layer in the model.
"""
model.add(Dense(units=2, activation='softmax'))
"""
We can now check out a `summary` of our model and see that everything is exactly the same as the original `vgg16_model`, except for now, the output layer has only `2` nodes, rather than 1000, and the number of `trainable parameters` has drastically decreased since we froze all the parameters in the earlier layers.
"""
model.summary()
"""
# Part - 2
"""
"""
### Train A Fine-Tuned Neural Network With TensorFlow's Keras API
"""
"""
Let's see how to train the fine-tuned VGG16 model to classify images as cats or dogs.
"""
"""
we’ll use the `Adam` optimizer with a learning rate of `0.0001`, `categorical_crossentropy` as our loss, and `‘accuracy’` as our metric.
"""
model.compile(optimizer=Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])
"""
Now, we’ll train the model using model.fit().

Note that the call to `fit()` is exactly the same as it was when we used it on the original CNN we built from scratch in a previous [notebook](https://www.kaggle.com/bavalpreet26/cnn-tutorial-keras-nb2), except for we're only running 5 epochs this time
"""
model.fit(x = train_batches, 
          steps_per_epoch = len(train_batches),
          epochs = 5,
          verbose = 2
         )
"""
Looking at the results from training, we can see just after 5 epochs, we have some pretty outstanding results, especially when you compare it to the results we got from our original model.

Our accuracy starts off at 96% and goes over 99% in just 5 epochs.
"""
"""
# Part-3 Predict
"""
"""
Now see how to use the fine-tuned VGG16 model that we trained above to predict on images of cats and dogs in our test set.
"""
"""
Picking up with the code, we’ll first get a batch of test samples and their corresponding labels from the test set, and plot them to see what the data looks like
"""
test_imgs, test_labels = next(test_batches)
plotImages(test_imgs)
print(test_labels)
"""
Recall that this is the same test set we used in a previous [notebook](https://www.kaggle.com/bavalpreet26/cnn-tutorial-keras-nb2) to test the model we built from scratch, and the color in the images appears to be distorted due to the VGG16 preprocessing we discussed previously.
"""
"""
We now call model.predict to have the model predict on the test data.
"""
predictions = model.predict(x = test_batches, steps = len(test_batches), verbose = 0)
"""
We pass in the test set, test_batches, and set steps to be then length of test_batches, steps specifies how many batches to yield from the test set before declaring one prediction round complete.
"""
"""
#### Plot Predictions with a Confusion Matrix
"""
"""
We’re now going to create a confusion matrix so we can visualize our predictions.
"""
# def plot_confusion_matrix(cm, classes,
#                           normalize=False,
#                           title='Confusion matrix',
#                           cmap=plt.cm.Blues):
#     """
#     This function prints and plots the confusion matrix.
#     Normalization can be applied by setting `normalize=True`.
#     """
#     plt.imshow(cm, interpolation='nearest', cmap=cmap)
#     plt.title(title)
#     plt.colorbar()
#     tick_marks = np.arange(len(classes))
#     plt.xticks(tick_marks, classes, rotation=45)
#     plt.yticks(tick_marks, classes)

#     if normalize:
#         cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
#         print("Normalized confusion matrix")
#     else:
#         print('Confusion matrix, without normalization')

#     print(cm)

#     thresh = cm.max() / 2.
#     for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
#         plt.text(j, i, cm[i, j],
#             horizontalalignment="center",
#             color="white" if cm[i, j] > thresh else "black")

#     plt.tight_layout()
#     plt.ylabel('True label')
#     plt.xlabel('Predicted label')
# test_batches.class_indices
cm = confusion_matrix(y_true=test_batches.classes, y_pred=np.argmax(predictions, axis=-1))
cm_plot_labels = ['cat','dog']
plot_confusion_matrix(cm=cm, classes=cm_plot_labels, title='Confusion Matrix')
"""
Inspired from [this channel](https://www.youtube.com/watch?v=oDHpqu52soI&list=PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL&index=13)
"""
"""
We can see that the model incorrectly predicted only few samples.So proving this model to be much more capable of generalizing than the previous CNN we built from scratch.
"""
"""
to be continue...
will share next kernal publically soon

link to next [part](https://www.kaggle.com/bavalpreet26/keras4/edit)
"""