#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Explanation: Copyright 2020 Google LLC
End of explanation
"""


# To prevent unnecessary warnings (e.g. FutureWarnings in TensorFlow)
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

# TensorFlow and tf.keras
import tensorflow as tf

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt
import os
import subprocess
import json
import requests
from tqdm import tqdm

print(tf.__version__)

"""
Explanation: Adversarial Learning: Building Robust Image Classifiers
<br>
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/neural-structured-learning/blob/master/neural_structured_learning/examples/notebooks/adversarial_cnn_transfer_learning_fashionmnist.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/neural-structured-learning/blob/master/neural_structured_learning/examples/notebooks/adversarial_cnn_transfer_learning_fashionmnist.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
  </td>
</table>

Overview
In this tutorial, we will explore the use of adversarial learning
(Goodfellow et al., 2014) for image
classification using Neural Structured Learning (NSL).
Adversarial attacks intentionally introduce some noise in the form of perturbations to input images to fool the deep learning model. For example, in a classification system, by adding an imperceptibly small vector whose elements are equal to the sign of the elements of the gradient of the loss function with respect to the input, we can change the model's classification of the image. 
CNN Classifier
The most popular deep learning models leveraged for computer vision problems are convolutional neural networks (CNNs)!

<font size=2>Created by: Dipanjan Sarkar</font>
We will look at how we can build, train and evaluate a multi-class CNN classifier in this notebook and also perform adversarial learning.
Transfer Learning
The idea is to leverage a pre-trained model instead of building a CNN from scratch in our image classification problem

<font size=2>Source: CNN Essentials</font>
Tutorial Outline
In this tutorial, we illustrate the following procedure of applying adversarial learning to obtain robust models using the Neural Structured Learning framework on a CNN model:

Create a neural network as a base model. In this tutorial, the base model is
    created with the tf.keras sequential API by wrapping a pre-trained VGG19 model which we use for fine-tuning using transfer learning
Train and evaluate the base model performance on organic FashionMNIST data
Perform perturbations using the fast gradient sign method (FSGM) technique and look at model weaknesses
Wrap the base model with the nsl.keras.AdversarialRegularization wrapper class,
    which is provided by the NSL framework, to create a new tf.keras.Model
    instance. This new model will include the adversarial loss as a
    regularization term in its training objective.
Convert examples in the training data to a tf.data.Dataset to train.
Train and evaluate the adversarial-regularized model
Generate perturbed dataset from the test data using FGSM and evaluate base model performance
Evaluate adversarial model performance on organic and perturbed test datasets

Load Dependencies
This leverages the tf.keras API style and hence it is recommended you try this out on TensorFlow 2.x
End of explanation
"""


fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

print('\nTrain_images.shape: {}, of {}'.format(train_images.shape, train_images.dtype))
print('Test_images.shape: {}, of {}'.format(test_images.shape, test_images.dtype))

"""
Explanation: Main Objective — Building an Apparel Classifier & Performing Adversarial Learning


We will keep things simple here with regard to the key objective. We will build a simple apparel classifier by training models on the very famous Fashion MNIST dataset based on Zalando’s article images — consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. The task is to classify these images into an apparel category amongst 10 categories on which we will be training our models on.


The second main objective here is to perturb and add some intentional noise to these apparel images to try and fool our classification model


The third main objective is to build an adversarial regularized model on top of our base model by training it on perturbed images to try and perform better on adversarial attacks


Here's an example how the data looks (each class takes three-rows):
<table>
  <tr><td>
    <img src="https://raw.githubusercontent.com/zalandoresearch/fashion-mnist/master/doc/img/fashion-mnist-sprite.png"
         alt="Fashion MNIST sprite"  width="600">
  </td></tr>
  <tr><td align="center">
    <a href="https://github.com/zalandoresearch/fashion-mnist">Fashion-MNIST samples</a> (by Zalando, MIT License).<br/>&nbsp;
  </td></tr>
</table>

Fashion MNIST is intended as a drop-in replacement for the classic MNIST dataset—often used as the "Hello, World" of machine learning programs for computer vision. You can access the Fashion MNIST dataset directly from TensorFlow.
Note: Although these are really images, they are loaded as NumPy arrays and not binary image objects.
We will build the following two deep learning CNN (Convolutional Neural Network) classifiers in this notebook.
- Fine-tuned pre-trained VGG-19 CNN (Base Model)
- Adversarial Regularization Trained VGG-19 CNN Model (Adversarial Model)
The idea is to look at how to use transfer learning where you fine-tune a pre-trained model to adapt it to classify images based on your dataset and then build a robust classifier which can handle adversarial attacks using adversarial learning.
Load Dataset
End of explanation
"""


train_images_3ch = np.stack([train_images]*3, axis=-1)
test_images_3ch = np.stack([test_images]*3, axis=-1)

print('\nTrain_images.shape: {}, of {}'.format(train_images_3ch.shape, train_images_3ch.dtype))
print('Test_images.shape: {}, of {}'.format(test_images_3ch.shape, test_images_3ch.dtype))

"""
Explanation: Fine-tuning a pre-trained VGG-19 CNN Model - Base Model
Here, we will use a VGG-19 model which was pre-trained on the ImageNet dataset by fine-tuning it on the Fashion-MNIST dataset. 
Model Architecture Details

<font size=2>Source: CNN Essentials</font>
Reshaping Image Data for Modeling
We do need to reshape our data before we train our model. Here we will convert the images to 3-channel images (image pixel tensors) as the VGG model was originally trained on RGB images
End of explanation
"""


def resize_image_array(img, img_size_dims):
  img = tf.image.resize(
      img, img_size_dims, method=tf.image.ResizeMethod.BICUBIC)
  img = np.array(img, dtype=np.float32)
  return img

%%time

IMG_DIMS = (32, 32)

train_images_3ch = np.array([resize_image_array(img, img_size_dims=IMG_DIMS) for img in train_images_3ch])
test_images_3ch = np.array([resize_image_array(img, img_size_dims=IMG_DIMS) for img in test_images_3ch])

print('\nTrain_images.shape: {}, of {}'.format(train_images_3ch.shape, train_images_3ch.dtype))
print('Test_images.shape: {}, of {}'.format(test_images_3ch.shape, test_images_3ch.dtype))

"""
Explanation: Resizing Image Data for Modeling
The minimum image size expected by the VGG model is 32x32 so we need to resize our images
End of explanation
"""


fig, ax = plt.subplots(2, 5, figsize=(12, 6))
c = 0
for i in range(10):
  idx = i // 5
  idy = i % 5 
  ax[idx, idy].imshow(train_images_3ch[i]/255.)
  ax[idx, idy].set_title(class_names[train_labels[i]])

"""
Explanation: View Sample Data
End of explanation
"""


# define input shape
INPUT_SHAPE = (32, 32, 3)

# get the VGG19 model
vgg_layers = tf.keras.applications.vgg19.VGG19(weights='imagenet', include_top=False, 
                                               input_shape=INPUT_SHAPE)

vgg_layers.summary()

"""
Explanation: Build CNN Model Architecture
We will now build our CNN model architecture customizing the VGG-19 model.
Build Cut-VGG19 Model
End of explanation
"""


# Fine-tune all the layers
for layer in vgg_layers.layers:
  layer.trainable = True

# Check the trainable status of the individual layers
for layer in vgg_layers.layers:
  print(layer, layer.trainable)

"""
Explanation: Set layers to trainable to enable fine-tuning
End of explanation
"""


# define sequential model
model = tf.keras.models.Sequential()

# Add the vgg convolutional base model
model.add(vgg_layers)

# add flatten layer
model.add(tf.keras.layers.Flatten())

# add dense layers with some dropout
model.add(tf.keras.layers.Dense(256, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=0.3))
model.add(tf.keras.layers.Dense(256, activation='relu'))
model.add(tf.keras.layers.Dropout(rate=0.3))

# add output layer
model.add(tf.keras.layers.Dense(10))

# compile model
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'])

# view model layers
model.summary()

"""
Explanation: Build CNN model on top of VGG19
End of explanation
"""


EPOCHS = 100
train_images_3ch_scaled = train_images_3ch / 255.
es_callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                               patience=2, 
                                               restore_best_weights=True,
                                               verbose=1)

history = model.fit(train_images_3ch_scaled, train_labels,
                    batch_size=32,
                    callbacks=[es_callback], 
                    validation_split=0.1, epochs=EPOCHS,
                    verbose=1)

"""
Explanation: Train CNN Model
End of explanation
"""


import pandas as pd

fig, ax = plt.subplots(1, 2, figsize=(10, 4))

history_df = pd.DataFrame(history.history)
history_df[['loss', 'val_loss']].plot(kind='line', 
                                      ax=ax[0])
history_df[['accuracy', 'val_accuracy']].plot(kind='line', 
                                              ax=ax[1]);

"""
Explanation: Plot Learning Curves
End of explanation
"""


test_images_3ch_scaled = test_images_3ch / 255.
predictions = model.predict(test_images_3ch_scaled)
predictions[:5]

prediction_labels = np.argmax(predictions, axis=1)
prediction_labels[:5]

from sklearn.metrics import confusion_matrix, classification_report

print(classification_report(test_labels, prediction_labels, 
                            target_names=class_names))
pd.DataFrame(confusion_matrix(test_labels, prediction_labels), 
             index=class_names, columns=class_names)

"""
Explanation: Evaluate Model Performance on Organic Test Data
Here we check the performance of our pre-trained CNN model on the organic test data (without introducing any perturbations)
End of explanation
"""


def get_model_preds(input_image, class_names_map, model):
  preds = model.predict(input_image)
  # Convert logits to probabilities by taking softmax.
  probs = np.exp(preds) / np.sum(np.exp(preds))
  top_idx = np.argsort(-probs)[0][0]
  top_prob = -np.sort(-probs)[0][0]
  top_class = np.array(class_names_map)[top_idx]
  return top_class, top_prob


def generate_adversarial_pattern(input_image, image_label_idx, model, loss_func):
  with tf.GradientTape() as tape:
    tape.watch(input_image)
    prediction = model(input_image)
    loss = loss_func(image_label_idx, prediction)
  # Get the gradients of the loss w.r.t to the input image.
  gradient = tape.gradient(loss, input_image)
  # Get the sign of the gradients to create the perturbation
  signed_grad = tf.sign(gradient)
  return signed_grad


def perform_adversarial_attack_fgsm(input_image, image_label_idx, cnn_model, class_names_map, loss_func, eps=0.01):
  # basic image shaping
  input_image = np.array([input_image])
  tf_img = tf.convert_to_tensor(input_image)
  # predict class before adversarial attack
  ba_pred_class, ba_pred_prob = get_model_preds(tf_img, class_names_map, cnn_model)
  # generate adversarial image
  adv_pattern = generate_adversarial_pattern(tf_img, image_label_idx, model, loss_func)
  clip_adv_pattern = tf.clip_by_value(adv_pattern, clip_value_min=0., clip_value_max=1.)

  perturbed_img = tf_img + (eps * adv_pattern)
  perturbed_img = tf.clip_by_value(perturbed_img, clip_value_min=0., clip_value_max=1.)
  # predict class after adversarial attack
  aa_pred_class, aa_pred_prob = get_model_preds(perturbed_img, class_names_map, cnn_model)

  # visualize results
  fig, ax = plt.subplots(1, 3, figsize=(15, 4))
  ax[0].imshow(tf_img[0].numpy())
  ax[0].set_title('Before Adversarial Attack\nTrue:{}  Pred:{}  Prob:{:.3f}'.format(class_names_map[image_label_idx],
                                                                                    ba_pred_class,
                                                                                    round(ba_pred_prob, 3)))
  
  ax[1].imshow(clip_adv_pattern[0].numpy())
  ax[1].set_title('Adversarial Pattern -  EPS:{}'.format(eps))
  
  ax[2].imshow(perturbed_img[0].numpy())
  ax[2].set_title('After Adversarial Attack\nTrue:{}  Pred:{}  Prob:{:.3f}'.format(class_names_map[image_label_idx],
                                                                                    aa_pred_class,
                                                                                    aa_pred_prob))

"""
Explanation: Adversarial Attacks with Fast Gradient Sign Method (FGSM)
What is an adversarial example?
Adversarial examples are specialised inputs created with the purpose of confusing a neural network, resulting in the misclassification of a given input. These notorious inputs are indistinguishable to the human eye, but cause the network to fail to identify the contents of the image. There are several types of such attacks, however, here the focus is on the fast gradient sign method attack, which is a white box attack whose goal is to ensure misclassification. A white box attack is where the attacker has complete access to the model being attacked. One of the most famous examples of an adversarial image shown below is taken from the aforementioned paper.

<font size=2>Source: Explaining and Harnessing Adversarial Examples, Goodfellow et al., 2014</font>
Here, starting with the image of a panda, the attacker adds small perturbations (distortions) to the original image, which results in the model labelling this image as a gibbon, with high confidence. The process of adding these perturbations is explained below.
Fast gradient sign method
The fast gradient sign method works by using the gradients of the neural network to create an adversarial example. For an input image, the method uses the gradients of the loss with respect to the input image to create a new image that maximises the loss. This new image is called the adversarial image. This can be summarised using the following expression:
$$adv_x = x + \epsilon*\text{sign}(\nabla_xJ(\theta, x, y))$$
where 

adv_x : Adversarial image.
x : Original input image.
y : Original input label.
$\epsilon$ : Multiplier to ensure the perturbations are small.
$\theta$ : Model parameters.
$J$ : Loss.

The gradients are taken with respect to the input image because the objective is to create an image that maximizes the loss. A method to accomplish this is to find how much each pixel in the image contributes to the loss value, and add a perturbation accordingly. This works pretty fast because it is easy to find how much each input pixel contributes to the loss by using the chain rule and finding the required gradients. Since our goal here is to attack a model that has already been trained, the gradient is not taken with respect to the trainable variables, i.e., the model parameters, which are now frozen.
So let's try and fool our pretrained VGG19 model.
Utility Functions for FGSM


get_model_preds(...): Helps in getting the top predicted class label and probability of an input image based on a specific trained CNN model


generate_adversarial_pattern(...): Helps in getting the gradients and the sign of the gradients w.r.t the input image and the trained CNN model


perform_adversarial_attack_fgsm(...): Create perturbations which will be used to distort the original image resulting in an adversarial image by adding epsilon to the gradient signs (can be added to gradients also) and then showcase model performance on the same
End of explanation
"""


scc = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

"""
Explanation: Get Loss Function for our problem
We use Sparse Categorical Crossentropy here as we focus on a multi-class classification problem
End of explanation
"""


def show_adv_attack_example(image_idx, image_dataset, 
                            image_labels, cnn_model,
                            class_names, loss_fn, eps):
  sample_apparel_img = image_dataset[image_idx]
  sample_apparel_labelidx = image_labels[image_idx]
  perform_adversarial_attack_fgsm(input_image=sample_apparel_img, 
                                  image_label_idx=sample_apparel_labelidx, 
                                  cnn_model=cnn_model, 
                                  class_names_map=class_names,
                                  loss_func=loss_fn, eps=eps)

show_adv_attack_example(6, test_images_3ch_scaled, 
                        test_labels, model,
                        class_names, scc, 0.05)

show_adv_attack_example(60, test_images_3ch_scaled, 
                        test_labels, model,
                        class_names, scc, 0.05)

show_adv_attack_example(500, test_images_3ch_scaled, 
                        test_labels, model,
                        class_names, scc, 0.05)

show_adv_attack_example(560, test_images_3ch_scaled, 
                        test_labels, model,
                        class_names, scc, 0.05)

"""
Explanation: Adversarial Attack Examples
Here we look at a few examples of applying thte FGSM adversarial attack on sample apparel images and how it affects our model predictions. We create a simple wrapper over our perform_adversarial_attack_fgsm function to try it out on sample images.
End of explanation
"""


!pip install neural-structured-learning

import neural_structured_learning as nsl

"""
Explanation: Adversarial Learning with Neural Structured Learning
We will now leverage Neural Structured Learning (NSL) to train an adversarial-regularized VGG-19 model.
Install NSL Dependency
End of explanation
"""


adv_multiplier = 0.45
adv_step_size = 0.95
adv_grad_norm = 'l2'

adversarial_config = nsl.configs.make_adv_reg_config(
  multiplier=adv_multiplier,
  adv_step_size=adv_step_size,
  adv_grad_norm=adv_grad_norm
)

adversarial_config

"""
Explanation: Adversarial Learning Configs

adv_multiplier: The weight of adversarial loss in the training
objective, relative to the labeled loss.
adv_step_size: The magnitude of adversarial perturbation.
adv_grad_norm: The norm to measure the magnitude of adversarial
perturbation.

Adversarial Neighbors are created leveraging the above config settings.
adv_neighbor = input_features + adv_step_size * gradient where adv_step_size is the step size (analogous to learning rate) for searching/calculating adversarial neighbors.
End of explanation
"""


vgg_layers = tf.keras.applications.vgg19.VGG19(weights='imagenet', include_top=False, 
                                               input_shape=INPUT_SHAPE)

# Fine-tune all the layers
for layer in vgg_layers.layers:
  layer.trainable = True

# Check the trainable status of the individual layers
for layer in vgg_layers.layers:
  print(layer, layer.trainable)

# define sequential model
base_model = tf.keras.models.Sequential()

# Add the vgg convolutional base model
base_model.add(vgg_layers)

# add flatten layer
base_model.add(tf.keras.layers.Flatten())

# add dense layers with some dropout
base_model.add(tf.keras.layers.Dense(256, activation='relu'))
base_model.add(tf.keras.layers.Dropout(rate=0.3))
base_model.add(tf.keras.layers.Dense(256, activation='relu'))
base_model.add(tf.keras.layers.Dropout(rate=0.3))

# add output layer
base_model.add(tf.keras.layers.Dense(10))

"""
Explanation: Feel free to play around with the hyperparameters and observe model performance
Fine-tuning VGG-19 CNN Model with Adversarial Learning - Adversarial Model
Create Base Model Architecture
End of explanation
"""


adv_model = nsl.keras.AdversarialRegularization(
  base_model,
  label_keys=['label'],
  adv_config=adversarial_config
)

adv_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'])

"""
Explanation: Setup Adversarial Model with Adversarial Regularization on Base Model
End of explanation
"""


from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(train_images_3ch_scaled, 
                                                  train_labels, 
                                                  test_size=0.1, 
                                                  random_state=42)
batch_size = 256

train_data = tf.data.Dataset.from_tensor_slices(
  {'input': X_train, 
    'label': tf.convert_to_tensor(y_train, dtype='float32')}).batch(batch_size)

val_data = tf.data.Dataset.from_tensor_slices(
  {'input': X_val, 
    'label': tf.convert_to_tensor(y_val, dtype='float32')}).batch(batch_size)


val_steps = X_val.shape[0] / batch_size

"""
Explanation: Format Training / Validation data into TF Datasets
End of explanation
"""


EPOCHS = 100

es_callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                                               patience=2, 
                                               restore_best_weights=False,
                                               verbose=1)

history = adv_model.fit(train_data, validation_data=val_data,
                        validation_steps=val_steps, 
                        batch_size=batch_size,
                        callbacks=[es_callback], 
                        epochs=EPOCHS,
                        verbose=1)

"""
Explanation: Train Model
End of explanation
"""


fig, ax = plt.subplots(1, 2, figsize=(10, 4))

history_df = pd.DataFrame(history.history)
history_df[['loss', 'val_loss']].plot(kind='line', 
                                      ax=ax[0])
history_df[['sparse_categorical_accuracy', 
            'val_sparse_categorical_accuracy']].plot(kind='line', 
                                                     ax=ax[1]);

"""
Explanation: Visualize Learning Curves
End of explanation
"""


predictions = adv_model.base_model.predict(test_images_3ch_scaled)
prediction_labels = np.argmax(predictions, axis=1)
print(classification_report(test_labels, prediction_labels, 
                            target_names=class_names))
pd.DataFrame(confusion_matrix(test_labels, prediction_labels), 
             index=class_names, columns=class_names)

"""
Explanation: VGG-19 Adversarial Model Performance on Organic Test Dataset
Here we check the performance of our adversarially-trained CNN model on the organic test data (without introducing any perturbations)
End of explanation
"""


def generate_perturbed_images(input_images, image_label_idxs, model, loss_func, eps=0.01):
  perturbed_images = []
  # don't use list on large data - used just to view fancy progress-bar
  for image, label in tqdm(list(zip(input_images, image_label_idxs))): 
    image = tf.convert_to_tensor(np.array([image]))
    adv_pattern = generate_adversarial_pattern(image, label, model, loss_func)
    perturbed_img = image + (eps * adv_pattern)
    perturbed_img = tf.clip_by_value(perturbed_img, clip_value_min=0., clip_value_max=1.)[0]
    perturbed_images.append(perturbed_img)

  return tf.convert_to_tensor(perturbed_images)

"""
Explanation: Almost similar performance as our non-adversarial trained CNN model!
Generate Adversarial Attacks (FGSM) on Test Data to create Perturbed Test Dataset
Here we create a helper function to help us create a perturbed dataset using a specific adversarial epsilon multiplier.
End of explanation
"""


perturbed_test_imgs = generate_perturbed_images(input_images=test_images_3ch_scaled, 
                                                image_label_idxs=test_labels, model=model, 
                                                loss_func=scc, eps=0.05)

"""
Explanation: Generate a Perturbed Test Dataset
We generate a perturbed version of the test dataset using an epsilion multiplier of 0.05 to test the performance of our base VGG model and adversarially-trained VGG model shortly.
End of explanation
"""


predictions = model.predict(perturbed_test_imgs)
prediction_labels = np.argmax(predictions, axis=1)
print(classification_report(test_labels, prediction_labels, 
                            target_names=class_names))
pd.DataFrame(confusion_matrix(test_labels, prediction_labels), 
             index=class_names, columns=class_names)

"""
Explanation: VGG-19 Base Model performance on Perturbed Test Dataset
Let's look at the performance of our base VGG-19 model on the perturbed dataset.
End of explanation
"""


predictions = adv_model.base_model.predict(perturbed_test_imgs)
prediction_labels = np.argmax(predictions, axis=1)
print(classification_report(test_labels, prediction_labels, 
                            target_names=class_names))
pd.DataFrame(confusion_matrix(test_labels, prediction_labels), 
             index=class_names, columns=class_names)

"""
Explanation: We can see that the performance of the base VGG-19 (non adversarial-trained) model reduces by almost 50% on the perturbed test dataset, bringing a powerful ImageNet winning model to its knees!
VGG-19 Adversarial Model performance on Perturbed Test Dataset
Evaluating our adversarial trained CNN model on the test dataset with perturbations. We see an approx. 38% jump in performance!
End of explanation
"""


f, ax = plt.subplots(2, 5, figsize=(30, 15))
for idx, i in enumerate([6, 7, 8 , 9, 10, 11, 95, 99, 29, 33]):
  idx_x = idx // 5
  idx_y = idx % 5 

  sample_apparel_idx = i
  sample_apparel_img =  tf.convert_to_tensor([perturbed_test_imgs[sample_apparel_idx]])
  sample_apparel_labelidx = test_labels[sample_apparel_idx]

  bm_pred = get_model_preds(input_image=sample_apparel_img, 
                            class_names_map=class_names, 
                            model=model)[0]
  am_pred = get_model_preds(input_image=sample_apparel_img, 
                            class_names_map=class_names, 
                            model=adv_model.base_model)[0]

  ax[idx_x, idx_y].imshow(sample_apparel_img[0])
  ax[idx_x, idx_y].set_title('True Label:{}\nBase VGG Model Pred:{}\nAdversarial Reg. Model Pred:{}'.format(class_names[sample_apparel_labelidx],
                                                                                                            bm_pred,
                                                                                                            am_pred))


"""
Explanation: Compare Model Performances on Sample Perturbed Test Examples
End of explanation
"""


