# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: py3-gpu-jupyter-ds
#     language: python
#     name: py3-gpu-jupyter-ds
# ---

# + [markdown] id="hRTa3Ee15WsJ"
# # Transfer learning and fine-tuning base EfficientNet_B0 model

# + id="gjWM9Lkqm0C1"
# # !pip install tf-nightly

# + id="TqOt6Sv7AsMi"
import matplotlib.pyplot as plt
import numpy as np
import os
import tensorflow as tf

from tensorflow.keras.preprocessing import image_dataset_from_directory

# + [markdown] id="v77rlkCKW0IJ"
# ## Data preprocessing

# + [markdown] id="0GoKGm1duzgk"
# ### Data download

# + [markdown] id="vHP9qMJxt2oz"
# In this tutorial, you will use a dataset containing several thousand images of cats and dogs. Download and extract a zip file containing the images, then create a `tf.data.Dataset` for training and validation using the `tf.keras.preprocessing.image_dataset_from_directory` utility. You can learn more about loading images in this [tutorial](https://www.tensorflow.org/tutorials/load_data/images).

# + id="ro4oYaEmxe4r"
_URL = 'https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip'
path_to_zip = tf.keras.utils.get_file('cats_and_dogs.zip', origin=_URL, extract=True)
PATH = os.path.join(os.path.dirname(path_to_zip), 'cats_and_dogs_filtered')

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'validation')

BATCH_SIZE = 32
IMG_SIZE = (224, 224)

train_dataset = image_dataset_from_directory(train_dir,
                                             shuffle=True,
                                             batch_size=BATCH_SIZE,
                                             image_size=IMG_SIZE)

# + id="cAvtLwi7_J__"
validation_dataset = image_dataset_from_directory(validation_dir,
                                                  shuffle=True,
                                                  batch_size=BATCH_SIZE,
                                                  image_size=IMG_SIZE)

# +
# # ?image_dataset_from_directory
# -

validation_dir
# !ls /root/.keras/datasets/cats_and_dogs_filtered/validation

# + [markdown] id="yO1Q2JaW5sIy"
# Show the first nine images and labels from the training set:

# + id="K5BeQyKThC_Y"
class_names = train_dataset.class_names

plt.figure(figsize=(10, 10))
# 从train_dataset里面拿一个batch size数量的图片和正解标签
for images, labels in train_dataset.take(1):
  for i in range(9):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(images[i].numpy().astype("uint8"))
    plt.title(class_names[labels[i]])
    plt.axis("off")

# + [markdown] id="EZqCX_mpV3Mx"
# As the original dataset doesn't contains a test set, you will create one. To do so, determine how many batches of data are available in the validation set using ```tf.data.experimental.cardinality```, then move 20% of them to a test set.

# + id="uFFIYrTFV9RO"
val_batches = tf.data.experimental.cardinality(validation_dataset)
# 验证集中一共有32个批次的batch数据（一个batch里面有32张图片），32个批次分成26和6，2个部分（验证和测试）
test_dataset = validation_dataset.take(val_batches//5)
validation_dataset = validation_dataset.skip(val_batches//5)

# + id="Q9pFlFWgBKgH"
print('Number of validation batches: %d' % tf.data.experimental.cardinality(validation_dataset))
print('Number of test batches: %d' % tf.data.experimental.cardinality(test_dataset))

# + [markdown] id="MakSrdd--RKg"
# ### Configure the dataset for performance

# + [markdown] id="22XWC7yjkZu4"
# Use buffered prefetching to load images from disk without having I/O become blocking. To learn more about this method see the [data performance](https://www.tensorflow.org/guide/data_performance) guide.

# + id="p3UUPdm86LNC"
AUTOTUNE = tf.data.experimental.AUTOTUNE

train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)
validation_dataset = validation_dataset.prefetch(buffer_size=AUTOTUNE)
test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)
# -

len(train_dataset),len(validation_dataset),len(test_dataset)

# + [markdown] id="MYfcVwYLiR98"
# ### Use data augmentation

# + [markdown] id="bDWc5Oad1daX"
# When you don't have a large image dataset, it's a good practice to artificially introduce sample diversity by applying random, yet realistic, transformations to the training images, such as rotation and horizontal flipping. This helps expose the model to different aspects of the training data and reduce [overfitting](https://www.tensorflow.org/tutorials/keras/overfit_and_underfit). You can learn more about data augmentation in this [tutorial](https://www.tensorflow.org/tutorials/images/data_augmentation).
# -

data_augmentation=tf.keras.Sequential([
#     水平 垂直 翻转
    tf.keras.layers.experimental.preprocessing.RandomFlip(mode='horizontal'),
#     旋转
    tf.keras.layers.experimental.preprocessing.RandomRotation(0.4),
#     缩放
    tf.keras.layers.experimental.preprocessing.RandomZoom(.5, .2),
    #     图片上下shift20%，左右shift20%
    tf.keras.layers.experimental.preprocessing.RandomTranslation(height_factor=0.2, width_factor=0.2),
#     独立的调整每个图片每个通道的对比度
    tf.keras.layers.experimental.preprocessing.RandomContrast(factor=0.1),
])

# + [markdown] id="s9SlcbhrarOO"
# Note: These layers are active only during training, when you call `model.fit`. They are inactive when the model is used in inference mode in `model.evaulate` or `model.fit`.

# + [markdown] id="9mD3rE2Lm7-d"
# Let's repeatedly apply these layers to the same image and see the result.
# -

image, label = next(iter(validation_dataset))
len(image)

# + id="aQullOUHkm67"
plt.figure(figsize=(3, 3))
first_image = image[0]
plt.subplot(1,1,1)
plt.imshow(first_image.numpy().astype("uint8"))
plt.title("original")
plt.axis('off')
plt.show()
plt.figure(figsize=(10, 10))
for i in range(9):
    ax = plt.subplot(3, 3, i + 1)
    augmented_image = data_augmentation(tf.expand_dims(first_image, 0))
    plt.imshow(augmented_image[0] / 255)
    plt.title("Data Augmentation")
    plt.axis('off')

# + [markdown] id="bAywKtuVn8uK"
# ### Rescale pixel values

# + [markdown] id="xnr81qRMzcs5"
# Note: Alternatively, you could rescale pixel values from `[0,255]` to `[-1, 1]` using a [Rescaling](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/Rescaling) layer.

# + id="R2NyJn4KQMux"
rescale = tf.keras.layers.experimental.preprocessing.Rescaling(1./127.5, offset= -1)
# -

# 通过preprocess_input处理，数据变成了-1到1之间的数值
a=rescale(images[0])
print(np.min(a),np.max(a))
# 原始图片是0到255之间的数值
np.min(images[0]),np.max(images[0])

# + [markdown] id="OkH-kazQecHB"
# ## Create the base model from the pre-trained convnets

# + id="19IQ2gqneqmS"
# Create the base model from the pre-trained model EfficientNetB0
# 设定维度为 (224, 224, 3)
IMG_SHAPE = IMG_SIZE + (3, )
base_model = tf.keras.applications.EfficientNetB0(input_shape=IMG_SHAPE,
                                                  include_top=False,
                                                  weights='imagenet',
                                                  drop_connect_rate=0.4)

# + [markdown] id="AqcsxoJIEVXZ"
# This feature extractor converts each `160x160x3` image into a `5x5x1280` block of features. Let's see what it does to an example batch of images:

# + id="Y-2LJL0EEUcx"
image_batch, label_batch = next(iter(train_dataset))
feature_batch = base_model(image_batch)
print(feature_batch.shape)

# + [markdown] id="rlx56nQtfe8Y"
# ## Feature extraction
# In this step, you will freeze the convolutional base created from the previous step and to use as a feature extractor. Additionally, you add a classifier on top of it and train the top-level classifier.

# + [markdown] id="CnMLieHBCwil"
# ### Freeze the convolutional base

# + [markdown] id="7fL6upiN3ekS"
# It is important to freeze the convolutional base before you compile and train the model. Freezing (by setting layer.trainable = False) prevents the weights in a given layer from being updated during training. MobileNet V2 has many layers, so setting the entire model's `trainable` flag to False will freeze all of them.

# + id="OTCJH4bphOeo"
base_model.trainable = False

# + [markdown] id="jsNHwpm7BeVM"
# ### Important note about BatchNormalization layers
#
# Many models contain `tf.keras.layers.BatchNormalization` layers. This layer is a special case and precautions should be taken in the context of fine-tuning, as shown later in this tutorial. 
#
# When you set `layer.trainable = False`, the `BatchNormalization` layer will run in inference mode, and will not update its mean and variance statistics. 
#
# When you unfreeze a model that contains BatchNormalization layers in order to do fine-tuning, you should keep the BatchNormalization layers in inference mode by passing `training = False` when calling the base model. Otherwise, the updates applied to the non-trainable weights will destroy what the model has learned.
#
# For details, see the [Transfer learning guide](https://www.tensorflow.org/guide/keras/transfer_learning).

# + id="KpbzSmPkDa-N"
# Let's take a look at the base model architecture
base_model.summary()
# -

# 查看层数和层的名字
len(base_model.layers),base_model.layers[0].name

# + [markdown] id="wdMRM8YModbk"
# ### Add a classification head

# + [markdown] id="QBc31c4tMOdH"
# To generate predictions from the block of features, average over the spatial `5x5` spatial locations, using a `tf.keras.layers.GlobalAveragePooling2D` layer to convert the features to  a single 1280-element vector per image.

# + id="dLnpMF5KOALm"
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
feature_batch_average = global_average_layer(feature_batch)
print(feature_batch_average.shape)
# -

# 一共有1280个元素，通过这些元素来判断是猫还是狗
feature_batch_average[0]

# + [markdown] id="O1p0OJBR6dOT"
# Apply a `tf.keras.layers.Dense` layer to convert these features into a single prediction per image. You don't need an activation function here because this prediction will be treated as a `logit`, or a raw prediction value.  Positive numbers predict class 1, negative numbers predict class 0.

# + id="Wv4afXKj6cVa"
prediction_layer = tf.keras.layers.Dense(1)
prediction_batch = prediction_layer(feature_batch_average)
print(prediction_batch.shape)
# -

prediction_batch[0]

# + [markdown] id="HXvz-ZkTa9b3"
# 通过使用functional API链接 data augmentation, rescaling, base_model and feature extractor layers   
# Build a model by chaining together the data augmentation, rescaling, base_model and feature extractor layers using the [Keras Functional API](https://www.tensorflow.org/guide/keras/functional). As previously mentioned, use training=False as our model contains a BatchNormalization layer.

# + id="DgzQX6Veb2WT"
# 使用functional API链接各个层
inputs = tf.keras.Input(shape=(IMG_SIZE[0], IMG_SIZE[1], 3))
x = data_augmentation(inputs)
# 数据标准化，使用的话精度不能收缩
# x = rescale(x)
# training=False表示推论模式，training=True表示训练模式
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = tf.keras.layers.BatchNormalization()(x)
outputs = prediction_layer(x)
model = tf.keras.Model(inputs, outputs)
# -

model.summary()

# + [markdown] id="g0ylJXE_kRLi"
# ### Compile the model
#
# Compile the model before training it. Since there are two classes, use a binary cross-entropy loss with `from_logits=True` since the model provides a linear output.

# + id="RpR8HdyMhukJ"
base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

# + id="I8ARiyMFsgbH"
model.summary()

# + [markdown] id="lxOcmVr0ydFZ"
# The 2.5M parameters in MobileNet are frozen, but there are 1.2K _trainable_ parameters in the Dense layer.  These are divided between two `tf.Variable` objects, the weights and biases.

# + id="krvBumovycVA"
len(model.trainable_variables)
# -

model.trainable_variables

# + [markdown] id="RxvgOYTDSWTx"
# ### Train the model
#
# After training for 10 epochs, you should see ~94% accuracy on the validation set.
#

# + id="Om4O3EESkab1"
initial_epochs = 10

loss0, accuracy0 = model.evaluate(validation_dataset)

# + id="8cYT1c48CuSd"
print("initial loss: {:.2f}".format(loss0))
print("initial accuracy: {:.2f}".format(accuracy0))

# + id="JsaRFlZ9B6WK"
history = model.fit(train_dataset,
                    epochs=initial_epochs,
                    validation_data=validation_dataset)

# + [markdown] id="Hd94CKImf8vi"
# ### Learning curves
#
# Let's take a look at the learning curves of the training and validation accuracy/loss when using the MobileNet V2 base model as a fixed feature extractor.

# + id="53OTCh3jnbwV"
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.ylabel('Accuracy')
plt.ylim([min(plt.ylim()),1])
plt.title('Training and Validation Accuracy')

plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.ylabel('Cross Entropy')
plt.ylim([0,1.0])
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()

# + [markdown] id="CqwV-CRdS6Nv"
# ## Fine tuning

# + [markdown] id="CPXnzUK0QonF"
# ### Un-freeze the top layers of the model
#

# + [markdown] id="rfxv_ifotQak"
# All you need to do is unfreeze the `base_model` and set the bottom layers to be un-trainable. Then, you should recompile the model (necessary for these changes to take effect), and resume training.

# + id="4nzcagVitLQm"
base_model.trainable = True

# + id="-4HgVAacRs5v"
# Let's take a look to see how many layers are in the base model
print("Number of layers in the base model: ", len(base_model.layers))

# Fine-tune from this layer onwards
# 因为这种架构包含了每一块到从第一层到最后一层的捷径。不关心块会严重损害最终表现。
# 119层开始是block5a_expand_conv的开始
fine_tune_at = 119

# 冻结了前面的118层，从119开始到末尾可以训练，这样提高了拟合的层数，应该可以获得更好的精度
# Freeze all the layers before the `fine_tune_at` layer
for layer in base_model.layers[:fine_tune_at]:
  layer.trainable =  False

# + [markdown] id="4Uk1dgsxT0IS"
# ### Compile the model
# -

# 如果使用Adam作为激活函数最终测试结果是(推荐)      
# 6/6 [==============================] - 0s 26ms/step - loss: 0.0465 - accuracy: 0.9896   
# Test accuracy : 0.9895833134651184   
#
# 如果使用RMSProp作为激活函数测试结果是   
# 6/6 [==============================] - 0s 26ms/step - loss: 0.0479 - accuracy: 0.9792   
# Test accuracy : 0.979166   

# + id="NtUnaz0WUDva"
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer = tf.keras.optimizers.Adam(lr=base_learning_rate/10),
              metrics=['accuracy'])

# + id="WwBWy7J2kZvA"
model.summary()

# + id="bNXelbMQtonr"
# 可以训练的层数
len(model.trainable_variables)

# + [markdown] id="4G5O4jd6TuAG"
# ### Continue training the model

# + id="ECQLkAsFTlun"
# initial_epoch的目的是从第10次epoch开始，再执行10次epoch
fine_tune_epochs = 7
total_epochs =  initial_epochs + fine_tune_epochs

history_fine = model.fit(train_dataset,
                         epochs=total_epochs,
                         initial_epoch=history.epoch[-1],
                         validation_data=validation_dataset)

# + id="PpA8PlpQKygw"
# 把上一次的acc加上这次history_fine.history['accuracy']的10次结果，一共是20次的结果保存
acc += history_fine.history['accuracy']
val_acc += history_fine.history['val_accuracy']

loss += history_fine.history['loss']
val_loss += history_fine.history['val_loss']

# + id="chW103JUItdk"
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
# plt.ylim([0.8, 1])
plt.ylim([min(plt.ylim()),1])
plt.plot([initial_epochs-1,initial_epochs-1],
          plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.ylim([0, 1.0])
plt.plot([initial_epochs-1,initial_epochs-1],
         plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()

# + [markdown] id="R6cWgjgfrsn5"
# ### Evaluation and prediction
# -

# ### 使用MobileNet模型进行迁移学习，最终的结果

# + id="2KyNhagHwfar"
# 使用MobileNet模型进行迁移学习，最终的结果
loss, accuracy = model.evaluate(test_dataset)
print('Test accuracy :', accuracy)

# + [markdown] id="8UjS5ukZfOcR"
# And now you are all set to use this model to predict if your pet is a cat or dog.
# -

image_batch, label_batch = test_dataset.as_numpy_iterator().next()
len(image_batch)

# + id="RUNoQNgtfNgt"
#Retrieve a batch of images from the test set
image_batch, label_batch = test_dataset.as_numpy_iterator().next()
predictions = model.predict_on_batch(image_batch).flatten()

# Apply a sigmoid since our model returns logits
predictions = tf.nn.sigmoid(predictions)
predictions = tf.where(predictions < 0.5, 0, 1)

print('Predictions:\n', predictions.numpy())
print('Labels:\n', label_batch)

plt.figure(figsize=(10, 10))
for i in range(9):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(image_batch[i].astype("uint8"))
    plt.title(class_names[predictions[i]])
    plt.axis("off")
# -



StarcoderdataJupyterScriptsDedupFiltered