title: Why does loss increase when I unfreeze all weights by transfer learing using Tensorflow2
body: <p><a href="https://i.stack.imgur.com/Xy5r7.png" rel="nofollow noreferrer">loss</a></p>
  <p>I have wrapped resnet50 model of tf.keras with some conv layers and dense layers. I used the pre-trained weights from ImageNet</p>
  <p>First I trained my model while I freezing weights of resnet50. The loss decreased to a good value. The final lr is 0.0025
  Then I unfreeze the weights of resnet50, and started training with a lower lr 0.0001. The loss from beginning is very high. Is the second fit reset the weights?</p>
  <pre><code># create a layer to process gray image to 3-channels image
  input_shape = IMG_SIZE+(1,)
  gray2color_layer = tf.keras.layers.Conv2D(3,1,activation='tanh',input_shape=input_shape)
  Rescaling = tf.keras.layers.experimental.preprocessing.Rescaling(127.5,offset=127.5)
  preprocess_res = tf.keras.applications.resnet50.preprocess_input
  
  # create base model(resnet 50)
  base_model = tf.keras.applications.ResNet50(include_top=False, 
                                             weights='imagenet', 
                                             # input_shape=(200,200,3),
                                             pooling='avg')
  
  # freeze base model
  base_model.trainable = False
  
  # use keras functional API to build the model
  inputs = tf.keras.Input(shape=input_shape,name='pd') #input tensor
  x = gray2color_layer(inputs)
  x = Rescaling(x)
  x = preprocess_res(x)
  x = base_model(x,training=False) # inference mode
  outputs = tf.keras.layers.Dense(6)(x)
  model = tf.keras.Model(inputs, outputs)
  
  # callbacks
  logdir = &quot;./logs/scalars/&quot; + datetime.now().strftime(&quot;%Y%m%d-%H%M%S&quot;)
  Call_TensorBoard = tf.keras.callbacks.TensorBoard(
     log_dir=logdir,
     histogram_freq=0,  # How often to log histogram visualizations
     embeddings_freq=0,  # How often to log embedding visualizations
     update_freq=&quot;epoch&quot;,
  ) 
  Call_EarlyStop = tf.keras.callbacks.EarlyStopping(monitor='val_loss',patience=8)
  Call_LR = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5,verbose=1,
                               patience=3, min_lr=0.0001)
  Call_LR2 = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5,verbose=1,
                               patience=3, min_lr=0.00001)
  
  # compile
  model.compile(optimizer=tf.keras.optimizers.Adam(0.01), 
               loss=loss_res(var=2),
               metrics=[tf.keras.metrics.MeanSquaredError(),mse_percent()])
  
  # fit
  history = model.fit(training_ds,epochs=2,callbacks=[Call_TensorBoard,Call_EarlyStop,Call_LR],validation_data=validation_ds)
  
  # unfreeze weights
  base_model.trainable = True
  
  # compile
  model.compile(optimizer=tf.keras.optimizers.Adam(0.0001), 
               loss=loss_res(var=2),
               metrics=[tf.keras.metrics.MeanSquaredError(),mse_percent()])
  
  # fit
  history_all = model.fit(training_ds,epochs=100,initial_epoch=history.epoch[-1],callbacks=[Call_TensorBoard,Call_EarlyStop,Call_LR2],validation_data=validation_ds)
  </code></pre>