title: Deep learning model predicting same output for every input
body: <p>My model should be able to multi-label each {image, text} into 4 classes with each class having {4,4,4,2} subclasses respectively. So for each input, the output should be a subclass of each class.</p>
  <p>Output should be of length 4.</p>
  <p>I'm labelling the images, with the input as {image, text content in the image} So the model should take input as both text-content present in the image, image data.</p>
  <p>To process the image data, I'm importing resnet50 as my base model.</p>
  <pre><code>base_model_1 = tf.keras.applications.ResNet50(input_shape=X[0].shape,
                                             include_top=False,
                                             weights='imagenet')
  base_model_1.trainable = True
  </code></pre>
  <p>Later joined few other layers to the <strong>base_model_1</strong>, so I can merge the image_model with text_model.</p>
  <p>Image model be like :</p>
  <pre><code>def image_model():
      image_input = tf.keras.Input(shape=(100, 100, 3), name = 'image_input')
      image_layers = preprocess_input(image_input)
      layer_bm_1 = base_model_1(image_layers, training=True)
      dropout_layer = Dropout(0.2)(layer_bm_1)
      layer_bm_1 = Conv2D(2048, kernel_size=3,padding='valid')(image_layers)
      dropout_layer = Dropout(0.3)(layer_bm_1)
      layer_bm_1 = Dense(1024)(layer_bm_1)
      dropout_layer = Dropout(0.5)(layer_bm_1)
      image_layers = GlobalAveragePooling2D()(layer_bm_1)
      image_layers = Dropout(0.5, name = 'dropout_layer')(image_layers)
      return image_input, image_layers
  </code></pre>
  <p>Text model be like :</p>
  <pre><code>def text_model():
      text_input = tf.keras.Input(shape=(1,),dtype=tf.string, name='text')
      text_layers = vectorize_layer(text_input)
      text_layers = tf.keras.layers.Embedding(vocab_size, embedding_dim,input_length= sequence_length , name=&quot;embedding&quot;)(text_layers)
      
      text_layers = tf.keras.layers.LSTM(1024, activation='relu', return_sequences=False)(text_layers)
      text_layers = tf.keras.layers.Dense(1024, activation=&quot;relu&quot;)(text_layers)
      #text_layers = tf.keras.layers.GlobalAveragePooling1D()(text_layers)
  
      return text_input, text_layers
  
  </code></pre>
  <p>The output of both layers are concatenated.</p>
  <pre><code>def model(image_layer, text_layer, image_input, text_input):
  
      concatenate = tf.keras.layers.concatenate([layer_1, layer_2])
  
      semi_final_layer = tf.keras.layers.Dense(2048, activation='relu')(concatenate)
      semi_final_layer = tf.keras.layers.Dense(1024, activation='relu')(semi_final_layer)
  
      prediction_layer_1 = tf.keras.layers.Dense(516, activation='relu')(semi_final_layer)
      prediction_layer_2 = tf.keras.layers.Dense(516, activation='relu')(semi_final_layer)
      prediction_layer_3 = tf.keras.layers.Dense(516, activation='relu')(semi_final_layer)
      prediction_layer_4 = tf.keras.layers.Dense(516, activation='relu')(semi_final_layer)
  
  
      prediction_layer_1 = tf.keras.layers.Dense(254, activation='relu')(prediction_layer_1)
      prediction_layer_2 = tf.keras.layers.Dense(254, activation='relu')(prediction_layer_2)
      prediction_layer_3 = tf.keras.layers.Dense(254, activation='relu')(prediction_layer_3)
      prediction_layer_4 = tf.keras.layers.Dense(254, activation='relu')(prediction_layer_4)
  
  
      output_1 = tf.keras.layers.Dense(4, activation='softmax', name = 'humour')(prediction_layer_1)
      output_2 = tf.keras.layers.Dense(4, activation='softmax', name = 'sarcasm')(prediction_layer_2)
      output_3 = tf.keras.layers.Dense(4, activation='softmax', name = 'offensive')(prediction_layer_3)
      output_4 = tf.keras.layers.Dense(2, activation='softmax', name = 'motivational')(prediction_layer_4)
  
      model = tf.keras.Model(inputs = [image_input, text_input] , 
                            outputs = [output_1, output_2, output_3, output_4])
      return model
  </code></pre>
  <pre><code># Function for decaying the learning rate.
  def decay(epoch):
    if epoch &lt; 3:
      return 1e-4
    elif epoch &gt;= 3 and epoch &lt; 5:
      return 1e-5
    else:
      return 1e-6
  
  # Callback for printing the LR at the end of each epoch.
  class PrintLR(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
      print('\nLearning rate for epoch {} is {}'.format(epoch + 1,
                                                        model.optimizer.lr.numpy()))
  
  callbacks = [
      tf.keras.callbacks.TensorBoard(log_dir='./logs'),
      tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_prefix,
                                         save_weights_only=True),
      tf.keras.callbacks.LearningRateScheduler(decay),
      PrintLR()
  ]
  </code></pre>
  <p>After training the above model with batchsize = 32, epochs = 10.
  Total dataset contain = 7000 datapoints, splitted into training, validation, testing data in 0.7, 0.2, 0.1 ratio respectively.</p>
  <p>Training, Validation accuracy, loss staying the same:</p>
  <pre><code>Epoch 1:
  loss: 4.3440 - humour_loss: 1.3068 - sarcasm_loss: 1.1911 - offensive_loss: 1.1937 - motivational_loss: 0.6525 - humour_categorical_accuracy: 0.3384 - sarcasm_categorical_accuracy: 0.5011 - offensive_categorical_accuracy: 0.3709 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2886 - val_humour_loss: 1.2852 - val_sarcasm_loss: 1.1741 - val_offensive_loss: 1.1755 - val_motivational_loss: 0.6537 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-04
  
  Epoch 2/10
  loss: 4.3215 - humour_loss: 1.3028 - sarcasm_loss: 1.1843 - offensive_loss: 1.1839 - motivational_loss: 0.6504 - humour_categorical_accuracy: 0.3491 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3769 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2617 - val_humour_loss: 1.2763 - val_sarcasm_loss: 1.1710 - val_offensive_loss: 1.1675 - val_motivational_loss: 0.6469 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3832 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-04
  
  Epoch 3/10
  loss: 4.3259 - humour_loss: 1.3029 - sarcasm_loss: 1.1861 - offensive_loss: 1.1851 - motivational_loss: 0.6519 - humour_categorical_accuracy: 0.3409 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3744 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2731 - val_humour_loss: 1.2760 - val_sarcasm_loss: 1.1784 - val_offensive_loss: 1.1723 - val_motivational_loss: 0.6465 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-04
  
  Epoch 4/10
  loss: 4.3079 - humour_loss: 1.2993 - sarcasm_loss: 1.1796 - offensive_loss: 1.1799 - motivational_loss: 0.6491 - humour_categorical_accuracy: 0.3458 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3756 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2787 - val_humour_loss: 1.2806 - val_sarcasm_loss: 1.1780 - val_offensive_loss: 1.1729 - val_motivational_loss: 0.6472 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-05
  
  Epoch 5/10
  loss: 4.3078 - humour_loss: 1.2990 - sarcasm_loss: 1.1808 - offensive_loss: 1.1796 - motivational_loss: 0.6485 - humour_categorical_accuracy: 0.3456 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3782 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2705 - val_humour_loss: 1.2787 - val_sarcasm_loss: 1.1744 - val_offensive_loss: 1.1708 - val_motivational_loss: 0.6467 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-05
  
  Epoch 6/10
  loss: 4.3078 - humour_loss: 1.2995 - sarcasm_loss: 1.1803 - offensive_loss: 1.1798 - motivational_loss: 0.6482 - humour_categorical_accuracy: 0.3456 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3733 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2710 - val_humour_loss: 1.2790 - val_sarcasm_loss: 1.1745 - val_offensive_loss: 1.1709 - val_motivational_loss: 0.6467 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-06
  
  Epoch 7/10
  loss: 4.3059 - humour_loss: 1.2983 - sarcasm_loss: 1.1792 - offensive_loss: 1.1798 - motivational_loss: 0.6486 - humour_categorical_accuracy: 0.3453 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3805 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2701 - val_humour_loss: 1.2786 - val_sarcasm_loss: 1.1742 - val_offensive_loss: 1.1706 - val_motivational_loss: 0.6467 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-06
  
  Epoch 8/10
  loss: 4.3060 - humour_loss: 1.2986 - sarcasm_loss: 1.1793 - offensive_loss: 1.1796 - motivational_loss: 0.6485 - humour_categorical_accuracy: 0.3456 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3765 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2701 - val_humour_loss: 1.2786 - val_sarcasm_loss: 1.1742 - val_offensive_loss: 1.1706 - val_motivational_loss: 0.6467 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-06
  
  Epoch 9/10
  loss: 4.3048 - humour_loss: 1.2987 - sarcasm_loss: 1.1789 - offensive_loss: 1.1792 - motivational_loss: 0.6480 - humour_categorical_accuracy: 0.3456 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3767 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2698 - val_humour_loss: 1.2785 - val_sarcasm_loss: 1.1742 - val_offensive_loss: 1.1705 - val_motivational_loss: 0.6466 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-06
  
  Epoch 10/10
  loss: 4.3065 - humour_loss: 1.2980 - sarcasm_loss: 1.1804 - offensive_loss: 1.1796 - motivational_loss: 0.6484 - humour_categorical_accuracy: 0.3456 - sarcasm_categorical_accuracy: 0.5034 - offensive_categorical_accuracy: 0.3827 - motivational_categorical_accuracy: 0.6486 - val_loss: 4.2702 - val_humour_loss: 1.2786 - val_sarcasm_loss: 1.1743 - val_offensive_loss: 1.1706 - val_motivational_loss: 0.6467 - val_humour_categorical_accuracy: 0.3885 - val_sarcasm_categorical_accuracy: 0.4978 - val_offensive_categorical_accuracy: 0.3975 - val_motivational_categorical_accuracy: 0.6517 - lr: 1.0000e-06
  
  </code></pre>
  <p>The output for a set of inputs from test_Data is :</p>
  <pre><code>[0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  [0.0219533  0.06215543 0.3258566  0.46769002 0.12721306]
  </code></pre>
  <p>The above output is the same for every test_input.</p>
  <p>Let me know any ideas to tackle the issue here.</p>
  <p>Edit 1 :
  input shape for image : (100,100,3)</p>