title: Custom Loss Function Leads to High MSE and an offset in the output Keras
body: <p>I am training a neural network for time series regression. The model is</p>
  <pre><code>####################################################################################################################
  # Define ANN Model
  # define two sets of inputs
  acc   =  layers.Input(shape=(3,1,))
  gyro  =  layers.Input(shape=(3,1,))
  
  # the first branch operates on the first input
  x = Conv1D(256, 1, activation='relu')(acc)
  x = Conv1D(128, 1, activation='relu')(x)
  x = Conv1D(128, 1, activation='relu')(x)
  x = MaxPooling1D(pool_size=3)(x)
  x = Model(inputs=acc, outputs=x)
  
  # the second branch opreates on the second input
  y = Conv1D(256, 1, activation='relu')(gyro)
  y = Conv1D(128, 1, activation='relu')(y)
  y = Conv1D(128, 1, activation='relu')(y)
  y = MaxPooling1D(pool_size=3)(y)
  y = Model(inputs=gyro, outputs=y)
  
  
  # combine the output of the three branches
  combined =  layers.concatenate([x.output,  y.output])
  
  # combined outputs
  z = Bidirectional(LSTM(128, dropout=0.25, return_sequences=False,activation='tanh'))(combined)
  z = Reshape((256,1),input_shape=(128,))
  z = Bidirectional(LSTM(128, dropout=0.25, return_sequences=False,activation='tanh'))(combined)
  
  #z = Dense(10, activation=&quot;relu&quot;)(z)
  z = Flatten()(z)
  z = Dense(4, activation=&quot;linear&quot;)(z)
  model = Model(inputs=[x.input, y.input], outputs=z)
  model.compile(loss=loss, optimizer = tf.keras.optimizers.Adam(),metrics=['mse'],run_eagerly=True)
  </code></pre>
  <p>I have tried to implement a custom loss function (based on different papers).</p>
  <p><strong>Math</strong></p>
  <p>The error will calculated as follows:</p>
  <pre><code>y_pred = [w x y z]
  y_true = [w1 x1 y1 z1]
  error = 2 * acos(w*w1 + x*x1 + y*y1 + z*z1)
  </code></pre>
  <p>Based on this formula I wrote the custom loss function:</p>
  <pre><code>def loss(y_true, y_pred):
      z = y_true * (y_pred )
      wtot = tf.reduce_sum(z,axis=1)
      error = 2*tf.math.acos(K.clip(tf.math.sqrt(wtot*wtot), -1.,1.))
      return error
  </code></pre>
  <p>But while the loss value is decreasing the MSE increased and I can see an offset in the output which will grow by the number of epochs. I understand that we do not optimize this Network for MSE but based on mathematics the MSE must be reduced or converge to some value near 1.</p>
  <p><em><strong>Orange is the Target/Reference</strong></em></p>
  <p><em><strong>Blue is the Network ouptut</strong></em></p>
  <p><strong>for 1 epoch</strong></p>
  <p><a href="https://i.stack.imgur.com/BeIEE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BeIEE.png" alt="enter image description here" /></a></p>
  <p><strong>for 10 epochs</strong></p>
  <p><a href="https://i.stack.imgur.com/6fgK2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6fgK2.png" alt="enter image description here" /></a></p>
  <p><strong>for 50 epochs</strong></p>
  <p><a href="https://i.stack.imgur.com/RADxm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RADxm.png" alt="enter image description here" /></a></p>