title: Why is the value of my loss so huge? Running an MLP on Tensorflow
body: <p>I am running a plain Sequential API MLP and encounter a huge loss in my output. I am not sure what is wrong and how to fix. I am using Batch Normalization layers as I read that would help and it actually does but my output is still ridiculous. Any other advice on the architecture is more than welcome.</p>
  <p>RMSE loss is about 56000</p>
  <pre><code>from tensorflow.keras.models import Sequential
  
  model = Sequential()
  model.add(Dense(3, input_dim=X_train_enc.shape[1], activation='relu', kernel_initializer='he_normal'))
  model.add(BatchNormalization())
  model.add(Dense(10, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(20, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(30, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(20, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(10, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(3, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dense(1, activation = 'linear'))
  
  # # summarize layers
  print(model.summary())
  
  from tensorflow.keras import backend
  from tensorflow.keras.losses import mean_squared_error
  
  #Creating/Defining our own metrics
  def mean_absolute_percentage_error(y_true, y_pred):
      diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true),
                                          K.epsilon(),
                                          None))
      return 100. * K.mean(diff, axis=-1)
  tf.keras.backend.set_epsilon(1) #https://stackoverflow.com/questions/49729522/why-is-the-mean-average-percentage-errormape-extremely-high
  
  
  def rmse(y_true, y_pred):
      return backend.sqrt(mean_squared_error(y_true, y_pred))
  
  #def rmse(y_true, y_pred):
  #    return backend.sqrt(backend.mean(backend.square(y_pred - y_true), axis=-1))
  
  Adamax = tf.keras.optimizers.Adamax(lr=0.02, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
  
  #Compiling model
  model.compile(optimizer='adam',loss=rmse, metrics=['accuracy',rmse, 'mae', 'mape'])#Metric functions are similar to loss functions, except that the results from evaluating a metric are not used when training the model
  
  # # Creating a checkpoint
  filepath=&quot;weights-improvement-{epoch:02d}-{val_rmse:.9f}.hdf5&quot;
  checkpoint = ModelCheckpoint(filepath, monitor='val_rmse', verbose=1, save_best_only=True, mode='min')
  # # monitor = &quot;val_loss&quot;, save_best_only=False
  callbacks_list = [checkpoint]
  
  history = model.fit(X_train_enc, y_train, epochs=200, batch_size=16, validation_split=0.2681867, callbacks=callbacks_list, verbose=1)
  
  
  </code></pre>