title: Wide and deep neural network - Why is the loss fluctuating?
body: <p>I'm trying to train a wide and deep neural network (<a href="https://www.tensorflow.org/tutorials/wide_and_deep" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/wide_and_deep</a>) with my own dataset.
  My dataset features are numeric values. It has many columns (280) and it is balanced by category. It's a binary classification problem with two labels: 0 and 1. All the code is the same as wide_deep.py (<a href="https://github.com/tensorflow/models/blob/master/official/wide_deep/wide_deep.py" rel="nofollow noreferrer">https://github.com/tensorflow/models/blob/master/official/wide_deep/wide_deep.py</a>), with a few changes because of the dataset:</p>
  
  <pre><code>_CSV_COLUMNS = [ names of my columns... ]
  
  
  _NUM_EXAMPLES = {
      'train': 40000,
      'validation': 10000,
  }
  
  ...
  
  def build_model_columns():
    """Builds a set of wide and deep feature columns."""
  
    columns = []
  
    for a in range(0, len(_CSV_COLUMNS)):
      _CSV_COLUMN_DEFAULTS.append([0.0])
      if (_CSV_COLUMNS[a] != "category"):
        columns.append(tf.feature_column.numeric_column(_CSV_COLUMNS[a]))
  
    wide_columns = columns
    deep_columns = columns
  
    return wide_columns, deep_columns
  
  ...
  
  
  def input_fn(data_file, num_epochs, shuffle, batch_size):
    """Generate an input function for the Estimator."""
    assert tf.gfile.Exists(data_file), (
        '%s not found.' % data_file)
  
    def parse_csv(value):
      print('Parsing', data_file)
      columns = tf.decode_csv(value, record_defaults=_CSV_COLUMN_DEFAULTS)
      features = dict(zip(_CSV_COLUMNS, columns))
      labels = features.pop('category')
      return features, tf.equal(labels, 0)
  
    # Extract lines from input files using the Dataset API.
    dataset = tf.data.TextLineDataset(data_file)
  
  
   ....
  
   results = model.evaluate(input_fn=lambda: input_fn(
      FLAGS.test_data, 1, False, FLAGS.batch_size))
  </code></pre>
  
  <p>The problem is that the loss decreases just a little at first and then it starts fluctuating.</p>
  
  <p>I've tried:</p>
  
  <ul>
  <li>Increasing the batch size</li>
  <li>Trying AdamOptimizer</li>
  <li>Normalizing the values in my dataset ([-1,1])</li>
  <li><p>Decreasing the learning rate, like this:</p>
  
  <pre><code>return tf.estimator.DNNLinearCombinedClassifier(
      model_dir=model_dir,
      linear_feature_columns=wide_columns,
      dnn_feature_columns=deep_columns,
      dnn_hidden_units=hidden_units,
      dnn_optimizer=tf.train.AdagradOptimizer(learning_rate=0.000001),
      config=run_config)
  </code></pre></li>
  </ul>
  
  <p>it doesn't matter how low the learning rate is, the loss won't go down.</p>
  
  <p>The output looks like this:</p>
  
  <pre><code>INFO:tensorflow:loss = 64.00549, step = 1
  INFO:tensorflow:loss = 41.65817, step = 101 (10.582 sec)
  INFO:tensorflow:loss = 35.131912, step = 201 (8.954 sec)
  INFO:tensorflow:loss = 41.854298, step = 301 (8.012 sec)
  INFO:tensorflow:loss = 24.876358, step = 401 (7.573 sec)
  INFO:tensorflow:loss = 44.788383, step = 501 (9.059 sec)
  INFO:tensorflow:loss = 33.58178, step = 601 (7.904 sec)
  INFO:tensorflow:loss = 35.278023, step = 701 (7.751 sec)
  INFO:tensorflow:loss = 28.880415, step = 801 (7.787 sec)
  INFO:tensorflow:loss = 33.041504, step = 901 (8.178 sec)
  ...
  </code></pre>
  
  <p>It will never be as high as 64 again, but it fluctuates and it won't go lower than 24. It seems to be learning something, because the accuracy is 0.81-0.82 (it won't get any better than that, either)
  What does it mean? Could my dataset have problems? Could it mean that the problem can't be solved?</p>