title: Output average loss on each epoch : weird results
body: <p>I implemented just a simple model for MNIST data set using Tensorflow.</p>
  
  <p>Here is the model:</p>
  
  <pre><code>X = tf.placeholder(tf.float32, [None, 784])
  Y_ = tf.placeholder(tf.float32, [None, 10])
  
  W = tf.Variable(tf.random_normal(shape=[784, 10], stddev=0.01), name="Weigths")
  b = tf.Variable(tf.zeros([1,10]), name="Bias")
  
  Y = tf.nn.softmax(tf.add(tf.matmul(X,W), b))
  </code></pre>
  
  <p>Here is how the cost function looks like:</p>
  
  <pre><code>entropy = tf.nn.softmax_cross_entropy_with_logits(labels=Y_, logits=Y)
  loss = tf.reduce_mean(entropy)
  </code></pre>
  
  <p>The backprop:</p>
  
  <pre><code>optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)
  </code></pre>
  
  <p>And here is my train loop:</p>
  
  <pre><code>for epoch in range(n_epochs):
      avg_loss = 0;
      n_batches = int(MNIST.train.num_examples/batch_size) 
      for i in range(n_batches):
          X_batch, Y_batch = MNIST.train.next_batch(batch_size)
          _, l, summary = sess.run([optimizer, loss, merged_summary], feed_dict={X: X_batch, Y_: Y_batch})
          writer.add_summary(summary, pos)
          avg_loss = l / n_batches
      print('Epoch :', epoch, 'AvgLoss =', avg_loss)
  print ("Accuracy:", acc.eval(feed_dict={X: MNIST.test.images, Y_: MNIST.test.labels}))
  </code></pre>
  
  <p>But I don't understand the result of my average cost on each epoch:</p>
  
  <pre><code>Epoch : 0 AvgLoss = 0.0028913601962
  Epoch : 1 AvgLoss = 0.00283967841755
  Epoch : 2 AvgLoss = 0.0028030406345
  Epoch : 3 AvgLoss = 0.002759949294
  Epoch : 4 AvgLoss = 0.00283429449255
  Epoch : 5 AvgLoss = 0.00276749762622
  Epoch : 6 AvgLoss = 0.00276815457778
  Epoch : 7 AvgLoss = 0.00279549772089
  Epoch : 8 AvgLoss = 0.00277937347239
  Epoch : 9 AvgLoss = 0.00274000016126
  Epoch : 10 AvgLoss = 0.00275734966451
  Epoch : 11 AvgLoss = 0.00278236475858
  Epoch : 12 AvgLoss = 0.00275594126094
  Epoch : 13 AvgLoss = 0.0027651628581
  Epoch : 14 AvgLoss = 0.00275661511855
  Epoch : 15 AvgLoss = 0.00275890090249
  Epoch : 16 AvgLoss = 0.00273716428063
  Epoch : 17 AvgLoss = 0.00273372628472
  Epoch : 18 AvgLoss = 0.0027502430569
  Epoch : 19 AvgLoss = 0.00279064221816
  Epoch : 20 AvgLoss = 0.00273178425702
  Epoch : 21 AvgLoss = 0.00277335535396
  Epoch : 22 AvgLoss = 0.00276518474926
  Epoch : 23 AvgLoss = 0.00276605887847
  Epoch : 24 AvgLoss = 0.00275481895967
  </code></pre>
  
  <p>It's not reducing each loop... But it gives me a OK accuracy:</p>
  
  <pre><code>Accuracy: 0.9295
  </code></pre>
  
  <p>Any ideas on why is it so ?</p>