title: reduce_mean/reduce_sum axis effect on training?
body: <p>When calculating loss, does optimization somehow take into account the axis by which the reduced mean or sum is calculated?</p>
  
  <p>For example...</p>
  
  <pre><code>X = [[.1, .2, .3, .4, .5],[.4, .3, .2, .1, .1]] # target Data
  Y = [[.2, .1, .3, .2, .1],[.3, .1, .1, .3, .5]] # output data
  
  lossA = tf.reduce_sum(tf.square(X - Y),0)
  lossB = tf.reduce_sum(tf.square(X - Y),1)
  
  errA = tf.reduce_mean(lossA)
  errB = tf.reduce_mean(lossB)
  </code></pre>
  
  <p>A shows the sum of the samples per channel, while B shows the sum of the channels per sample, which generates a different mean err.</p>
  
  <pre><code>lossA: [ 0.02      ,  0.05000001,  0.01      ,  0.08000001,  0.32000002]
  lossB: [ 0.22000001,  0.26000002]
  
  errA: 0.096000008
  errB: 0.24000001
  </code></pre>
  
  <p>however if one were to use reduce_mean() for the loss calculation the final mean errors are equal.</p>
  
  <pre><code>lossA = tf.reduce_mean(tf.square(X - Y),0)
  lossB = tf.reduce_mean(tf.square(X - Y),1)
  
  
  lossA: [ 0.01      ,  0.025     ,  0.005     ,  0.04000001,  0.16000001]
  lossB: [ 0.044,  0.052]
  
  errA: 0.048000004
  errB: 0.048000004
  </code></pre>
  
  <p>regardless of using mean/sum I'm wondering if the optimizer will distribute the error differently to the weights depending on which axis I've indicated.  Or is this simply not taken into account during backprop?</p>
  
  <p>My intuition is that by reducing the sum along the 0 axis the optimizer would know better which path to pass a larger portion of the error along through backprop.  But I may have just thoroughly confused myself and this isn't actually relevant to the backprop calculation.</p>