title: Accumulate gradients with distributed strategy in Tensorflow 2
body: <p>I have implemented a distributed strategy to train my model on multiple GPUs.</p>
  <pre><code>strategy = tf.distribute.MirroredStrategy(devices=devices[:FLAGS.n_gpus])
  strategy.run(fn=self.train_step, args=(model, data))
  </code></pre>
  <p>My model now got more complex and bigger and I had to reduce the batch size to fit it onto the GPUs.
  The gradient is quite noisy now and I want to increase the batch size again by accumulating gradients.</p>
  <p>Now my question is: is this even possible when using a mirrored strategy? I know that loss and gradients are combined across the replicas anyway, so is there a way to sum them across the replicas AND e.g. a loop running over the batches? I tried the straight-forward thing and returned the per replica calculated gradients to add and apply them outside the <code>strategy.run()</code> like that:</p>
  <pre><code>for b in batches:
      per_replica_gradients = strategy.run(fn=self.train_step, args=(model, data))
      total_gradient += per_replica_gradients
  optimizer.apply_gradients(zip(total_gradient, model.trainable_variables)
  </code></pre>
  <p>but Tensorflow tells me that this is not possible and the gradients have to be applied withing the <code>strategy.run()</code>. This also makes sense to me but I wonder whether there is a possibility to accumulate gradients AND use a mirrored strategy?</p>