title: File too large Failed to flush events
body: <p>I am trying to train a network. And the training breaks with an error saying that the file is too large and failed to flush events (tf.summary). When I checked file was actually 4GB+</p>
  <pre><code>2021-08-08 15:16:44.725004: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at summary_kernels.cc:142 : Out of range: gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2; File too large
      Failed to flush 11 events to gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2
      Could not flush events file.
  </code></pre>
  <p>I am working with the delf implementation (Train script found here: <a href="https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/train.py" rel="nofollow noreferrer">https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/train.py</a> ).
  I encountered this issue earlier with an object detection model as well. As I remember I simply avoided the issue by omitting the event writing. But I would like to find a solution/reason for this issue.</p>
  <p>I found questions about missing event file, but wasn't able to find a good answer for my problem. I would appreciate your help on this matter.</p>
  <p>Update on what I have tried so far</p>
  <ul>
  <li>Changing max_queue parameter - Noticed that tf.summary.create_file_writer has a max_queue parameter with a default value is 10. So I assumed it gives the error failed to flush 11 events because it exceeds this default values. So I tried with different values 20, 200 and ended up getting errors saying Failed to flush 21 events, 201 events etc.</li>
  <li>Changing flush_millis - Changed this to a lower value to flush events more frequently</li>
  <li>summary_writer.flush() - The code did not have flush() probably because it is supposed to flush after flush_millis times), yet I tried to add flush() to see if it helps</li>
  </ul>
  <p>Unfortunately none of this worked.</p>
  <p>Full error stack from the initial error</p>
  <pre><code>    2021-08-08 15:16:44.725004: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at summary_kernels.cc:142 : Out of range: gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2; File too large
      Failed to flush 11 events to gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2
      Could not flush events file.
  Traceback (most recent call last):
    File &quot;train.py&quot;, line 486, in &lt;module&gt;
      app.run(main)
    File &quot;/home/user/.local/lib/python3.8/site-packages/absl/app.py&quot;, line 312, in run
      _run_main(main, args)
    File &quot;/home/user/.local/lib/python3.8/site-packages/absl/app.py&quot;, line 258, in _run_main
      sys.exit(main(argv))
    File &quot;train.py&quot;, line 410, in main
      desc_dist_loss, attn_dist_loss = distributed_train_step(input_batch)
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py&quot;, line 889, in __call__
      result = self._call(*args, **kwds)
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py&quot;, line 917, in _call
      return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py&quot;, line 3023, in __call__
      return graph_function._call_flat(
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py&quot;, line 1960, in _call_flat
      return self._build_call_outputs(self._inference_function.call(
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py&quot;, line 591, in call
      outputs = execute.execute(
    File &quot;/home/user/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py&quot;, line 59, in quick_execute
      tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
    (0) Out of range:  gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2; File too large
      Failed to flush 11 events to gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2
      Could not flush events file.
       [[node attention/percent_25/write_summary (defined at /home/user/.local/lib/python3.8/site-packages/tensorboard/plugins/scalar/summary_v2.py:89) ]]
    (1) Out of range:  gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2; File too large
      Failed to flush 11 events to gldv2_training/train_logs/events.out.tfevents.1628396331.XX.9717.627.v2
      Could not flush events file.
       [[node attention/percent_25/write_summary (defined at /home/user/.local/lib/python3.8/site-packages/tensorboard/plugins/scalar/summary_v2.py:89) ]]
       [[GroupCrossDeviceControlEdges_0/Identity_2/_107]]
  0 successful operations.
  0 derived errors ignored. [Op:__inference_distributed_train_step_12517]
  
  Errors may have originated from an input operation.
  Input Source operations connected to node attention/percent_25/write_summary:
   batch_images/write_summary/writer (defined at /home/user/.local/lib/python3.8/site-packages/tensorboard/plugins/image/summary_v2.py:140)
  
  Input Source operations connected to node attention/percent_25/write_summary:
   batch_images/write_summary/writer (defined at /home/user/.local/lib/python3.8/site-packages/tensorboard/plugins/image/summary_v2.py:140)
  
  Function call stack:
  distributed_train_step -&gt; distributed_train_step
  </code></pre>