title: tensorflow RuntimeError: Coordinator stopped with threads still running: Thread-#
body: <p>When doing validation I am always running into "RuntimeError: Coordinator stopped with threads still running"
  Like some error message below:</p>
  
  <pre><code>Traceback (most recent call last):
    File "cli.py", line 78, in &lt;module&gt;
      main()
    File "cli.py", line 67, in main
      run.train(config, val_config=val_config)
    File "F:\NLP-cse517_2017_UW\hw5\run.py", line 206, in train
      train_run.join_queue(coord)
    File "F:\NLP-cse517_2017_UW\hw5\run.py", line 114, in join_queue
      iq.join(coord)
    File "F:\NLP-cse517_2017_UW\hw5\input_queue.py", line 60, in join
      coord.join(self.threads)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\training\coordinator.py", line 397, in join
      " ".join(stragglers))
  RuntimeError: Coordinator stopped with threads still running: Thread-4
  Exception in thread Thread-4:
  Traceback (most recent call last):
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 1136, in _do_call
      return fn(*args)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 1118, in _run_fn
      status, run_metadata)
    File "C:\Anaconda2\envs\python35\lib\contextlib.py", line 66, in __exit__
      next(self.gen)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
      pywrap_tensorflow.TF_GetCode(status))
  tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
           [[Node: random_shuffle_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](random_shuffle_queue, _arg_x_0_3, _arg_q_0_1, _arg_y1_0_5, _arg_y2_0_6, _arg_x_len_0_4, _arg_q_len_0_2, _arg_idxs_0_0)]]
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "C:\Anaconda2\envs\python35\lib\threading.py", line 914, in _bootstrap_inner
      self.run()
    File "C:\Anaconda2\envs\python35\lib\threading.py", line 862, in run
      self._target(*self._args, **self._kwargs)
    File "F:\NLP-cse517_2017_UW\hw5\input_queue.py", line 49, in _thread_main
      sess.run(self._enqueue_op, feed_dict=feed_dict)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 786, in run
      run_metadata_ptr)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 994, in _run
      feed_dict_string, options, run_metadata)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 1129, in _do_run
      target_list, options, run_metadata)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\client\session.py", line 1149, in _do_call
      raise type(e)(node_def, op, message)
  tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
           [[Node: random_shuffle_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](random_shuffle_queue, _arg_x_0_3, _arg_q_0_1, _arg_y1_0_5, _arg_y2_0_6, _arg_x_len_0_4, _arg_q_len_0_2, _arg_idxs_0_0)]]
  
  Caused by op 'random_shuffle_queue_enqueue', defined at:
    File "cli.py", line 78, in &lt;module&gt;
      main()
    File "cli.py", line 67, in main
      run.train(config, val_config=val_config)
    File "F:\NLP-cse517_2017_UW\hw5\run.py", line 183, in train
      train_run = Run(config, val_config=val_config)
    File "F:\NLP-cse517_2017_UW\hw5\run.py", line 31, in __init__
      self.iq = InputQueue(config, self.data)
    File "F:\NLP-cse517_2017_UW\hw5\input_queue.py", line 32, in __init__
      for placeholders in self._placeholders_list
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 322, in enqueue
      self._queue_ref, vals, name=scope)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 1589, in _queue_enqueue_v2
      name=name)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
      op_def=op_def)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
      original_op=self._default_original_op, op_def=op_def)
    File "C:\Anaconda2\envs\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
      self._traceback = _extract_stack()
  
  CancelledError (see above for traceback): Enqueue operation was cancelled
           [[Node: random_shuffle_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](random_shuffle_queue, _arg_x_0_3, _arg_q_0_1, _arg_y1_0_5, _arg_y2_0_6, _arg_x_len_0_4, _arg_q_len_0_2, _arg_idxs_0_0)]]
  </code></pre>
  
  <p>Sometimes this can be fixed with specifying 'TF_CPP_MIN_LOG_LEVEL', so there are no warnings ahead. </p>
  
  <pre><code>import os
  os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
  </code></pre>
  
  <p>Anyone knows how to fix this?</p>