title: Tensorflow CNN training stops for no reason
body: <p>I have this Python code for training a CNN using Tensorflow.
  It was working perfectly until I decided to use <em>MirroredStrategy</em> to improve the performance. Then it was necessary to make some changes:</p>
  
  <ol>
  <li>I had to stop using <code>tf.estimator.inputs.numpy_input_fn()</code> since MirroredStrategy it requires a input function that returns a Dataset.</li>
  <li>I basically copied the <code>train_input_fn</code> from <a href="https://www.tensorflow.org/guide/datasets_for_estimators" rel="nofollow noreferrer">Datasets for Estimators</a> to use as my input function</li>
  </ol>
  
  <p>Now it seems to work without any problem but it is not training anything at all and after a few seconds it stops running without any error.</p>
  
  <p>Few more details:</p>
  
  <ol>
  <li>I'm monitoring my CPU and GPU usage during the training, and when I run the code the GPU frequency raises until the program stops.</li>
  <li>I already tried to change the batch size, but no changes.</li>
  <li>My GPU is a GTX 1070 the driver was updated as well and I'm using <code>tensorflow-gpu</code> version 1.12.0 in my environment</li>
  </ol>
  
  <h2>Kernel output:</h2>
  
  <pre><code>In [1]: runfile('D:/MS/CNN.py', wdir='D:/MS')
  INFO:tensorflow:Initializing RunConfig with distribution strategies.
  INFO:tensorflow:Not using Distribute Coordinator.
  INFO:tensorflow:Using config: {'_model_dir': 'models3', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
  graph_options {
    rewrite_options {
      meta_optimizer_iterations: ONE
    }
  }
  , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': &lt;tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x0000026729AC1320&gt;, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': &lt;tensorflow.python.training.server_lib.ClusterSpec object at 0x0000026729AC14A8&gt;, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
  INFO:tensorflow:Device is available but not used by distribute strategy: /device:CPU:0
  INFO:tensorflow:Configured nccl all-reduce.
  INFO:tensorflow:Calling model_fn.
  INFO:tensorflow:batch_all_reduce invoked for batches size = 16 with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10
  
  In [1]:
  </code></pre>
  
  <h2>Training code:</h2>
  
  <pre><code>def train_input_fn(features, labels, batch_size):
      # Convert the inputs to a Dataset.
      dataset = tf.data.Dataset.from_tensor_slices((features, labels))
      # Shuffle, repeat, and batch the examples.
      dataset = dataset.shuffle(1000).repeat(count=100).batch(batch_size)
      # Return the dataset.
      return dataset
  
  gpu_distribution = tf.contrib.distribute.MirroredStrategy()
  run_config = tf.estimator.RunConfig(train_distribute=gpu_distribution)
  classifier = tf.estimator.Estimator(model_fn= cria_rede, config=run_config, model_dir='models3')
  classifier.train(input_fn=lambda: train_input_fn(x_data, y_data, 2), steps= 3000)
  #x_data and y_data are bidimensional numpy.ndarrays
  </code></pre>