<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 1.12</li>
<li>Python version: 3.6.5</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: N/A</li>
<li>GPU model and memory: N/A</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
Running below code that I found in many pages on the net, I faced some problems:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import json
import os
import tensorflow as tf
from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets
data_dir = '.\\MNIST_data'
log_dir = '.\log_dist'
batch_size = 512
tf.logging.set_verbosity(tf.logging.INFO)

def keras_model(lr, decay):
    &quot;&quot;&quot;Return a CNN Keras model&quot;&quot;&quot;
    input_tensor = tf.keras.layers.Input(shape=(784,), name='input')

    temp = tf.keras.layers.Reshape([28, 28, 1], name='input_image')(input_tensor)
    for i, n_units in enumerate([32, 64]):
        temp = tf.keras.layers.Conv2D(n_units, kernel_size=3, strides=(2, 2),
                                      activation='relu', name='cnn'+str(i))(temp)
        temp = tf.keras.layers.Dropout(0.5, name='dropout'+str(i))(temp)
    temp = tf.keras.layers.GlobalAvgPool2D(name='average')(temp)
    output = tf.keras.layers.Dense(10, activation='softmax', name='output')(temp)

    model = tf.keras.models.Model(inputs=input_tensor, outputs=output)
    optimizer = tf.keras.optimizers.Adam(lr=lr, decay=decay)
    model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    print(model.summary())
    return model


def main():
    &quot;&quot;&quot;Main function&quot;&quot;&quot;
    data = read_data_sets(data_dir,
                          one_hot=False,
                          fake_data=False)
    model = keras_model(lr=0.001, decay=0.001)
    config = tf.estimator.RunConfig(
                model_dir=log_dir,
                save_summary_steps=1,
                save_checkpoints_steps=100)
    estimator = tf.keras.estimator.model_to_estimator(model, model_dir=log_dir, config=config)

    train_input_fn = tf.estimator.inputs.numpy_input_fn(
                         x={'input': data.train.images},
                         y=data.train.labels,
                         num_epochs=None,   # run forever
                         batch_size=batch_size,
                         shuffle=True)
    eval_input_fn = tf.estimator.inputs.numpy_input_fn(
                         x={'input': data.test.images},
                         y=data.test.labels,
                         num_epochs=1,
                         shuffle=False)

    train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn,
                                        max_steps=2000)
    eval_spec = tf.estimator.EvalSpec(input_fn=eval_input_fn,
                                      #throttle_secs=1,
                                      steps=None    # until the end of evaluation data
                                      )

    evaluate_result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
    print(&quot;Evaluation results:&quot;)
    for key in evaluate_result[0].keys():
        print(&quot;   {}: {}&quot;.format(key, evaluate_result[0][key]))"><pre class="notranslate"><code class="notranslate">import json
import os
import tensorflow as tf
from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets
data_dir = '.\\MNIST_data'
log_dir = '.\log_dist'
batch_size = 512
tf.logging.set_verbosity(tf.logging.INFO)

def keras_model(lr, decay):
    """Return a CNN Keras model"""
    input_tensor = tf.keras.layers.Input(shape=(784,), name='input')

    temp = tf.keras.layers.Reshape([28, 28, 1], name='input_image')(input_tensor)
    for i, n_units in enumerate([32, 64]):
        temp = tf.keras.layers.Conv2D(n_units, kernel_size=3, strides=(2, 2),
                                      activation='relu', name='cnn'+str(i))(temp)
        temp = tf.keras.layers.Dropout(0.5, name='dropout'+str(i))(temp)
    temp = tf.keras.layers.GlobalAvgPool2D(name='average')(temp)
    output = tf.keras.layers.Dense(10, activation='softmax', name='output')(temp)

    model = tf.keras.models.Model(inputs=input_tensor, outputs=output)
    optimizer = tf.keras.optimizers.Adam(lr=lr, decay=decay)
    model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    print(model.summary())
    return model


def main():
    """Main function"""
    data = read_data_sets(data_dir,
                          one_hot=False,
                          fake_data=False)
    model = keras_model(lr=0.001, decay=0.001)
    config = tf.estimator.RunConfig(
                model_dir=log_dir,
                save_summary_steps=1,
                save_checkpoints_steps=100)
    estimator = tf.keras.estimator.model_to_estimator(model, model_dir=log_dir, config=config)

    train_input_fn = tf.estimator.inputs.numpy_input_fn(
                         x={'input': data.train.images},
                         y=data.train.labels,
                         num_epochs=None,   # run forever
                         batch_size=batch_size,
                         shuffle=True)
    eval_input_fn = tf.estimator.inputs.numpy_input_fn(
                         x={'input': data.test.images},
                         y=data.test.labels,
                         num_epochs=1,
                         shuffle=False)

    train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn,
                                        max_steps=2000)
    eval_spec = tf.estimator.EvalSpec(input_fn=eval_input_fn,
                                      #throttle_secs=1,
                                      steps=None    # until the end of evaluation data
                                      )

    evaluate_result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
    print("Evaluation results:")
    for key in evaluate_result[0].keys():
        print("   {}: {}".format(key, evaluate_result[0][key]))
</code></pre></div>
<p dir="auto">And then the rest of the code just included the TF_CONFIG definition for chief, worker and ps. I faced below issues:</p>
<ul dir="auto">
<li>I was able to run this code on Tensorflow 1.12 but not on Tensorflow 1.13, where I got the error <code class="notranslate">ValueError: Can not squeeze dim[1], expected a dimension of 1, got 10 for 'metrics/acc/remove_squeezable_dimensions/Squeeze' (op: 'Squeeze') with input shapes: [512,10]</code>. What is the reason?</li>
<li>I could get evaluation results printed at the end of training when I was running program in non-distributed mode, but I get below error when it tries to print the final evaluation results in distributed mode:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;mnist_estimator.py&quot;, line 81, in &lt;module&gt;
    main()
  File &quot;mnist_estimator.py&quot;, line 62, in main
    for key in evaluate_result[0].keys():
TypeError: 'NoneType' object is not subscriptable"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "mnist_estimator.py", line 81, in &lt;module&gt;
    main()
  File "mnist_estimator.py", line 62, in main
    for key in evaluate_result[0].keys():
TypeError: 'NoneType' object is not subscriptable
</code></pre></div>
<ul dir="auto">
<li>The final loss for distributed learning was higher than non-distributed learning (for the same number of training steps). What can be the reason? Is it the nature of distribution?</li>
<li>When running in distributed mode, the chief or worker are not waiting for the other party to start and immediately starts training (when the other party joins they do the task together, though). I thought they should wait for each other to be ready (as it was in my previous experiences with Tensorflow distributed training), isn't it?</li>
<li>What I read in Tensorflow-related pages about data-parallelism is that there are the same copies of code for different servers except in assignment in TF_CONFIG. The chief synchronizes the parameters update and parameter servers keeps the parameters, but I don't clearly understand who split the data between different workers. Is there just one copy at the chief server and it will split the data and send batches to the workers, or the workers each have a local copy of data and do the splitting and skip some data themselves?</li>
</ul>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): 1.12 &amp; 1.13</li>
<li>Python version: 3.6.5</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: N/A</li>
<li>GPU model and memory: N/A</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I am working on distributed learning in tensorflow through estimators API using below simple code template:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="runConfig = tf.estimator.RunConfig(session_config=config, 
                                model_dir=log_dir,
                                save_summary_steps=1,
                                save_checkpoints_steps=train_steps)
estimator = tf.keras.estimator.model_to_estimator(model, model_dir=log_dir, config=runConfig) 
train_spec = tf.estimator.TrainSpec(input_fn=lambda: read_dataset(...), max_steps=...) 
eval_spec = tf.estimator.EvalSpec(input_fn=lambda: read_dataset(...), start_delay_secs=1,
                                throttle_secs=1, steps=None) 
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"><pre class="notranslate"><code class="notranslate">runConfig = tf.estimator.RunConfig(session_config=config, 
                                model_dir=log_dir,
                                save_summary_steps=1,
                                save_checkpoints_steps=train_steps)
estimator = tf.keras.estimator.model_to_estimator(model, model_dir=log_dir, config=runConfig) 
train_spec = tf.estimator.TrainSpec(input_fn=lambda: read_dataset(...), max_steps=...) 
eval_spec = tf.estimator.EvalSpec(input_fn=lambda: read_dataset(...), start_delay_secs=1,
                                throttle_secs=1, steps=None) 
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
</code></pre></div>
<p dir="auto">My model is defined as a simple 2-layer LSTM model in keras, and read_dataset() functions return datasets that I use for training and validation purposes.  The training/validation data files  and model directory are set in a shared place available to all workers. The whole code is exactly the same for servers (ps, chief, and worker) except the task setting in TF_CONFIG.<br>
When I train model in single-worker configuration, the loss graph I see in tensorboard is gradually downward and reasonable.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/17579773/65541564-975a6a80-dedb-11e9-9a56-f77d2b5906a8.jpg"><img src="https://user-images.githubusercontent.com/17579773/65541564-975a6a80-dedb-11e9-9a56-f77d2b5906a8.jpg" alt="single" style="max-width: 100%;"></a></p>
<p dir="auto">When using two machines of one chief and one worker, the total run time is less (as expected) but the loss graph is very noisy and higher than single-server case.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/17579773/65541692-d092da80-dedb-11e9-96bc-a3a3b0f0d92e.jpg"><img src="https://user-images.githubusercontent.com/17579773/65541692-d092da80-dedb-11e9-96bc-a3a3b0f0d92e.jpg" alt="double" style="max-width: 100%;"></a></p>
<p dir="auto">I expected to see the same performance in both cases, but it seems that training in the second server ruins the situation. Is there any special provision/setting/additional code that I should include in my work?</p>