title: Non Deterministic Results Using GPUs with Tensorflow and Tensorflow Serving . .. Why?
body: <p>We have an object detection model developed in Tensorflow (1.10 and 1.3) that uses a standard CNN and some extra layers.  We host the model in Tensorflow Serving 1.13.0 using a saved model format, on Nvidia Tesla V100 GPUs with Cuda 10 and CUDNN 7.4.x. (We use the Google containers images and/or dockerfiles for Tensorflow serving.) </p>
  
  <p>We run unit tests to ensure that prediction results are what we expect.  These all work great on CPU.  But when we run them on the above GPU/CUDA/CUDNN configuration, we get differences in the prediction probabilities ranging from .001 to .0005.</p>
  
  <p>Our goals are to understand:</p>
  
  <ul>
  <li>why this happens?</li>
  <li>is there anything we can do to prevent it?</li>
  <li>If there is something we can do to prevent it, does that entail some sort of trade off, such as performance?</li>
  </ul>
  
  <p>We have tried the following experiments:</p>
  
  <ol>
  <li><p>Multiple runs of the same model on tensorflow GPU using checkpoint with batchsize of 1 </p>
  
  <ul>
  <li>results identical </li>
  </ul></li>
  <li><p>Multiple runs of the same model on GPU using checkpoint with various batchsizes</p>
  
  <ul>
  <li>results off by .001</li>
  </ul></li>
  <li><p>Multiple runs of the same model on CPU  using checkpoint with various batchsizes</p>
  
  <ul>
  <li>results identical </li>
  </ul></li>
  <li><p>Multiple runs of the same model on tensorflow serviing GPU using checkpoint with batchsize of 1</p>
  
  <ul>
  <li>results identical </li>
  </ul></li>
  <li><p>Comparing runs with checkpoint to runs with saved model on GPU</p>
  
  <ul>
  <li>results off by .005</li>
  </ul></li>
  <li><p>Compare runs with checkpoint to runs with savedmodel on CPU</p>
  
  <ul>
  <li>results identical</li>
  </ul></li>
  <li><p>Experimented with changing the batch_size and setting <code>TF_CUDNN_USE_AUTOTUNE=0</code> on GPU</p>
  
  <ul>
  <li>reduces max difference from .001 to .0005</li>
  </ul></li>
  <li><p>Experimented with adding <code>intra_op_parallelism_threads=1</code>, <code>inter_op_parallelism_threads=1</code> didn’t make any difference when used with <code>TF_CUDNN_USE_AUTOTUNE=0</code></p>
  
  <ul>
  <li>results no different than the above</li>
  </ul></li>
  </ol>
  
  <p>IN SUMMARY: We have a few cases where the results of running inference on GPU  are different:</p>
  
  <ol>
  <li>Using a checkpoint versus a saved model.</li>
  <li>Batchsize = 1 versus various batch sizes</li>
  <li>Setting <code>TF_CUDNN_USE_AUTOTUNE=0</code> reduces the difference when using various batch sizes</li>
  </ol>
  
  <p>This happens with TF 1.10 AND 1.13.1</p>
  
  <p>Again,  our goals are to understand:</p>
  
  <ol>
  <li>Why this happens?  </li>
  <li>Is there anything we can do to prevent it?  </li>
  <li>If there is something we can do to prevent it, does that entail some sort of trade off, such as performance?  </li>
  </ol>