title: Tensorflow-serving docker container adds the GPU device but GPU has 0% utilization
body: <h3>Hi, I'm having issues with dockerized TF Serving seeing but not using my GPU.</h3>
  <p>It adds the GPU as device 0, allocates memory on it, but then loads the ML model into CPU device memory and runs inference using only the CPUs. GPU-util on nvidia-smi never leaves 0%.</p>
  <p>Could anyone help me figure out why this is happening, and what should be changed?</p>
  <p><strong>The setup:</strong></p>
  <p><strong>OS:</strong> Amazon/Deep Learning AMI (Ubuntu 18.04) on EC2 g4dn.xlarge</p>
  <p><strong>GPU:</strong> Tesla T4</p>
  <p><strong>Model:</strong> pretrained <a href="https://huggingface.co/gpt2" rel="nofollow noreferrer">gpt2-xl tensorflow from huggingface</a>, which I froze into a SavedModel and uploaded to S3.</p>
  <p><strong>Docker:</strong> came stock with Deep Learning AMI. I've already checked and confirmed that nvidia-smi runs containerized, so it's not a nvidia+docker issue.</p>
  <p><strong>TF Serving:</strong> I use the below Dockerfile to pull the latest-gpu image and download the model directly into it at buildtime:</p>
  <pre><code>FROM tensorflow/serving:latest-gpu
  
  RUN apt-get update
  
  ENV TZ=America
  RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime &amp;&amp; echo $TZ &gt; /etc/timezone
  RUN apt-get update
  RUN apt-get install -y awscli
  
  ENV AWS_ACCESS_KEY_ID=...
  ENV AWS_SECRET_ACCESS_KEY=...
  
  ARG model_name
  ENV MODEL_NAME=$model_name
  
  # Use AWS CLI to download the SavedModel into the docker container from S3 bucket
  RUN aws s3 cp s3://v3-models/models/pretrained_tf_serving/${MODEL_NAME} /models/${MODEL_NAME} --recursive
  
  EXPOSE 8500
  </code></pre>
  <p><strong>I build and run the above Dockerfile with these commands:</strong></p>
  <pre><code>#!/bin/bash
  
  # first build the image with the model_name arg, and tag it as xl-serving
  docker build -t xl-serving --build-arg model_name=gpt2-xl ../../model_server
  
  # then run it with gpus, exposing gRPC port
  docker run -it --rm --gpus all --runtime nvidia -p 8500:8500 xl-serving 
  </code></pre>
  <p><strong>Running the serving container prints this output.</strong> Notice that the GPU is added.</p>
  <pre><code>2020-11-06 05:25:34.671071: I tensorflow_serving/model_servers/server.cc:87] Building single TensorFlow model file config:  model_name: gpt2-xl model_base_path: /models/gpt2-xl
  2020-11-06 05:25:34.671274: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
  2020-11-06 05:25:34.671295: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: gpt2-xl
  2020-11-06 05:25:34.771644: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: gpt2-xl version: 1}
  2020-11-06 05:25:34.771673: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: gpt2-xl version: 1}
  2020-11-06 05:25:34.771687: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: gpt2-xl version: 1}
  2020-11-06 05:25:34.771724: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/gpt2-xl/1
  2020-11-06 05:25:35.222512: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
  2020-11-06 05:25:35.222545: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:234] Reading SavedModel debug info (if present) from: /models/gpt2-xl/1
  2020-11-06 05:25:35.222672: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2020-11-06 05:25:35.223994: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
  2020-11-06 05:25:35.262238: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:35.263132: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
  pciBusID: 0000:00:1e.0 name: Tesla T4 computeCapability: 7.5
  coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
  2020-11-06 05:25:35.263149: I external/org_tensorflow/tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
  2020-11-06 05:25:35.263236: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:35.264122: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:35.264948: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
  2020-11-06 05:25:36.185140: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
  2020-11-06 05:25:36.185165: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
  2020-11-06 05:25:36.185171: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
  2020-11-06 05:25:36.185334: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:36.186222: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:36.187046: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2020-11-06 05:25:36.187852: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13896 MB memory) -&gt; physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5)
  2020-11-06 05:25:37.279837: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:199] Restoring SavedModel bundle.
  2020-11-06 05:25:56.154008: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:183] Running initialization op on SavedModel bundle at path: /models/gpt2-xl/1
  2020-11-06 05:25:57.551535: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:303] SavedModel load for tags { serve }; Status: success: OK. Took 22777844 microseconds.
  2020-11-06 05:25:57.832736: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:59] No warmup data file found at /models/gpt2-xl/1/assets.extra/tf_serving_warmup_requests
  2020-11-06 05:25:57.835030: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: gpt2-xl version: 1}
  2020-11-06 05:25:57.838329: I tensorflow_serving/model_servers/server.cc:367] Running gRPC ModelServer at 0.0.0.0:8500 ...
  [warn] getaddrinfo: address family for nodename not supported
  2020-11-06 05:25:57.840415: I tensorflow_serving/model_servers/server.cc:387] Exporting HTTP/REST API at:localhost:8501 ...
  [evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
  </code></pre>
  <p>I then hit this server with a single, non-batched gRPC call. It will successfully run and return a correct GPT2 output. However, it takes as long as the same setup takes on a CPU. htop shows that 8gb of ram (gpt2-xl model size) is loaded into the CPU machine. It then shows the TF Serving process running, and maxing out one or two CPU cores. It appears to only run on CPU.</p>
  <p>This is what <strong>nvidia-smi</strong> looks like while the call is running. Notice the allocated memory, and 0% GPU-Util:</p>
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
  | N/A   36C    P0    26W /  70W |  14240MiB / 15109MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
                                                                                 
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    0   N/A  N/A     13357      C   tensorflow_model_server         14221MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p>I've scoured the web and can't find any advice for this. Closest I found was this github issue: <a href="https://github.com/tensorflow/serving/issues/1440" rel="nofollow noreferrer">GPU utilization with TF serving #1440</a>, for which the fixes did not work for me. They were dealing with low GPU-util, I'm dealing with 0%.</p>
  <p><strong>Any advice on what the issue is?</strong></p>
  <p>Thank you very much. I've been banging my head against the wall for days on this, so I very much appreciate your help :)</p>
  <h2>Update #1 :</h2>
  <p>I've written a python script (below) to use tensorflow==2.3.0 to load the model and run it. It's running in a conda env with CUDA=11.0. It successfully runs inference on the GPU, and it's a good 15x faster than what I'm getting on TF-serving.</p>
  <pre><code>import tensorflow as tf
  import numpy as np
  
  model = tf.saved_model.load('/home/ubuntu/models/gpt2-xl/1/')
  servable = model.signatures[&quot;forward&quot;]
  
  # Create input tensor
  tensor_in = tf.constant([[198, 15667,  6530, 25, 29437, 1706, 1610, 977, 948, 33611]])
  
  # Run a loop of 10 inferences on the model, to predict the next 10 tokens.
  for i in range(10):
      pred = servable(tensor_in)
      logits = pred['output_0']
      logits = logits[:, -1, :] / 0.8
      next_id = tf.random.categorical(tf.nn.log_softmax(logits, axis=-1), num_samples=1)
      next_id = tf.dtypes.cast(next_id, tf.int32).numpy()
      tensor_in = np.concatenate((tensor_in, next_id), axis=1)
  </code></pre>
  <p>Up next: will be trying running tf-serving outside of container. Update to come...</p>