title: Is it possible to run multiple tensorflow serving containers on single GPU node kubernetes
body: <p>I am running a Tensorflow model on an AKS cluster with GPU nodes. The model currently runs in a single TF Serving container (<a href="https://hub.docker.com/r/tensorflow/serving" rel="nofollow noreferrer">https://hub.docker.com/r/tensorflow/serving</a>) in a single pod on a single GPU node.</p>
  <p>By default the TF serving container will claim all available RAM in the pod, but I can downscale the memory request by the container in my <code>deployment.yaml</code> file and still get the same results in acceptable processing time. I was wondering if there is any possibility to run two TF models in parallel on the same GPU node. Memory-wise it should work, but when I try to adapt the replicaset of my deployment to two, it tries to deploy two pods but the second one is hanging on the status pending.</p>
  <pre><code>$ kubectl get po -n myproject -w
  NAME                                 READY   STATUS    RESTARTS   AGE
  myproject-deployment-cb7769df4-ljcfc   1/1     Running   0          2m
  myproject-deployment-cb7769df4-np9qd   0/1     Pending   0          26s
  </code></pre>
  <p>If I describe the pod I get the following error</p>
  <pre><code>$ kubectl describe po -n myproject myproject-deployment-cb7769df4-np9qd
  Name:           myproject-deployment-cb7769df4-np9qd
  Namespace:      myproject
  &lt;...&gt;
  Events:
    Type     Reason            Age   From                Message
    ----     ------            ----  ----                -------
    Warning  FailedScheduling  105s  default-scheduler   0/1 nodes are available: 1 Insufficient nvidia.com/gpu.
  </code></pre>
  <p>Since the first pod 'claims' the GPU, the second one cannot use it anymore and remains in status pending.
  I see two different possiblities:</p>
  <ol>
  <li>Run two TF serving containers in one pod on one GPU node</li>
  <li>Run two pods, each with one TF serving container on one GPU node</li>
  </ol>
  <p>Is any of the options above feasible?</p>
  <p>My deployment can be found below.</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: myproject-deployment
    labels:
      app: myproject-server
    namespace: myproject
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: myproject-server
    template:
      metadata:
        labels:
          app: myproject-server
      spec:
        containers:
        - name: server
          image: tensorflow/serving:2.3.0-gpu
          ports:
          - containerPort: 8500
          volumeMounts:
            - name: azurestorage
              mountPath: /models
          resources:
            requests:
              memory: &quot;10Gi&quot;
              cpu: &quot;1&quot;
            limits:
              memory: &quot;12Gi&quot;
              cpu: &quot;2&quot;
              nvidia.com/gpu: 1
          args: [&quot;--model_config_file=/models/models.config&quot;, &quot;--monitoring_config_file=/models/monitoring.config&quot;]
        volumes:
        - name: azurestorage
          persistentVolumeClaim:
            claimName: pvcmodels
  
  </code></pre>