title: Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Socket closed)>
body: <ul>
  <li>tensorflow-gpu 1.10.0</li>
  <li>tensorflow-server 1.10.0</li>
  </ul>
  
  <p>I have deployed a tensorflow server which serves several models.
  The client code is like <code>client.py</code> this and I call the predict function.</p>
  
  <pre><code>channel = implementations.insecure_channel(host, port)
  stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
  request = predict_pb2.PredictRequest()
  
  def predict(data, shape, model_name, signature_name="predict"):
      request.model_spec.name = model_name
      request.model_spec.signature_name = signature_name
      request.inputs['image'].CopyFrom(tf.contrib.util.make_tensor_proto(data, shape=shape))
      result = stub.Predict(request, 10.0)
      return result.outputs['prediction'].float_val[0]
  </code></pre>
  
  <p>I have about 100 clients with the same configuration.
  And here is a sample code to call the <code>predict</code> function:</p>
  
  <pre><code>from client import predict
  while True:
      print(predict(data, shape, model_name))
      # time.sleep some while
  </code></pre>
  
  <p>At first, when I run the client code, I can receieve the reponse <strong>correctly</strong>.
  But after several hours, the client crashed with the error</p>
  
  <pre><code>_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Socket closed)
  </code></pre>
  
  <p>I have try to modify my client code to</p>
  
  <pre><code>def predict(data, shape, model_name, signature_name="predict"):
      channel = implementations.insecure_channel(host, port)
      stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
      request = predict_pb2.PredictRequest()
      request.model_spec.name = model_name
      request.model_spec.signature_name = signature_name
      request.inputs['image'].CopyFrom(tf.contrib.util.make_tensor_proto(data, shape=shape))
      result = stub.Predict(request, 10.0)
      return result.outputs['prediction'].float_val[0]
  </code></pre>
  
  <p>that means I try to establish the connection with the tfs server each time the <code>predict</code> function is called. But this code also failed just like before.</p>
  
  <p>So what should I do to deal with this situation?</p>