title: Kubernetes pods graceful shutdown with TCP connections (Spring boot)
body: <p>I am hosting my services on azure cloud, sometimes I get "BackendConnectionFailure" without any apparent reason, after investigation I found a correlation between this exception and autoscale (scaling down) almost at the same second in most of the cases.</p>
  
  <p>According to documentation termination grace period by default is 30 seconds, which is the case. The pod will be marked terminating and the loadbalancer will not consider it anymore, so receiving no more requests. According to this if my service takes far less time than 30 seconds, I should not need prestop hook or any special implementation in my application (please correct me if I am wrong).</p>
  
  <p>If the previous paragraph is correct, why does this exception occur relatively frequent? My thought is when the pod is marked terminating and the loadbalancer does not forward anymore requests to the pod while it should do.</p>
  
  <p>Edit 1:</p>
  
  <p>The Architecture is simply like this</p>
  
  <p>Client -> Firewall(azure) -> API(azure APIM) -> Microservices(Spring boot) -> backend(third party) or azure RDB depending on the service</p>
  
  <p>I think the Exception comes from APIM, I found two patterns for this exception:</p>
  
  <ol>
  <li><p><code>Message   The underlying connection was closed: The connection was closed unexpectedly.
  Exception type  BackendConnectionFailure 
  Failed method   forward-request</code></p></li>
  </ol>
  
  <p><code>Response time  10.0 s</code></p>
  
  <ol start="2">
  <li><p><code>Message    The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
  Exception type  BackendConnectionFailure
  Failed method   forward-request</code></p></li>
  </ol>
  
  <p><code>Response time  3.6 ms</code></p>