title: Kubernetes nginx ingress controller kills connections
body: <p>I am a newbie to k8s and I am trying to deploy a private docker registry in Kubernetes.</p>
  <p>The problem is that whenever I have to upload a heavy image (1GB size) via <code>docker push</code>, the command eventually returns EOF.</p>
  <p>Apparently, I believe the issue has to do with kubernetes ingress nginx controller.</p>
  <p>I will provide you with some useful information, in case you need more, do not hesitate to ask:</p>
  <p>Docker push (to internal k8s docker registry) fail:</p>
  <pre><code>[root@bastion ~]# docker push docker-registry.apps.kube.lab/example:stable
  The push refers to a repository [docker-registry.apps.kube.lab/example]
  c0acde035881: Pushed 
  f6d2683cee8b: Pushed 
  00b1a6ab6acd: Retrying in 1 second 
  28c41b4dd660: Pushed 
  36957997ca7a: Pushed 
  5c4d527d6b3a: Pushed 
  a933681cf349: Pushing [==================================================&gt;] 520.4 MB
  f49d20b92dc8: Retrying in 20 seconds 
  fe342cfe5c83: Retrying in 15 seconds 
  630e4f1da707: Retrying in 13 seconds 
  9780f6d83e45: Waiting 
  EOF
  </code></pre>
  <p>Ingress definition:</p>
  <pre><code>apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: docker-registry
    namespace: docker-registry
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: &quot;0&quot;
      nginx.ingress.kubernetes.io/proxy-connect-timeout: &quot;86400&quot;
      nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;86400&quot;
      nginx.ingress.kubernetes.io/proxy-send-timeout: &quot;86400&quot;
  spec:    
    rules: 
    - host: docker-registry.apps.kube.lab
      http:
        paths:
        - backend:
            serviceName: docker-registry
            servicePort: 5000
          path: /  
  </code></pre>
  <p>Docker registry configuration (/etc/docker/registry/config.yml):</p>
  <pre><code>version: 0.1
  log: 
    level: info
    formatter: json
    fields:
      service: registry
  storage:
    redirect:
      disable: true                                                                                                                                                                                                 
    cache:
      blobdescriptor: inmemory
    filesystem:
      rootdirectory: /var/lib/registry
  http:
    addr: :5000
    host: docker-registry.apps.kube.lab
    headers:
      X-Content-Type-Options: [nosniff]
  health:
    storagedriver:
      enabled: true
      interval: 10s
      threshold: 3
  
  </code></pre>
  <p>Docker registry logs:</p>
  <pre><code>{&quot;go.version&quot;:&quot;go1.11.2&quot;,&quot;http.request.host&quot;:&quot;docker-registry.apps.kube.lab&quot;,&quot;http.request.id&quot;:&quot;c079b639-0e8a-4a27-96fa-44c4c0182ff7&quot;,&quot;http.request.method&quot;:&quot;HEAD&quot;,&quot;http.request.remoteaddr&quot;:&quot;10.233.70.0&quot;,&quot;http.request.uri&quot;:&quot;/v2/example/blobs/sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;http.request.useragent&quot;:&quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))&quot;,&quot;level&quot;:&quot;debug&quot;,&quot;msg&quot;:&quot;authorizing request&quot;,&quot;time&quot;:&quot;2020-11-07T14:43:22.893626513Z&quot;,&quot;vars.digest&quot;:&quot;sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;vars.name&quot;:&quot;example&quot;}
  {&quot;go.version&quot;:&quot;go1.11.2&quot;,&quot;http.request.host&quot;:&quot;docker-registry.apps.kube.lab&quot;,&quot;http.request.id&quot;:&quot;c079b639-0e8a-4a27-96fa-44c4c0182ff7&quot;,&quot;http.request.method&quot;:&quot;HEAD&quot;,&quot;http.request.remoteaddr&quot;:&quot;10.233.70.0&quot;,&quot;http.request.uri&quot;:&quot;/v2/example/blobs/sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;http.request.useragent&quot;:&quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))&quot;,&quot;level&quot;:&quot;debug&quot;,&quot;msg&quot;:&quot;GetBlob&quot;,&quot;time&quot;:&quot;2020-11-07T14:43:22.893751065Z&quot;,&quot;vars.digest&quot;:&quot;sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;vars.name&quot;:&quot;example&quot;}
  {&quot;go.version&quot;:&quot;go1.11.2&quot;,&quot;http.request.host&quot;:&quot;docker-registry.apps.kube.lab&quot;,&quot;http.request.id&quot;:&quot;c079b639-0e8a-4a27-96fa-44c4c0182ff7&quot;,&quot;http.request.method&quot;:&quot;HEAD&quot;,&quot;http.request.remoteaddr&quot;:&quot;10.233.70.0&quot;,&quot;http.request.uri&quot;:&quot;/v2/example/blobs/sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;http.request.useragent&quot;:&quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))&quot;,&quot;level&quot;:&quot;debug&quot;,&quot;msg&quot;:&quot;filesystem.GetContent(\&quot;/docker/registry/v2/repositories/example/_layers/sha256/751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee/link\&quot;)&quot;,&quot;time&quot;:&quot;2020-11-07T14:43:22.893942372Z&quot;,&quot;trace.duration&quot;:74122,&quot;trace.file&quot;:&quot;/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go&quot;,&quot;trace.func&quot;:&quot;github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent&quot;,&quot;trace.id&quot;:&quot;11e24830-7d16-404a-90bc-8a738cab84ea&quot;,&quot;trace.line&quot;:95,&quot;vars.digest&quot;:&quot;sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;vars.name&quot;:&quot;example&quot;}
  {&quot;err.code&quot;:&quot;blob unknown&quot;,&quot;err.detail&quot;:&quot;sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;err.message&quot;:&quot;blob unknown to registry&quot;,&quot;go.version&quot;:&quot;go1.11.2&quot;,&quot;http.request.host&quot;:&quot;docker-registry.apps.kube.lab&quot;,&quot;http.request.id&quot;:&quot;c079b639-0e8a-4a27-96fa-44c4c0182ff7&quot;,&quot;http.request.method&quot;:&quot;HEAD&quot;,&quot;http.request.remoteaddr&quot;:&quot;10.233.70.0&quot;,&quot;http.request.uri&quot;:&quot;/v2/example/blobs/sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;http.request.useragent&quot;:&quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))&quot;,&quot;http.response.contenttype&quot;:&quot;application/json; charset=utf-8&quot;,&quot;http.response.duration&quot;:&quot;1.88607ms&quot;,&quot;http.response.status&quot;:404,&quot;http.response.written&quot;:157,&quot;level&quot;:&quot;error&quot;,&quot;msg&quot;:&quot;response completed with error&quot;,&quot;time&quot;:&quot;2020-11-07T14:43:22.894147954Z&quot;,&quot;vars.digest&quot;:&quot;sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee&quot;,&quot;vars.name&quot;:&quot;example&quot;}
  10.233.105.66 - - [07/Nov/2020:14:43:22 +0000] &quot;HEAD /v2/example/blobs/sha256:751620502a7a2905067c2f32d4982fb9b310b9808670ce82c0e2b40f5307a3ee HTTP/1.1&quot; 404 157 &quot;&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))&quot;
  
  </code></pre>
  <p>I believe the issue has to do with ingress controller because when EOF error shows up, there is something weird in <strong>ingress-controller logs</strong>:</p>
  <pre><code>10.233.70.0 - - [07/Nov/2020:14:43:41 +0000] &quot;PUT /v2/example/blobs/uploads/dab984a8-7e71-4481-91fb-af53c7790a20?_state=usMX2WH24Veunay0ozOF-RMZIUMNTFSC8MSPbMcxz-B7Ik5hbWUiOiJleGFtcGxlIiwiVVVJRCI6ImRhYjk4NGE4LTdlNzEtNDQ4MS05MWZiLWFmNTNjNzc5MGEyMCIsIk9mZnNldCI6NzgxMTczNywiU3RhcnRlZEF0IjoiMjAyMC0xMS0wN1QxNDo0MzoyOFoifQ%3D%3D&amp;digest=sha256%3A101c41d0463bc77661fb3343235b16d536a92d2efb687046164d413e51bd4fc4 HTTP/1.1&quot; 201 0 &quot;-&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))&quot; 606 0.026 [docker-registry-docker-registry-5000] [] 10.233.70.84:5000 0 0.026 201 06304ff584d252812dff016374be73ae
  172.16.1.123 - - [07/Nov/2020:14:43:42 +0000] &quot;HEAD /v2/example/blobs/sha256:101c41d0463bc77661fb3343235b16d536a92d2efb687046164d413e51bd4fc4 HTTP/1.1&quot; 200 0 &quot;-&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))&quot; 299 0.006 [docker-registry-docker-registry-5000] [] 10.233.70.84:5000 0 0.006 200 a5a93c7b7f4644139fcb0697d3e5e43f
  I1107 14:44:05.285478       6 main.go:184] &quot;Received SIGTERM, shutting down&quot;
  I1107 14:44:05.285517       6 nginx.go:365] &quot;Shutting down controller queues&quot;
  I1107 14:44:06.294533       6 status.go:132] &quot;removing value from ingress status&quot; address=[172.16.1.123]
  I1107 14:44:06.306793       6 status.go:277] &quot;updating Ingress status&quot; namespace=&quot;kube-system&quot; ingress=&quot;example-ingress&quot; currentValue=[{IP:172.16.1.123 Hostname:}] newValue=[]
  I1107 14:44:06.307650       6 status.go:277] &quot;updating Ingress status&quot; namespace=&quot;kubernetes-dashboard&quot; ingress=&quot;dashboard&quot; currentValue=[{IP:172.16.1.123 Hostname:}] newValue=[]
  I1107 14:44:06.880987       6 status.go:277] &quot;updating Ingress status&quot; namespace=&quot;test-nfs&quot; ingress=&quot;example-nginx&quot; currentValue=[{IP:172.16.1.123 Hostname:}] newValue=[]
  I1107 14:44:07.872659       6 status.go:277] &quot;updating Ingress status&quot; namespace=&quot;test-ingress&quot; ingress=&quot;example-ingress&quot; currentValue=[{IP:172.16.1.123 Hostname:}] newValue=[]
  I1107 14:44:08.505295       6 queue.go:78] &quot;queue has been shutdown, failed to enqueue&quot; key=&quot;&amp;ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:&lt;nil&gt;,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}&quot;
  I1107 14:44:08.713579       6 status.go:277] &quot;updating Ingress status&quot; namespace=&quot;docker-registry&quot; ingress=&quot;docker-registry&quot; currentValue=[{IP:172.16.1.123 Hostname:}] newValue=[]
  I1107 14:44:09.772593       6 nginx.go:373] &quot;Stopping admission controller&quot;
  I1107 14:44:09.772697       6 nginx.go:381] &quot;Stopping NGINX process&quot;
  E1107 14:44:09.773208       6 nginx.go:314] &quot;Error listening for TLS connections&quot; err=&quot;http: Server closed&quot;
  2020/11/07 14:44:09 [notice] 114#114: signal process started
  10.233.70.0 - - [07/Nov/2020:14:44:16 +0000] &quot;PATCH /v2/example/blobs/uploads/adbe3173-9928-4eb5-97bb-7893970f032a?_state=nEr2ip9eoLNCTe8KQ6Ck7k3C8oS9IY7AnBOi1_f5mSl7Ik5hbWUiOiJleGFtcGxlIiwiVVVJRCI6ImFkYmUzMTczLTk5MjgtNGViNS05N2JiLTc4OTM5NzBmMDMyYSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0xMS0wN1QxNDo0MzoyOC45ODY3MTQwNTlaIn0%3D HTTP/1.1&quot; 202 0 &quot;-&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))&quot; 50408825 46.568 [docker-registry-docker-registry-5000] [] 10.233.70.84:5000 0 14.339 202 55d9cab4f915f54e5c130321db4dc8fc
  10.233.70.0 - - [07/Nov/2020:14:44:19 +0000] &quot;PATCH /v2/example/blobs/uploads/63d4a54a-cdfd-434b-ae63-dc434dcb15f9?_state=9UK7MRYJYST--u7BAUFTonCdPzt_EO2KyfJblVroBxd7Ik5hbWUiOiJleGFtcGxlIiwiVVVJRCI6IjYzZDRhNTRhLWNkZmQtNDM0Yi1hZTYzLWRjNDM0ZGNiMTVmOSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0xMS0wN1QxNDo0MzoyMy40MjIwMDI4NThaIn0%3D HTTP/1.1&quot; 202 0 &quot;-&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))&quot; 51842691 55.400 [docker-registry-docker-registry-5000] [] 10.233.70.84:5000 0 18.504 202 1f1de1ae89caa8540b6fd13ea5b165ab
  10.233.70.0 - - [07/Nov/2020:14:44:50 +0000] &quot;PATCH /v2/example/blobs/uploads/0c97923d-ed9f-4599-8a50-f2c21cfe85fe?_state=WmIRW_3owlin1zo4Ms98UwaMGf1D975vUuzbk1JWRuN7Ik5hbWUiOiJleGFtcGxlIiwiVVVJRCI6IjBjOTc5MjNkLWVkOWYtNDU5OS04YTUwLWYyYzIxY2ZlODVmZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0xMS0wN1QxNDo0MzoyMC41ODA5MjUyNDlaIn0%3D HTTP/1.1&quot; 202 0 &quot;-&quot; &quot;docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))&quot; 192310965 89.937 [docker-registry-docker-registry-5000] [] 10.233.70.84:5000 0 22.847 202 d8971d2f543e936c2f805d5b257f1130
  I1107 14:44:50.832669       6 nginx.go:394] &quot;NGINX process has stopped&quot;
  I1107 14:44:50.832703       6 main.go:192] &quot;Handled quit, awaiting Pod deletion&quot;
  I1107 14:45:00.832892       6 main.go:195] &quot;Exiting&quot; code=0
  [root@bastion registry]# 
  </code></pre>
  <p>After that happens, ingres-controller pod is not ready, and after some seconds it is again ready.</p>
  <p>Is it to do with config reload of kubernetes nginx ingress controller? In such case, do I have to add any special variable to nginx.conf?</p>
  <p>Any help is welcome! Kind regards!</p>
  <p><strong>EDIT</strong></p>
  <p>The moment EOF appears, ingress-nginx crashes, and pods become not ready.</p>
  <pre><code>[root@bastion ~]# kubectl get po 
  NAME                                        READY   STATUS      RESTARTS   AGE
  ingress-nginx-admission-create-lbmd6        0/1     Completed   0          5d4h
  ingress-nginx-admission-patch-btv27         0/1     Completed   0          5d4h
  ingress-nginx-controller-7dcc8d6478-n8dkx   0/1     Running     3          15m
  
   Warning  Unhealthy  29s (x8 over 2m39s)   kubelet                   Liveness probe failed: Get http://10.233.70.100:10254/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  
  </code></pre>
  <p>As a consequence, any of my applications are not reachable:</p>
  <pre><code>[root@bastion ~]# curl http://hello-worrld.apps.kube.lab
  Hello, world!
  Version: 1.0.0
  Hostname: web-6785d44d5-4r5q5
  [root@bastion ~]# date
  sáb nov  7 18:58:16 -01 2020
  
  [root@bastion ~]# curl http://hello-worrld.apps.kube.lab
  curl: (52) Empty reply from server
  [root@bastion ~]# date
  sáb nov  7 18:58:53 -01 2020
  </code></pre>
  <p>Is the issue to do with performance of nginx? If so, what options would you recommend me to tweak ingress-nginx?</p>