title: Gitlab docker registry pull failed EOF
body: <h1>1. Introduction</h1>
  
  <p>I've got Gitlab setup using a docker image <a href="https://github.com/sameersbn/docker-gitlab" rel="nofollow noreferrer">https://github.com/sameersbn/docker-gitlab</a> (sameersbn/docker-gitlab) and want to setup gitlab private registry. </p>
  
  <h2>1.1  Problem</h2>
  
  <p>All of this is behind reverse proxy load balancer using <code>nginx</code>. I am able to push into registry, but pull has some problem I am not able to resolve. </p>
  
  <p>The output tells me <code>retrying in 5sec</code> and then increases over time, but after about minute it outputs <code>Download complete</code> on some layers. When all layers outputs <code>Download complete</code>. Pull results then like <code>Unexpected EOF</code>.</p>
  
  <p>Infrastructure goes like this:</p>
  
  <p><code>
  Proxy load balancer -----&gt; gitlab registry
                                       |
                                       |
  Another proxy load balancer -----&gt; Gitlab
  </code></p>
  
  <h1>2. Terminal outputs</h1>
  
  <p>Docker login and push works fine. The problem is with pull.</p>
  
  <h2>2.1 Docker pull</h2>
  
  <h3>2.1.1 expected result</h3>
  
  <pre><code>root@server:~# docker pull registry.dalten.com/docker/php5.6-cli/image
  Using default tag: latest
  latest: Pulling from docker/php5.6-cli/image
  10a267c67f42: Already exists 
  370377701f89: Pulling fs layer 
  455c73a122bc: Pulling fs layer 
  2cdcd82f5338: Pulling fs layer 
  3ac634f20449: Waiting 
  a19eeb6546ae: Waiting 
  4ca536f5a464: Waiting 
  4788559f39db: Waiting 
  cc1917e72a80: Waiting 
  42feb0ca362a: Pull complete 
  f74f29b91bbc: Pull complete 
  d104213bfd22: Pull complete 
  6db9d2d35d48: Pull complete 
  806b8ac2f0b7: Pull complete 
  661bb76178ac: Pull complete 
  3727be25d45f: Pull complete 
  04fed213644a: Pull complete 
  a78f18e044f7: Pull complete 
  849a2f576c66: Pull complete 
  259300decf26: Pull complete 
  0d9f42b121be: Pull complete 
  2483d273017a: Pull complete 
  3dfaf57e1d54: Pull complete 
  92fbf8438206: Pull complete 
  7d5100ec575c: Pull complete 
  52b1dff7320d: Pull complete 
  500fa7f84392: Pull complete 
  aff937c2773b: Pull complete 
  92f8bc1201f3: Pull complete 
  Digest: sha256:d3d06d2cebd333af21565639503ddded64db47d5641ee9ff93167200b5fcad7a
  Status: Downloaded newer image for registry.dalten.com/docker/php5.6-cli/image:latest
  </code></pre>
  
  <h3>2.1.2 Actual behavior</h3>
  
  <pre><code>docker pull registry.dalten.com/docker/php5.6-cli/image
  Using default tag: latest
  latest: Pulling from docker/php5.6-cli/image
  10a267c67f42: Already exists 
  370377701f89: Pulling fs layer 
  455c73a122bc: Pulling fs layer 
  2cdcd82f5338: Pulling fs layer 
  3ac634f20449: Waiting 
  a19eeb6546ae: Waiting 
  a19eeb6546ae: Downloading 4.398 MB/4.398 MB
  4788559f39db: Downloading 102.6 MB/102.6 MB
  cc1917e72a80: Download complete 
  42feb0ca362a: Download complete 
  f74f29b91bbc: Download complete 
  d104213bfd22: Download complete 
  6db9d2d35d48: Download complete 
  806b8ac2f0b7: Download complete 
  661bb76178ac: Download complete 
  3727be25d45f: Download complete 
  04fed213644a: Download complete 
  a78f18e044f7: Download complete 
  849a2f576c66: Download complete 
  259300decf26: Download complete 
  0d9f42b121be: Download complete 
  2483d273017a: Downloading 24.12 MB/24.12 MB
  3dfaf57e1d54: Download complete 
  92fbf8438206: Download complete 
  7d5100ec575c: Download complete 
  52b1dff7320d: Download complete 
  500fa7f84392: Download complete 
  aff937c2773b: Download complete 
  92f8bc1201f3: Download complete 
  unexpected EOF
  </code></pre>
  
  <h2>2.3 Docker login</h2>
  
  <pre><code>sudo docker login registry.dalten.com
  Username (xxxxxx): 
  Password: 
  Login Succeeded
  </code></pre>
  
  <h2>2.4 Docker push</h2>
  
  <pre><code>sudo docker push registry.dalten.com/docker/php5.6-cli/image
  The push refers to a repository [registry.dalten.com/docker/php5.6-cli/image]
  d8b353eb3025: Pushed 
  f2e85bc0b7b1: Pushed 
  fc9e1e5e38f7: Pushed 
  fe9a3f9c4559: Pushed 
  6a8bf8c8edbd: Pushed 
  latest: digest: sha256:ea1d854d38be82f54d39efe2c67000bed1b03348bcc2f3dc094f260855dff368 size: 1357
  </code></pre>
  
  <h1>3. Configurations</h1>
  
  <p>Configurations contains docker-compose.yml files which I use for running services. Gitlab is running fine. Every configuration works. The only problem is with registry.</p>
  
  <p>Another thing is that registry works while in insecured state. The problem must be hidden somewhere inside load balancer (nginx) configuration. (Some header has not been sent, limit exceeded, or so)</p>
  
  <h2>3.1 Gitlab Docker-compose</h2>
  
  <p><em>Note: I cutted compose of irrelevant information</em> </p>
  
  <pre><code>version: '2'
  services:
    registry:
      image: registry:latest
      environment:
        REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
        REGISTRY_AUTH_TOKEN_REALM: https://git.dalten.cz/jwt/auth
        REGISTRY_AUTH_TOKEN_SERVICE: container_registry
        REGISTRY_LOG_LEVEL: debug
        REGISTRY_STORAGE_DELETE_ENABLED: 'true'
        REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/fullchain.pem
        REGISTRY_HTTP_SECRET: secret
      volumes:
      - /mnt/storage/srv/gitlab1/data/shared/registry:/registry
      - /mnt/storage/srv/gitlab1/certs:/certs
      ports:
      - 5000:5000/tcp
    gitlab:
      image: sameersbn/gitlab:latest
      hostname: git.dalten.cz
      environment:
        GITLAB_REGISTRY_ENABLED: 'true'
        GITLAB_REGISTRY_HOST: registry.dalten.com
        GITLAB_REGISTRY_API_URL: http://registry:5000
        GITLAB_REGISTRY_ISSUER: gitlab-issuer
        GITLAB_REGISTRY_CERT_PATH: /certs/fullchain.pem
        GITLAB_REGISTRY_KEY_PATH: /certs/privkey.pem
      volumes:
      - /mnt/storage/srv/gitlab1/certs:/certs
      - /mnt/storage/srv/gitlab1/data:/home/git/data/
      ports:
      - 8643:80/tcp
      - 1022:22/tcp
  </code></pre>
  
  <h2>3.2 Nginx registry docker-compose</h2>
  
  <pre><code>version: '2'
  services:
    nginx:
      image: sameersbn/nginx:1.10.3
      volumes:
      - /srv/nginx/sites-enabled:/etc/nginx/sites-enabled
      - /mnt/storage/lb/letsencrypt:/etc/nginx/ssl
      ports:
      - 80:80/tcp
      - 443:443/tcp
  </code></pre>
  
  <h3>3.2.1 Nginx Virtual host file</h3>
  
  <p>I use Nginx for my reverse proxy configuration (Load Balancer). There are stored all certificates for https traffic.</p>
  
  <pre><code>    # For versions of Nginx &gt; 1.3.9 that include chunked transfer encoding
  # support Replace with appropriate values where necessary
  upstream docker-registry {
   server 10.126.0.242:5000;
  }
  
  server {
    listen 80;
    server_name registry.dalten.com
    return 301 https://$http_host:$request_uri;
    access_log /var/log/nginx/access.log;
    error_log  /var/log/nginx/error.log;
  }
  
  server {
   listen 443 default_server;
   server_name registry.dalten.com
  
   add_header Docker-Distribution-Api-Version registry/2.0 always;
  
   ssl on;
   ssl_certificate /etc/nginx/ssl/live/registry.dalten.com/fullchain.pem;
   ssl_certificate_key /etc/nginx/ssl/live/registry.dalten.com/privkey.pem;
  
    ssl_session_timeout 5m;
  
   client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image
  uploads
   # required to avoid HTTP 411: see Issue #1486
   # (https://github.com/dotcloud/docker/issues/1486)
   chunked_transfer_encoding on;
   proxy_set_header X-Original-URI $request_uri;
   proxy_set_header Docker-Distribution-Api-Version registry/2.0;
  
  access_log /var/log/nginx/access.log;
   error_log /var/log/nginx/error.log;
  
  location /
  {
       # let Nginx know about our auth file
       proxy_pass http://docker-registry;
       proxy_set_header Host $host; # required for docker client's sake
       proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
   }
   location /_ping {
       proxy_pass http://docker-registry;
   }
   location /v1/_ping {
       proxy_pass http://docker-registry;
   }
   location /v2/ {
       # To add basic authentication to v2 use auth_basic setting plus
       # add_header
       add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
       proxy_pass http://docker-registry;
       proxy_set_header Host $http_host; # required for docker client's sake
       proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_read_timeout 900;
   }
  }
  </code></pre>
  
  <h1>3. Logs</h1>
  
  <p>If you require any more logs feel free to ask!</p>
  
  <h2>3.1 <a href="https://pastebin.com/gTthx9EY" rel="nofollow noreferrer">Docker pull</a></h2>
  
  <h1>4. Thank you for all your help!</h1>