title: Why does pushing to private, secured docker registry fail?
body: <p>I would like to run a private, secure, authenticated docker registry on a self healing AWS ECS cluster. The cluster setup is done and works properly, but I struggled getting the <code>registry:latest</code> running. The problem was, that every time I push an image, pushing the blobs fails, and goes into a retry cycle unless I get a timeout.</p>
  
  <p>To make sure, that my ECS setup isn’t the blocker, I tried to setup everything locally using Docker4Mac 1.12.0-a.</p>
  
  <p>First, the very basic setup works. I created my own version of the registry image, where there I put my TLS certificate bundle and key as well as the necessary <code>htpasswd</code> file directly into the image. [I know, this is insecure, I just do that for the testing purpose]. So here is my <code>Dockerfile</code>:</p>
  
  <pre><code>FROM registry:latest
  
  COPY htpasswd /etc/docker
  COPY server_bundle.pem /etc/docker
  COPY server_key.pem /etc/docker
  </code></pre>
  
  <p><code>server_bundle.pem</code> has a <strong>wildcard certificate</strong> for my domain <code>mydomain.com</code> (CN=*.mydomain.com) as the first one, followed by the intermediate CA certificates, so clients should be happy. My <code>htpasswd</code> file was created using the recommended approach:</p>
  
  <pre><code>docker run --entrypoint htpasswd registry:2 -Bbn heyitsme mysupersecurepassword &gt; htpasswd
  </code></pre>
  
  <p>I build my image:</p>
  
  <pre><code>docker build -t heyitsme/registry .
  </code></pre>
  
  <p>and afterwards I run a very basic version w/o TLS and authentication:</p>
  
  <pre><code>docker run --restart=always -p 5000:5000 heyitsme/registry
  </code></pre>
  
  <p>and I can actually pull, tag and re-push an image:</p>
  
  <pre><code>docker pull alpine
  docker tag alpine localhost:5000/alpine
  docker push localhost:5000/alpine
  </code></pre>
  
  <p>This works. Next I make TLS and basic auth work via environment variables:</p>
  
  <pre><code>docker run -h registry.mydomain.com --name registry --restart=always -p 5000:5000 \ 
    -e REGISTRY_HTTP_HOST=http://registry.mydomain.com:5000 \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/docker/server_bundle.pem \
    -e REGISTRY_HTTP_TLS_KEY=/etc/docker/server_key.pem \
    -e REGISTRY_AUTH=htpasswd \
    -e REGISTRY_AUTH_HTPASSWD_REALM=Registry-Realm \
    -e REGISTRY_AUTH_HTPASSWD_PATH=/etc/docker/htpasswd heyitsme/registry
  </code></pre>
  
  <p>For the time being I create an entry in <code>/etc/hosts</code> which says:</p>
  
  <pre><code>127.0.0.1 registry.mydomain.com
  </code></pre>
  
  <p>And then I login: </p>
  
  <pre><code>docker login registry.mydomain.com:5000
  Username: heyitsme
  Password: ***********
  Login Succeeded
  </code></pre>
  
  <p>So now, let’s tag and push the image here:</p>
  
  <pre><code>docker tag alpine registry.mydomain.com:5000/alpine
  docker push registry.mydomain.com:5000/alpine
  
  The push refers to a repository [registry.mydomain.com:5000/alpine]
  4fe15f8d0ae6: Retrying in 4 seconds
  </code></pre>
  
  <p>What happens is, that the docker clients tries to push fragments and fails. Then it retries and fails again until I get a timeout. So next check, whether the <code>V2</code> API works properly:</p>
  
  <pre><code>curl -i -XGET https://registry.mydomain.com:5000/v2/
  HTTP/1.1 401 Unauthorized
  Content-Type: application/json; charset=utf-8
  Docker-Distribution-Api-Version: registry/2.0
  Www-Authenticate: Basic realm="Registry-Realm"
  X-Content-Type-Options: nosniff
  Date: Thu, 15 Sep 2016 10:06:04 GMT
  Content-Length: 87
  
  {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
  </code></pre>
  
  <p>Ok, as expected. So let’s authenticate next time:</p>
  
  <pre><code>curl -i -XGET https://heyitsme:mysupersecretpassword@registry.mydomain.com:5000/v2/
  HTTP/1.1 200 OK
  Content-Length: 2
  Content-Type: application/json; charset=utf-8
  Docker-Distribution-Api-Version: registry/2.0
  X-Content-Type-Options: nosniff
  Date: Thu, 15 Sep 2016 10:06:16 GMT
  
  {}%
  </code></pre>
  
  <p>Works. But pushing still fails. </p>
  
  <p>The logs say:</p>
  
  <pre><code>time="2016-09-15T10:24:34Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry-Realm\": invalid authorization credential" go.version=go1.6.3 http.request.host="registry.mydomain.com:5000" http.request.id=6d2ec080-6824-4bf7-aac2-5af31db44877 http.request.method=GET http.request.remoteaddr="172.17.0.1:40878" http.request.uri="/v2/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))" instance.id=be3a8877-de64-4574-b47a-70ab036e7b79 version=v2.5.1
  172.17.0.1 - - [15/Sep/2016:10:24:34 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))"
  time="2016-09-15T10:24:34Z" level=info msg="response completed" go.version=go1.6.3 http.request.host="registry.mydomain.com:5000" http.request.id=8f81b455-d592-431d-b67d-0bc34155ddbf http.request.method=POST http.request.remoteaddr="172.17.0.1:40882" http.request.uri="/v2/alpine/blobs/uploads/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))" http.response.duration=30.515131ms http.response.status=202 http.response.written=0 instance.id=be3a8877-de64-4574-b47a-70ab036e7b79 version=v2.5.1
  172.17.0.1 - - [15/Sep/2016:10:24:34 +0000] "POST /v2/alpine/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(darwin\\))"
  2016/09/15 10:24:34 http: TLS handshake error from 172.17.0.1:40886: tls: first record does not look like a TLS handshake
  </code></pre>
  
  <p>I also tested different versions of the original <code>registry</code> image, especially several versions above <code>2</code>. All yield the same error. If someone could help me on that issue, that would be awesome.</p>