title: SSH from shared Gitlab runner stopped working
body: <p><em>This <strong>did</strong> work previously!</em></p>
  <p>My deployment step in my pipeline SSH's onto a DO box &amp; pulls the code from a docker registry. As mentioned, this worked previously &amp; this was my <code>deploy</code> step in my <code>.gitlab-ci.yml</code> back then which worked fine inspiration from <a href="https://medium.com/@codingfriend/continuous-integration-and-deployment-with-gitlab-docker-compose-and-digitalocean-6bd6196b502a" rel="nofollow noreferrer">here</a> under <code>Using SSH</code>:</p>
  <pre class="lang-yaml prettyprint-override"><code>deploy:
    stage: deploy
    image: docker:stable-dind
    only:
      - master
    services:
      # Specifying the DinD version here as the latest DinD version introduced a timeout bug
      # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
      - docker:19.03.5-dind
    variables:
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: &quot;&quot;
    environment:
      name: production
    when: manual
    before_script:
      - mkdir -p ~/.ssh
      - echo &quot;$DEPLOYMENT_SERVER_PRIVATE_KEY&quot; | tr -d '\r' &gt; ~/.ssh/id_rsa
      - chmod 600 ~/.ssh/id_rsa
      - eval &quot;$(ssh-agent -S)&quot;
      - ssh-add ~/.ssh/id_rsa
      - ssh-keyscan -H $DEPLOYMENT_SERVER_IP &gt;&gt; ~/.ssh/known_hosts
    script:
      - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
        &quot;docker stop ${CI_PROJECT_NAME};
        docker rm ${CI_PROJECT_NAME};
        docker container prune -f;
        docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
        docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
        docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
        docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;&quot;
  </code></pre>
  <p>Once I just attempted to run the <code>deploy</code> step &amp; failed. Coming back with this error:</p>
  <pre class="lang-sh prettyprint-override"><code>...
   $ mkdir -p ~/.ssh
   $ echo &quot;${DEPLOYMENT_SERVER_PRIVATE_KEY}&quot; | tr -d '\r' &gt; ~/.ssh/id_rsa
   $ chmod 600 ~/.ssh/id_rsa
   $ eval &quot;$(ssh-agent -s)&quot;
   Agent pid 22
   $ ssh-add ~/.ssh/id_rsa
   Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
   $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} &gt;&gt; ~/.ssh/known_hosts
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   $ ssh gitlab@${DEPLOYMENT_SERVER_IP} &quot;docker stop ${CI_PROJECT_NAME}; docker rm ${CI_PROJECT_NAME}; docker container prune -f; docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH}; docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}; docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest; docker run -d -p ${PORT}:${PORT} --restart always -m 2g --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;&quot;
   ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
  Running after_script
  00:02
  Uploading artifacts for failed job
  00:01
   ERROR: Job failed: exit code 255
  </code></pre>
  <p>Steps I took to set this up originally</p>
  <ul>
  <li>Run <code>ssh-keygen -t rsa -b 2048</code> on DO box <em>(with no password)</em></li>
  <li>Added <strong>public</strong> key into <code>authorized_keys</code> on the DO box</li>
  <li>Copy the <strong>private</strong> key into a CI variable <code>DEPLOYMENT_SERVER_PRIVATE_KEY</code></li>
  </ul>
  <p>I know the port is open for SSH as I am able to SSH from my local machine into <code>gitlab</code> user. I have now changed my deployment step <em>(based on comments from <a href="https://forum.gitlab.com/t/ssh-to-remote-server-refused-if-done-via-gitlab-ci/31478" rel="nofollow noreferrer">here</a>, <a href="https://docs.gitlab.com/ee/ci/ssh_keys/README.html#ssh-keys-when-using-the-docker-executor" rel="nofollow noreferrer">this article</a>, &amp; <a href="https://medium.com/@hfally/a-gitlab-ci-config-to-deploy-to-your-server-via-ssh-43bf3cf93775" rel="nofollow noreferrer">this one</a>)</em> to:</p>
  <pre class="lang-yaml prettyprint-override"><code>deploy:
    stage: deploy
    image: docker:stable-dind
    only:
      - master
    services:
      # Specifying the DinD version here as the latest DinD version introduced a timeout bug
      # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
      - docker:19.03.5-dind
    variables:
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: &quot;&quot;
    environment:
      name: production
    when: manual
    before_script:
      - 'which ssh-agent || ( apt-get update -y &amp;&amp; apt-get install openssh-client -y )'
      - eval $(ssh-agent -s)
      - echo &quot;$DEPLOYMENT_SERVER_PRIVATE_KEY&quot; | tr -d '\r' | ssh-add - &gt; /dev/null
      - mkdir -p ~/.ssh
      - chmod 700 ~/.ssh
      - '[[ -f /.dockerenv ]] &amp;&amp; echo -e &quot;Host *\n\tStrictHostKeyChecking no\n\n&quot; &gt; ~/.ssh/config'
      - cat ~/.ssh/config
      - echo ${CI_REGISTRY_USER}
      - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} &gt;&gt; ~/.ssh/known_hosts
    script:
      - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
        &quot;docker stop ${CI_PROJECT_NAME};
        docker rm ${CI_PROJECT_NAME};
        docker container prune -f;
        docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
        docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
        docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
        docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;&quot;
  </code></pre>
  <p>Still to no avail! The verbosing logging of <code>ssh</code> spit out:</p>
  <pre class="lang-sh prettyprint-override"><code>...
   $ which ssh-agent || ( apt-get update -y &amp;&amp; apt-get install openssh-client -y )
   /usr/bin/ssh-agent
   $ eval $(ssh-agent -s)
   Agent pid 18
   $ echo &quot;$DEPLOYMENT_SERVER_PRIVATE_KEY&quot; | tr -d '\r' | ssh-add - &gt; /dev/null
   Identity added: (stdin) ((stdin))
   $ mkdir -p ~/.ssh
   $ chmod 700 ~/.ssh
   $ [[ -f /.dockerenv ]] &amp;&amp; echo -e &quot;Host *\n\tStrictHostKeyChecking no\n\n&quot; &gt; ~/.ssh/config
   $ cat ~/.ssh/config
   Host *
      StrictHostKeyChecking no
   $ echo ${CI_REGISTRY_USER}
   gitlab-ci-token
   $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} &gt;&gt; ~/.ssh/known_hosts
   # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
   $ ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
   OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
   debug1: Reading configuration data /root/.ssh/config
   debug1: /root/.ssh/config line 1: Applying options for *
   debug1: Reading configuration data /etc/ssh/ssh_config
   debug2: resolve_canonicalize: hostname 134.xxx.xxx.xxx is address
   Pseudo-terminal will not be allocated because stdin is not a terminal.
   debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
   debug2: ssh_connect_direct
   debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
   debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
   ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
   ERROR: Job failed: exit code 255
  </code></pre>
  <p>I also added the <code>-T</code> option <a href="https://stackoverflow.com/questions/7114990/pseudo-terminal-will-not-be-allocated-because-stdin-is-not-a-terminal">suggested here</a> to disable pseudo-tty allocation but all that did was remove the pseudo line from the logs.</p>
  <p><strong>EDIT</strong></p>
  <p>Looking at the logs on the DO box (<em><code>/var/log/auth.log</code></em>), I've got the error:</p>
  <pre><code>Jun 22 15:53:37 exchange-apis sshd[16159]: Connection closed by 35.190.162.232 port 49750 [preauth]
  Jun 22 15:53:38 exchange-apis sshd[16160]: Connection closed by 35.190.162.232 port 49754 [preauth]
  Jun 22 15:53:38 exchange-apis sshd[16162]: Connection closed by 35.190.162.232 port 49752 [preauth]
  Jun 22 15:53:38 exchange-apis sshd[16163]: Unable to negotiate with 35.190.162.232 port 49756: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
  Jun 22 15:53:38 exchange-apis sshd[16161]: Unable to negotiate with 35.190.162.232 port 49758: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]
  </code></pre>
  <p>Googling this error, common cause seems to be due to OpenSSH dropping support for DSA keys. However, not sure why this would effect me as I generated an RSA key pair. Anyway, running <code>dpkg --list | grep openssh</code> spits out:</p>
  <pre><code>ii  openssh-client                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) client, for secure access to remote machines
  ii  openssh-server                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) server, for secure access from remote machines
  ii  openssh-sftp-server                    1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines
  </code></pre>
  <p>&amp; <code>sshd -v</code> spits out:</p>
  <pre><code>OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
  </code></pre>
  <p>Nevertheless, worked of the answers; <a href="https://stackoverflow.com/questions/34208495/unable-to-negotiate-with-xx-xxx-xx-xx-no-matching-host-key-type-found-their-of">here</a> &amp; <a href="https://askubuntu.com/questions/836048/ssh-returns-no-matching-host-key-type-found-their-offer-ssh-dss">here</a> so my <code>deploy</code> stage is now:</p>
  <pre class="lang-yaml prettyprint-override"><code>deploy:
    stage: deploy
    image: docker:stable-dind
    only:
      - master
    services:
      # Specifying the DinD version here as the latest DinD version introduced a timeout bug
      # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
      - docker:19.03.5-dind
    variables:
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: &quot;&quot;
    environment:
      name: production
    when: manual
    before_script:
      - 'which ssh-agent || ( apt-get update -y &amp;&amp; apt-get install openssh-client -y )'
      - mkdir -p ~/.ssh
      - echo &quot;$DEPLOYMENT_SERVER_PRIVATE_KEY&quot; | tr -d '\r' &gt; ~/.ssh/id_rsa
      - chmod 600 ~/.ssh/id_rsa
      - '[[ -f /.dockerenv ]] &amp;&amp; echo -e &quot;Host *\n\tStrictHostKeyChecking no\n\tHostkeyAlgorithms +ssh-dss\n\tPubkeyAcceptedKeyTypes +ssh-dss\n\n&quot; &gt; ~/.ssh/config'
      - cat ~/.ssh/config
      - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} &gt;&gt; ~/.ssh/known_hosts
      - chmod 644 ~/.ssh/known_hosts
    script:
      - ssh -oHostKeyAlgorithms=+ssh-dss gitlab@${DEPLOYMENT_SERVER_IP} ls
  </code></pre>
  <p>Still got no look with that &amp; I get the same error in the output of the runner &amp; the log on the DO box. Any ideas?</p>